Computer and Robot Vision II

Computer and Robot Vision II

Chapter 15Motion and Surface Structure from

Time Varying Image Sequences

Presented by: 傅楸善 & 王林農0917 533843

[email protected]指導教授 : 傅楸善博士

mailto:[email protected]

DC & CV Lab.DC & CV Lab.CSIE NTU

15.1 Introduction

Motion analysis involves estimating the relative motion of objects with respect to each other and the camera given two or more perspective projection images in a time sequence.


15.1 Introduction (cont’)

Real-world applications:

industrial automation and inspection, robot assembly, autonomous vehicle navigation, biomedical engineering, remote sensing, general 3D-scene understanding


15.1 Introduction (cont’)

object motion and surface structure recovery from: observed optic flow point correspondences


15.2 The Fundamental Optic Flow Equation

(x, y, z): 3D point on moving rigid body (u, v): perspective projection on the image pla

ne f: camera constant (u, v): velocity of the point (u, v)

. .


15.2 The Fundamental Optic Flow Equation (cont’)

take time derivatives of both sides

yields the fundamental optic flow equation:

y

x

z

f

v

u

yzzy

xzzx

z

f

v

u2

z

y

x

v

u

f

f

zv

u 0

0

1

.

.. .. .

.

.

.

.

.


15.2 The Fundamental Optic Flow Equation (cont’)

general solution: (λ is a free variable)

f

v

u

v

u

f

z

z

y

x

0

.

.

.

.

.


15.2.1 Translational Motion

Known: N-point optic flow field:

Unknown: corresponding unknown 3D points: all points moving with same but unknown velocity (x, y, z)

can be solved up to a multiplicative constant

Nnnnnn vuvu 1)},,,{(

. .

Nnnnn zyx 1)},,{(

. . .


15.2.2 Focus of Expansion and Contraction

Known: 3D motion is translational one 2D projected point (u, v) has no motion:

thus translational motion is in a direction along the ray of sight

0vu. .


15.2.2 Focus of Expansion and Contraction (cont’)

focus of expansion (FOE): if 3D point field moving toward camera

FOE: motion-field vectors radiate outward from that point

focus of contraction (FOC): if 3D point field moving away from camera

FOC: vectors radiate inward toward diametrically opposite point flow pattern of the motion field of a forward-moving observer



15.2.3 Moving Line Segment

Known: fixed distance between two unknown 3D points

translational motion with common velocity

(x, y, z) corresponding optic flow:

),,(),,,( 222111 zyxzyx

. . .

)()( 22221111 v,u,,vu,v,u,,vu


15.2.3 Moving Line Segment (cont’)

Unknown: : two unknown 3D points

common velocity: (x, y, z)

),,(),,,( 222111 zyxzyx

. . .



From the perspective projection equations:

From the optic flow equation:



From the known length of the line segment:

The optic flow equation (15.9) permits us to obtain a least squares solution for z in terms of z1 and z2, from



We obtain

Substituting this back into the equation, we can solve z2 in terms of z1:z2=kz1



Substitute the relations for (x1, y1, z1) from equations into Eq.(15.10) to obtain

Hence:


15.2.4 Optic Flow Acceleration Invariant

Since differentiating general solution in Sec 15.2 and s

olve for (x, y, z)

zf .

.. .. ..

f

v

u

f

zv

u

f

zv

u

f

z

z

y

x

0

2

0

..

..

..

..

..

.

.. .

.


joke


15.3 Rigid-Body Motion

Rigid-body motion: no relative motion of points w.r.t. (with respect to) one another

Rigid-body motion: points maintain fixed position relative to one another

Rigid-body motion: all points move with the body as a whole


15.3 Rigid-Body Motion (cont’)

R(t): rotation matrix T(t): translation vector p(0): initial position of given point R(0)=I, T(0)=0 p(t): position of given point at time t



Rigid-body motion in displacement vectors:

velocity vector: time derivative of its position:

)()0()()( tTptRtP

)()0()()( tTptRtP .. .



Since

(a) translational-motion field under projection onto hemispherical surface only translational-component motion useful in determining scene structure

(b) rotational-motion field under projection onto hemispherical surface rotational-motion field provides no information about scene structure

)()()()()()()()(

)]()()[()0(11

1

tTtTtRtRtptRtRtp

tTtptRp

. . .





we can describe rigid-body motion in instantaneous velocity by

)()()()( tktpttp .



: angular velocities in three axes : translational velocities in three axes from rigid-body-motion equation

zyx ,,

zyx kkk ,,

zyx

yxz

xzy

z

y

x

z

y

x

kxy

kzx

kyz

k

k

k

z

y

x

k

z

y

x

z

y

x

.

.

.



and perspective projection equation

we can determine an expression for z:

f

v

u

f

z

z

y

x

.

zyx kuvf

zz )(

.



after simplification

z

y

x

z

y

x

k

k

k

u

v

f

uv

f

uf

f

vf

f

uv

z

uz

u

z

fz

f

v

u

22

22

0

0

..



image velocity: expressed as sum of translational field and rotational field

(x, y, z): 3D coordinate before rigid-body motion in displacement vectors

(x’, y’, z’): 3D coordinate after rigid-body motion in displacement vectors

: rotation angles in three axes : translation in three axes

),,( zyx ),,( zyx ttt



Rigid-body motion in displacement vectors:

z

y

x

t

t

t

z

y

x

RRR

z

y

x

yxyxy

zxzyxzxzyxzy

zxzyxzxzyxzy

z

y

x

xyz

coscoscossinsin

sinsincossincoscoscossinsinsinsincos

sinsincossincossincoscossinsincoscos

)()()(

'

'

'



motion in displacement vector and instantaneous velocity is different:

e.g. moon encircling earth instantaneous velocity: first order approximation of

displacement vector first order approximation: when small,

0sinsin,1cos,sin



first order approximation: when time=1 thus x=(x’ - x)/1 first order approximation:

.

zzyyxxzzyyxx tktktk ,,,,,

zxy

yzx

xyz

z

y

x

xy

xz

yz

tyxz

txzy

tzyx

t

t

t

z

y

x

z

y

x

1

1

1

'

'

'


joke


15.4 Linear Algorithms for Motion and Surface Structure from Optic Flow

15.4.1 The Planar Patch Case : arbitrary object point on planar

patch at time t : central projective coordinates of p(t) o

nto image plane z= f

)]'(),(),([)( tztytxtp

)](),([ tvtu

]'),(),([)(

)(

)(

)()(

)(

)()(

ftvtuf

tztp

tz

tyftv

tz

txftu


15.4.1 The Planar Patch Case

: instantaneous velocity of moving image point

: optic flow image point : instantaneous rotational angular

velocity : instantaneous translational velocity

)](),([ tvtu)](),([ tvtu

. .

)](),(),(),([ tvtutvtu . .

)(t

)(tk


15.4.1 The Planar Patch Case (cont’)

unit vector n(t): orthogonal to moving planar patch rigid planar patch motion represented by rigid-moti

on constraint:

1)()'(

)()()()(

tptn

tktptwtp.



from above two equations:

Let

Rigid-motion constraint could be written as

pknpknpp

pknpp

)'('

0

0

0

' ,,

'

12

13

23

321



denote the 3 x 3 matrix by W and its three row vectors by

W: called planar motion parameter matrix since skew symmetric

'kn',',' 321

''' nkknWW



above equation can be written as

from perspective projection equations:

taking time derivatives of these equations we have

vzzv

uzzu

fy

x

v

u

f

z

y

x

z

y

x

W

z

y

x

1..

..

.

.

.

.

.



substitute equations into above equations:

from third row

substitute z to obtain optical flow-planar motion equation

v

u

f

v

u

v

u

f

f

v

u

f

zz

f

v

u

z

fz

vzzv

uzzu

32

1

3

3

2

1

'1

'

'

'

'

'

'

. .

. ..

.

.

.

.



we have 2N linear equations: n=1,…,N:

optic flow-planar motion recovery: first solve W then find

n

nn

n

n

n

v

u

f

v

u

v

u

f 32

1 '1

'

'

.

.

nk,,


15.4.2 General Case Optic Flow-Motion Equation

1. set up optic flow-motion equation not involving depth information

2. solve it by using linear least-squares technique


15.4.3 A Linear Algorithm for Solving Optic FlowMotion Equations


15.5.4 Mode of Motion, Direction of Translation, and Surface Structure

mode of motion: whether translation k=0 or not direction of translation: direction of k surface structure: relative depth when k 0


15.4.5 Linear Optic Flow-Motion Algorithm and Simulation Results

motion and shape recovery algorithms should answer three questions:

minimum number of points to compute motion and shape

what set of optic flow points violate rank assumption e.g. collinearity…

What’s the accuracy of estimated motion from noisy optic flow?


joke


15.5 The Two View-Linear Motion Algorithm


15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review

Two View-Planar Motion Equation imaging geometry for two view-planar motion rigid planar patch in motion in half-space z< 0



15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)

: arbitrary object point before motion : same object point after motion : central projective coordinates of

f : camera constant

),(),,(

)',,(

)',,(

2211

2222

1111

vuvu

zyxp

zyxp

21, pp

2

22

2

22

1

11

1

11

,

,

zyfvz

xfu

zyfvz

xfu



R0: 3 X 3 rotational matrix, R0’R0=I,|R0|=1

t0: 3 X 1 translational vector

n0: 3 X 1 normal vector



Rigid-body-motion equation relates p1 to p2 as follows:

planarity constrains p1 by

combining two equations produces planar rigid-body-motion-equation

10002

10

0102

)'(

1'

pntRp

pn

tpRp



Projecting the planar rigid-body motion onto the image plane z = f produces

Let the planar rigid –motion parameter matrix be defined by

Where bi, i=1,2,3, are three row vectors of B.



Then above equation could be written as



From above we derive the two view-planar motion equation

With the natural constraint



Now the planar motion recovery problem involves first solving the planar rigid-motion parameter matrix B and then estimating 000 and,, rtR


15.5.2 General Curved Patch Motion Recovery from Two Perspective Views A Simplified Linear Algorithm

discard planar patch assumption, consider general curved patch


15.5.3 Determining Translational Orientation


15.5.4 Determining Mode of Motion and Relative Depths


15.5.5 A Simplified Two View-Motion Linear Algorithm


15.5.6 Discussion and Summary

when no noise appears: algorithm extremely accurate

when small noise appears: it works well except mode of motion incorrect


15.6 Linear Algorithm for Motion and Structure from Three Orthographic Views

Ullman (1979) showed that for the orthographic case four-point correspondences over three views are sufficient to determine the motion and structure of the four-point rigid configuration


Shimon Ullman, The Interpretation of Visual Motion

The MIT Press, Cambridge MA. 1979


15.6 Linear Algorithm for Motion and Structure from Three Orthographic Views

to infer depth information: translation needed in perspective projection

to infer depth information: rotation useless in perspective projection

to infer depth information: rotation needed in orthographic projection

to infer depth information translation useless in orthographic projection



15.6.1 Problem Formulation

image plane stationary three orthographic views at time

(x, y, z): object-space coordinates of point P at t1

(x’, y’, z’): object-space coordinates of point P at t2

(x”, y”, z”): object-space coordinates of point P at t3

(u, v): image-space coordinates of P at t1

(u’, v’): image-space coordinates of P at t2

(u”, v”): image-space coordinates of P at t3

321 ,, ttt


15.6.1 Problem Formulation (cont’)

: rotation matrix : translation vector

(x’, y’, z’)’ = R(x’, y’, z’)+Tr

(x”, y”, z”)” = S(x”, y”, z”)+Ts

1313

3333

)(,)(

)(,)(

sisrir

ijij

sTtT

sSrR



Known: four image-point correspondences

Unkown:

4,3,2,1),,,(

),(),,(

4,3,2,1),","()','(),(

izyx

TSTR

iuuuuuu

iii

sr

iiiiii



note that with orthographic projections

therefore it is obvious that tr3, ts3 can never be determined

we are trying to determine:

)","()","(

)','()','(

),(),(

yxvu

yxvu

yxvu

4,3,2,,2,1,,,, 1 izzittSR isiri


15.6.2 Determining ),,,(),,,,(,, 32312313323123133333 ssssrrrrsr


15.6.3 Solving a Unique Orthonormal Matrix R


15.6.4 Linear Algorithm to Uniquely Solve R, s, a3


15.6.5 Summary

Given two orthographic views, one cannot finitely determine the motion and structure of a rigid body, no matter how many point correspondences are used, as shown by Huang.


Joke


15.7 Developing a Highly Robust Estimator for General Regression


15.7.1 Inability of the Classical Robust M-Estimator to Render High Robustness

Classical robust estimator, such as M-, L-, or R-estimator:

1. optimal or nearly optimal at assumed noise distribution

2. relatively small performance degradation with small number of outliers

3. larger deviations from assumed distribution do not cause catastrophe

MF-estimator with new property much stronger than property 3

relatively small performance degradation with larger deviations from assumed distribution


15.7.2 Partially Modeling Log Likelihood Function by Using Heuristics

MF-estimator: robust regression more appropriate model-fitting


15.7.3 Discussion

M-, L-, R and MF-estimator: all residual based


15.7.4 MF-Estimator

MF-estimator: combine Bayes statistical decision rule with heuristics


15.8 Optic Flow-Instantaneous Rigid-Motion Segmentation and Estimation

formulate optic flow-single rigid-motion estimation into general regression


15.8.1 Single Rigid Motion

P(t): position vector of an object point at the time t

[X(t), Y(t)]: central projective coordinate of P(t)

)]'(),(),([)( tztytxtp


15.8.1 Single Rigid Motion (cont’)

]'),(),([)(

)(

)(

)()(

)(

)()(

ftYtXf

tztp

tz

tyftY

tz

txftX



: noisy optic flow image point Instantaneous representation of the rigid motion is

described by

Instantaneous rotational angular velocity of rigid motion:

Instantaneous translational angular velocity of rigid motion:

)]'(),(),([)(

)]'(),(),([)(

)()()()(

)]}(),([)],(),({[

321

321

tktktktk

tttt

tktpttp

tvtutYtX



Differentiating above equation

where for simplicity the time variable t has been omitted

)'0,,()',,( vuf

zfYX

f

zp



Combine above two equations:

)'0,,()',,()',,( vuf

zfYX

f

zkfYX

f

z


15.8.2 Multiple Rigid Motions

We turn the optic flow-multiple rigid –motion segmentation and estimation problem into a number of successive optic flow-single rigid-motion estimation problems.


joke


15.9 Experimental Protocol

Simulate simplest location estimation Simulate Optic flow-rigid-motion

segmentation and estimation


15.9 Experimental Protocol (cont’)






15.10 Motion and Surface Structure from Line Correspondences

Discussion concerns only the general rigid motion of straight-line structure


15.10.1 Problem Formulation

Cartesian reference system-central projection





l: line in 3D space L: projection of the line on image plane z = f z = f : image frame : known plane line L is in; projective plane of l : set of lines in 3D space : lines moved by rigid motion (R’ , T’)’ at time t’ : lines moved by rigid motion (R” , T”)” at time t”

},...,{

},...,{

},...,{

""1

''1

1

k

k

k

ll

ll

ll



: projections of lines

; respective projective planes

}"{}{}{

}{}{}{

k1k1k1

k1k1k1

",...,,',...,',,...,

L",...,L",L',...,L',L,...,L



Known: K triples of line correspondences in three views

Unkown: rotations and translations: 3D lines

kiLLL iii ,...,1,"'

",',",' TTRR

Kili ,...,1,


15.10.2 Solving Rotation Matrices R’, R” and Translations T’,R”

: the normals of the ith projective planes

Then

",',

",',

iii

iii nnn

0)""(

0)''(

0

"

'

TpRpn

TpRpn

pn

ti

ti

ti


15.10.2 Solving Rotation Matrices R’, R” and Translations T’,R”

Above two equations define a unique three-dimensional line solution if and only if

2''

Rn

nRank

ti

ti


15.10.3 Solving Three-Dimensional Line Structure

Once rotation martrices R’, R” and translations T’, T” are solved, each three-dimensional line li can be determined


15.11 Multiple Rigid Motions from Two Perspective Views

15.11.1 Problem Statement imaging geometry for two-view-motion



15.11.1 Problem Statement

How many good point correspondences are needed in order to apply the nonlinear least-squares estimator?


15.11.2 Simulated Experiments


15.11.2 Simulated Experiments (cont’)




15.12 Rigid Motion from Three Orthographic Views


15.12.1 Problem Formulation and Algorithm

same as Sec. 15.6, instead of linear algorithms, formulate model-fitting problem


15.12.2 Simulated Experiments






15.12.3 Further Research on the MF-Estimator

two problems to be solved for MF-estimator to be practically useful:

distance problem requirement for a good initial approximation


difficulty of motion and shape recovery: ambiguity of displacement field

Fuh. Ph.D. Thesis, Fig 4.1



15.13 Literature Review

15.13.1 Inferring Motion and Surface Structure



classifications for methods of inferring 3D motion and shape use of individual sets of feature points use of local optic flow information about a single p

oint use of the entire optic flow field



Despite all the results obtained over the years, almost none of these inference techniques have been successfully applied to feature-point correspondences calculated from real imagery


Joke


15.13.2 Computing Optic Flow or Image-Point Correspondences

problem source contains abundant information occlusion boundaries specular points near a focus of expansion noise and digitization effects in image formation


15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)

motion parallax: apparent relative motion between objects and observer

points in observer’s direction of translation remain relatively unchanged information available to a moving observer




impart time dimension to image data spatiotemporal image data block




motion field: assignment of vectors to image points representing motion

angular velocity of fixed scene: inversely proportional to distance pilot in straight-ahead level flight on an overcast day





motion field of pilot looking straight ahead in motion direction

zero image velocity: at approach point and at infinity (along horizon)




motion field of pilot looking to the right in level flight

focus of expansion here: at infinity to the left focus of contraction here: at infinity to the righ

t of the figure




spatiotemporal image data acquired by a camera,- caption -

straight streaks at block top due to translating parallel to image plane



joke


B.K.P. Horn, Robot Vision, The MIT Press, Cambridge, MA, 1986

Chapter 12 Motion Field & Optical Flow optic flow: apparent motion of brightness patt

erns during relative motion


12.1 Motion Field

motion field: assigns velocity vector to each point in the image

Po: some point on the surface of an object

Pi: corresponding point in the image

vo: object point velocity relative to camera

vi: motion in corresponding image point


12.1 Motion Field (cont’)

ri: distance between perspectivity center and image point

ro: distance between perspectivity center and object point

f’: camera constant z: depth axis, optic axis object point displacement causes correspondi

ng image point displacement





Velocities:

where ro and ri are related by



differentiation of this perspective projection equation yields


joke


12.2 Optical Flow

optical flow need not always correspond to the motion field

(a) perfectly uniform sphere rotating under constant illumination:

no optical flow, yet nonzero motion field (b) fixed sphere illuminated by moving light

source: nonzero optical flow, yet zero motion field



12.2 Optical Flow (cont’)

not easy to decide which P’ on contour C’ corresponds to P on C




optical flow: not uniquely determined by local information in changing

irradiance at time t at image point (x, y)

components of optical flow vector



assumption: irradiance the same at time

fact: motion field continuous almost everywhere



expand above equation in Taylor series

e: second- and higher-order terms in cancelling E( x, y, t), dividing through by



which is actually just the expansion of the equation

abbreviations:



we obtain optical flow constraint equation:

flow velocity (u, v): lies along straight line perpendicular to intensity gradient




rewrite constraint equation:

aperture problem: cannot determine optical flow along isobrightness contour


12.3 Smoothness of the Optical Flow

motion field: usually varies smoothly in most parts of image

try to minimize a measure of departure from smoothness


12.3 Smoothness of the Optical Flow (cont’)

error in optical flow constraint equation should be small

overall, to minimize


12.3 Smoothness of the Optical Flow (cont’)

large if brightness measurements are accurate

small if brightness measurements are noisy


12.4 Filling in Optical Flow Information

regions of uniform brightness: optical flow velocity cannot be found locally

brightness corners: reliable information is available


12.5 Boundary Conditions

Well-posed problem: solution exists and is unique

partial differential equation: infinite number of solution unless with boundary


joke


12.6 The Discrete Case

first partial derivatives of u, v: can be estimated using difference



12.6 The Discrete Case (cont’)

measure of departure from smoothness:

error in optical flow constraint equation:

to seek set of values that minimize



dieffrentiating e with respect to



where are local average of u, v (9 neighbors? )

extremum occurs where the above derivatives of e are zero:



determinant of 2x2 coefficient matrix:

so that



suggests iterative scheme such as

new value of (u, v): average of surrounding values minus adjustment




first derivatives estimated using first differences in 2x2x2 cube



12.6 The Discrete Case (cont’) consistent estimates of three first partial deriv

atives:



four successive synthetic images of rotating sphere




estimated optical flow after 1, 4, 16, and 64 iterations




(a) estimated optical flow after several more iterations

(b) computed motion field



12.7 Discontinuities in Optical Flow

discontinuities in optical flow: on silhouettes where occlusion occurs


Joke


Project due May 2

implementing Horn & Schunck optical flow estimation as above

synthetically translate lena.im one pixel to the right and downward

Try 10 1, 0.1, of

Documents

Computer and Robot Vision II