Image processing and computer vision Chapter 9: 3D reconstruction and pose estimation from N-frames using Factorization (Structure From Motion SFM) for

Image processing and computer vision

Chapter 9: 3D reconstruction and pose estimation from N-frames using

Factorization (Structure From Motion SFM) for affine cameras

Factorization v4g 1

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade

Methods

• Different camera models– Perspective camera– Affine cameras

• Orthographic camera• Weak perspective camera

• Factorization (linear/fast, but not too accurate)– Affine/Orthographic approach

• Bundle adjustment (slower but more accurate)

Factorization v4g 2


Relate world 3D to camera 3D coordinates, see 156[1]

• Camera motion (rotation=R, translation=C) will cause change of pixel position (x,y)

• Pw=World 3D coordinates=[Xw,Yw,Zw]T

• Pc=camera 3D coordinates =[Xc,Yc,Zc] T

3

2

1

333231

232221

131211

,

1

10

*

1

c

c

c

C

rrr

rrr

rrr

R

Z

Y

X

CRR

Z

Y

X

C

Z

Y

X

R

Z

Y

X

w

w

w

c

c

c

w

w

w

c

c

c

Factorization v4g 3


Different camera models

• Perspective camera– No approximation– Mathematically non-linear , so some people do not

prefer it.

• Affine cameras – Approximated cameras (object far away)– Mathematically linear, good for calculation, e.g

• Orthographic camera• Weak perspective camera

Factorization v4g 4

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Exercise 1: Camera and world coordinates

•

(*)|

10

1

10

scoordinate homogenuos Use

)points 3D are ,( ),(

scoordinate osinhomogenu

. 3D tod transfomeis point 3D A

int

'

''

'

XCIRKx

XRCR

Z

Y

X

RCRX

RCRX

XXCXRX

XX

cam

camcam

cam

:

., using explicitly (*) Rewrite

(*)| (ii)

is What (i)

,

Given :Exercise

321int

int

3

2

1

333231

232221

131211

Answer

,c,c,cs,u,v,rK

XCIRKx

RC?

c

c

c

, and C[su,sv,s]x

rrr

rrr

rrr

R

ij

T

Factorization v4g 5

World coordinates

Camera coordinates

C

Ycam

Xcam

Zcam

Zw

Xw

Yw


Perspective Camera:camera motion is R,T

• Perspective projection matrix, to make life easier set Kint=I3

) of row theis :( .

,

1

1

1

ssume],*|[

3

2

1

333231

232221

131211

34333231

24232221

14131211

3333231

2232221

1131211

)43(

intint)43(

RirNote

c

c

c

,C

rrr

rrr

rrr

R

pppp

pppp

pppp

Crrrr

Crrrr

Crrrr

P

KaCRRKP

thiT

T

T

T

eperspectiv

eperspectiv

Factorization v4g 6

Constraint:Orthogonal matrix:QT=Q-1

QTQ=QQT=IIf Q is orthogonal

•


Perspective Projective

•

Factorization v4g 7

Model M at t=1

c (Image center, ox,oy)

F=focal length

image

Oc=(0,0,0) (Camera center)

Xc-axis

Zc-axis

Yc-axis

v-axis

u-axis

X,Y,Z

(u,v)

(0,0) of image plane

Principal axis

Principalplane

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Some interesting fact

• The principal plane is the vector P3 (3th row of P)

). of row 3rd theof vector the(or

:tionrepresenta in vector plane The

above. as formula thehas plane Principal theSo

equation) plane a is(it 0 is (**) of rowlast The

plane. principal on the isit So

0.but Z anything becan YX,that situation therepresents (**)

00 isit scoordinate ousinhomogeneIn

(**)

1110

possible) are values ZY, X,any so center, at worldnot is camera (The

infinity at is image its plane, Principal on the is point a If

3

34333231

34333231

34333231

24232221

14131211

int

PP

pppp

pZpYpXp

],[,y/x/

Z

Y

X

pppp

pppp

pppp

Z

Y

X

PZ

Y

X

KKy

x

],[X,Y,Z

T

TT

ext

T

Factorization v4g 8

Principal planeZ=0, X,Y can be values

Cameracenter

Image center

Z Principal axis

Y

X

At the principal Plane Z=0

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Exercise 2: More interesting facts

• The principal axis is the vector m3 (3th row of M)

Axis. and Plane Principal for the formulars theWrite:

4315

8762

0491

e.g

). of row 3rd theof vector the( isdirection axis Principal

(ii) and (i) comparingBy direction. axis Principal

)(

isequation plane above the vector tonormal thedefinitionBy

)(0 be shown tobeen has plane Principal The

)(

1

|

formanother in it expresscan or we

)(

1

3131313

333231

34333231

3333231

2232221

1131211

34333231

24232221

14131211

int

Question

,P

mmmmm

z

ppp z

iiipZpYpXp

iiZ

Y

X

cmmm

cmmm

cmmm

CM

s

sv

su

iZ

Y

X

pppp

pppp

pppp

KK

s

sv

su

T

T

ext

Factorization v4g 9

Principal plane

Principal axis

Cameracenter

Image center

Z

Y

X


Affine camera model

• Approximated cameras (object far away)• Mathematically linear, good for

calculation, e.g.• Orthographic camera• Weak perspective camera

Factorization v4g 10

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade An affine camera is an approximation of the perspective camera

• Recall: Perspective camera matrix• Constraint:

– The 3x3 sub_matrix [p11 to p33] is orthogonal--it is a 3x3 rotation matrix. If Q is an orthogonal matrix:

• QT=Q-1

• QTQ=QQT=1

• Affine camera matrix• Last row is [0 0 0 1]• Constraint:

– Rank 2 for [submatrix(P11P23)]

orthogonal is :constraint

,

],*|[

333231

232221

131211

34333231

24232221

14131211

)43(

int)43(

ppp

ppp

ppp

pppp

pppp

pppp

P

TRRKP

eperspectiv

eperspectiv


1000

~~~~

~~~~~

24232221

14131211

pppp

pppp

Paffine

Constraint:Rank 2

•

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Exercise 3:From perspective to affine

if depth (Z) is large enough, a perspective camera is shown below

•

34333231

24232221

14131211

3333231

2232221

1131211

intint43

'

'

'

'|

pppp

pppp

pppp

Crrrr

Crrrr

Crrrr

KCIRKPT

T

T

.set read, easier toit make To

. 1,0 then if

so ,orthogonal is :Note

large. is if

become will

'

'

'

:

3

3

22

11

int43

3333231

2232221

1131211

int43

tCrd

else*rrji

R

t

dr

Crr

Crr

KP

Crrrr

Crrrr

Crrrr

KP

showExercise

Tt

jiT

tT

TT

TT

T

T

T


C’ is the position of the camera in world coordinates. Imaging the camera is now moving backward along the principal axis -r3 at a rate of (t) pixels per second.C’ can now be replaced by C-(t)r3, and at t=0, the camera is at C. Note: the principal axis =(r3)3x1, where r3T is the 3rd row of r (proved earlier).

1 hence ,0

and ,0 hence ,0 If

so orthongal, is

matrix rotation of

row theis

13

12

11

131211

23

22

21

131211

333231

232221

131211

1

r

r

r

rrr*rr

r

r

r

rrr*rj, ri

rrr

rrr

rrr

R

ir

iiT

jiT

thT


Exercise 4: Perspective to Affine• In execise3, the object will become smaller and smaller but if we zoom the

camera simultaneously by a magnification factor (f f(dt/d0)) the object will maintain the same size on screen, where d0=dt=0.

camera. affinean is So

1000is of rowlast but the unchanged, rows first two The

000

show,When

:Exercise

same. theare they see shouldyou and above of sidesboth expand,doubt in If

1

1

1

)/(

)/(

1

)/(

)/(

)(

.earlier set We

0

430

24232221

14131211

0

43

3

22

11

int0

43

3

22

11

33

int

43

3

22

11

33

int

0

0

43

3

22

11

33

int

3

)P(

]d [)P(

d

pppp

pppp

)P(d

dt

dddd

dr

Crr

Crr

Kd

d

dr

Crr

Crr

dd

K

dr

Crr

Crr

dd

dd

K

d

d

d

d

dr

Crr

Crr

dd

dd

KtP

tCrd

t

t

ot

t

oT

TT

TT

t

tT

TT

TT

t

o

tT

TT

TT

ot

ot

t

t

tT

TT

TT

ot

ot

Tt

Factorization v4g13

Magnification matrix

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade The proof showed that • When the object is very far away d =dt=

• By definition dt=-r31C+t , r is constant (the rotation is fixed) , C is the initial position of the camera and we view it using a large zoom lens (f f(dt= /d0)).

• The above camera viewing process is an affine viewing process. Note f=focal length.

• In reality if the object is far away from the camera, we can approximate it by using the affine camera math model. The advantage is the system is linear and is easier to be handled mathematically by linear algebra.



Example:Vertigo (film) Hitchcock, 1958http://www.youtube.com/watch?v=GnpZN2HQ3OQ (around 2:05)

• The same set of objects• Perspective Affine• Zoomed out Zoomed in• Object size depends on distance Object size less sensitive to distance

Factorization v4g

15

Note the ratio between the floor and hands: Larger

The ratio between the floor and hands: smaller

http://www.youtube.com/watch?v=GnpZN2HQ3OQ


Perspective V.s. orthographic

• Orthographic camera is an approximated camera model that assumes no depth change

• Orthographic camera is a type of affine camera

Factorization v4g16

http://www.soulsofdistortion.nl/images/last_supper.jpghttp://1.bp.blogspot.com/_FOIrYyQawGI/SGLoj58NBiI/AAAAAAAAAoQ/pYvm01Xj8Bg/s1600/LastSupper.jpg


Affine approximation• Camera focal length =1• t3 =0 (no motion in Z direction, Z is a constant)

• Rotation is affine

e_pointis_an_imag),(

)on translati(affine 12 is ),(

[1] p166 see 2),rank rotation, (affine 32 is

1

~*

*

,

21

T

T

affine

vux

ttt

M

Z

Y

X

P

s

vs

us

ort

Z

Y

X

Mv

ux


1000

~~~~

~~~~~

24232221

14131211

pppp

pppp

Paffine

M2x3, t2x1

Affine rotation, Affine ‘

translation


Examples of affine cameras• Orthographic camera• Weak perspective camera • both have the same projective matrix form

1000

~~~~

~~~~~

24232221

14131211

pppp

pppp

Paffine



Orthographic paintings, sizes of objects do not depend on distances from objects to the viewer

• Orthographic, Orthographic


http://www.love-egypt.com/images/painting1.jpg 唐人：宮樂圖http://www.nigensha.co.jp/kokyu/ch/p01.html

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Other approximated camera models: Weak perspective camera: A type of affine cameras

• (Full) Perspective camera image (ui,vi)– ui=f*Xi / Zi

– vi=f*Yi / Zi

– (ui,vi) depends on Zi

• Weak perspective camera image (ui,vi) – ui=f*Xi / Zmean

– vi=f*Yi / Zmean

– All (ui,vi)s depend on the same depth (Zmean)



Structure from motion (SFM)

Find the 3D structure/model of an object from a sequence of images taking when the object (or

camera) is in motionWe will introduce the method using the Factorization method for affine cameras


Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Exercise 5 : Structure from motion for affine cameras

a) Discuss differences between the affine and perspective camera. b) Which terms in an affine camera matrix govern affine rotation?c) Which terms in an affine camera matrix govern affine translation? • Input a sequence of image features xj(j=1,..n) of

an object, for time i=1,2,…,,.• Find

– Structure of the object, Xj=(X,Y,Z)j for all features points, j=1,2,…N

– camera motion, PAffine for each i

ii

affine

i

affine

tMtbbb

taaa

iP

pppp

pppp

iP

101000

)(

notations ofset another useor

1000

~~~~

~~~~

12322321

1321

24232221

14131211

Factorization v4g22

Crrrr

Crrrr

Crrrr

PT

T

T

eperspectiv3

333231

2232221

1131211

)43(

camera eperspectiv tocompared As

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Tracking

(Click picture to see movie)

• Demo for • tracking


Demohttp://www.youtube.com/watch?v=RXpX9TJlpd0

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Factorization for Affine Camera P.437[1]

•

)index (feature ,...1),index sample (time ,...,1

[1] p166 see),on translati(affine ,

2),rank rotation, (affine ,

101000

rotation affine:,

2

112

321

32132

12322321

1321

nji

t

tt

bbb

aaaM

tMtbbb

taaa

P

MNotet

Z

Y

X

Mv

ux

affine


Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Factorization for Affine Camera P.437[1]

• :

2

,,,

2

,,,

2

1

321

321

)(minˆmin

minimized is when ,,

find topoints) image measured (the given theuse

is algorithmion factorizat theof target The

(affine)point model 3D theis

point feature image 2D measured theis

point, feature image 2D estimated theis ˆ

)index (feature ,...1),index sample (time ,...,1

)on translati(affine ,

2),rank rotation, (affine ,

ji

ij

iij

XtMji

ij

ij

XtM

ij

i

ij

j

ij

ij

tXMxxxE

EtXM

x

X

x

x

nji

t

tt

bbb

aaaM

jii

jii

Factorization v4g25

t

Z

Y

X

Mv

ux

slidelast in the Defined


Motion of the object• 3D Model=Xj : where j=feature index =1,2…n

features

Factorization v4g26

Time (i)1 2 3 …

=controid

1)ˆ,ˆ( itM

2)ˆ,ˆ( itM

3)ˆ,ˆ( itM

i

ij

),t(Mi

..n,j ,x

,...Γ,i

1232 is at time Pose :Find To

312index feature features imageExtract

21index at time imagean Capture

itM )ˆ,ˆ(

12

ijx

11

ijx

1

injx

i

njx

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Factorization for Affine Camera

• Find affine M1,…,M • Minimization of re-

projection error• i = 1,…, (time sample

index) • j = 1,…,n (model point

index)

n vectorTranslatio (2x1)

ation affine_rot (2x3)

point model (3x1)

point image (2x1)an is

)(2

,

i

i

j

Tij

ij

ij

ji

ij

iij

t

M

X

,vux

where

tXMx



Find translation (t) first: a simple task for affine cameras

• At time i, feature index j• For affine cameras translation =[ui

j, vij]T (size is 2x1)

• Centroid is the translation (Centroid =mean of all 2D feature points at time i)

point image (2x1)an is

1

Tij

ij

ij

j

ij

ii

,vux

xn

xt



Exercise 6: iij

ij xxx

• At time i, features are xj

• x1=[4, 5]• x2=[6,7]• x3=[1, 8]• Mean is ??_________

• The new xj’ is xj’• X1’=?________• x2’=?_________• x3’=?_________


• So translation is found• t(x,y)=• ?


•

i

ij

),t(Mi

..n,j ,x

,...Γ,i

1232 is at time Pose :Find To

312index feature features imageExtract

21index at time imagean Capture



Build the measurement matrix W from measurements x’ after translation ti is eliminated

• Time index i=1,2,..,• Feature index j=1,2,..,n

• x’=[u’,v’]T is an (2x1) image point (translation eliminated). )2(21

112

11

21

112

11

)2(

'...''

::::

'...''

'...''

::::

'...''

n

Γn

ΓΓ

n

Γn

ΓΓ

n

n

vvv

vvv

uuu

uuu

W

)eliminatedon (translatipoint image (2x1) a is'

''

v

ux i

j

iji

j


Time Indexi=1,2..

Feature index :j=1,2,,nW=measured 2D image feature (translation eliminated)

ixx

xxv

uxx

xn

xt

ji

iiji

j

iji

jt

ij

j

ij

ii

i

at time all ofmean where

,

1

'

''

eliminated

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Construct the input data set

• We can factorize W to obtain Mi and Xj for all i,j because

)found be to( at timerotation affine (2x3)

)(min)('min

achieved. ison Minimizati

when

'...''

::::

'...''

'...''

::::

'...''

2

,,

2

,,

321

32321

131211

321

131211

2

)2(21

112

11

21

112

11

)2(

iM

XMWXMx

X...XX

bbb

:::

bbb

aaa

:::

aaa

MXW

vvv

vvv

uuu

uuu

W

i

jij

i

XMjij

iij

XM

nn

ΓΓ,iΓ,iΓ,i

,i,i,i

Γ,iΓ,iΓ,i

,i,i,i

n)Γ(

n

Γn

ΓΓ

n

Γn

ΓΓ

n

n

ji

ji

2

,,

2

,,

)(min

,)('min

find want towe

, eliminated is After

jij

i

XM

jij

iij

XM

i

XMW

orXMx

t

ji

ji

Factorization v4g32

Time Indexi=1,2..

Feature index :j=1,2,,n

Mi

Xi

W=measured 2D image feature (translation eliminated)Week 9 begins

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Ideal W is Rank 3 • From Linear algebra, the rank of a product of A,B is

the minimum of { rank (A), rank (B)}.

3rank best at is ˆ so ,ˆˆˆ ideal

'ˆ'ˆˆ

ˆ ideal and measured minimize To

...

:::

:::

ˆ

is ˆ Ideal

3322

2222

321

32,3,2,1

1,31,21,1

,3,2,1

1,31,21,1

)2(

)2(

WXMW

XMxxxWWWW

WW

XXX

bbb

bbb

aaa

aaa

W

W

nbybynby

ijj

iij

ij

ij

ij

ijijij

F

nn

iii

iii

iii

iii

n

n


Both are 3, using linear algebra theoryso the product is rank 3


Use SVD to set rank 3 constraint to W, (Note solution is not unique, See P438 [1])

•

n

nnT

nT

T

n

nnT

nn

Tn

S

S

VV

U

VandS

rankS

W

VSUW

USVW

2

33

3

2232

n542

2222

2

of aluessingular v

3first theofmatrix diagonal theis'

of rows 3first theis'

of columns 3first theis 'U

'','Uin aresolution Then the

0,.., settingby 3 make weSo

norm Frobenius in the closest tomatrix 3rank theis

ˆ ideal

If

)or

or (

norm Frobenius theis

2p case specialA

,

norm

2

2

2

1

n

ii

pn

i

p

ip

xx

-normlnorm

Euclidean

xx

p


Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade What is SVD?

Singular Value Decomposition• A is mxn, decompose it into 3

matrices: U, S, V• U is mxm is an orthogonal

matrix• S is mxn (diagonal matrix)• V is nxn is an orthogonal

matrix

• 1, 2, n, are singular values• Columns of vectors of U=left

singular vectors• Columns of vectors of V=right

singular vectors

xonal_matriiki/Orthogedia.org/w//en.wikip:http

and

unit orthogonal are rows and columns

entries real matrix square a is R

Orthogonal is If :Reference

1

RR

IRRRR

vectors

with

R

T

TT

0...

0

.

.

0

21

2

1

n

n

Tnmnmmm

S

VSUA



SVD (singular value decomposition)

• SVD

,,nj

AA

xxAA

VV

VV

UU

UU

ASVD

jj

nT

T

n

T

nnn

n

nmmm

m

nm

1

.. that such , )( of valueseigen are

)(

valueseigen withRelation

...

...

.....

.....

.....

...

0

.

.

0

...

.....

.....

.....

...

)(

211,2,3...

21

,1,

,11,1

2

1

,1,

,11,1


Right singular vectors

…Right singular vectors

Singular values .,..., 2211 etc

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade See how SVD can be used here

•

nn

iii

iii

iii

iii

n XXX

bbb

bbb

aaa

aaa

W

321

32,3,3,3

1,31,31,3

,3,3,3

1,31,21,1

)2( ...

:::

:::

ˆ

n

T

nnn

n

nmmm

m

nmnm

VV

VV

UU

UU

WSVDW

...

...

.....

.....

.....

...

0

.

0

...

.....

.....

.....

...

)(

21

,1,

,11,1

3

2

1

,1,

,11,1


3 columns

3 rows

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade The Factorization Algorithm (part 1)

•

VvvvXXX

Mbbb

aaami

WWσ,σ,σ λ

uuuuuu

bbb

:::

bbb

aaa

:::

aaa

M

σσ

Vcv

Scσ

Ucu

USVWSVDUSV

Wn

xxx

xn

xt

Tnn

iii

iiii

T,,

Γ

ΓiΓiΓi

iii

ΓiΓiΓi

iii

s

nnth

c

nth

c

thc

TT

nΓ

iij

ij

j

ij

ii

of columns 3 First ][.. matrix point ModelThe

of is at time pose The

of valueseigenlargest theare :Note

:poses The

discarded) are other , aluessignular vonly (use results :(step4)

of column theis

of aluesingular v theis

of column theis

)(by find then

,data input 2 theconstruct :(step3)

effect ion translateliminate data tocenter :(step2)

n)calculatio on(translati ,1

:(step1)

321321

321

321

321321

332211332211

32321

13

12

11

321

13

12

11

3,2,1

2

22

2

nn

Γ

ΓiΓiΓi

iii

ΓiΓiΓi

iii

n

XXX

bbb

:::

bbb

aaa

:::

aaa

W

321

32321

13

12

11

321

13

12

11

)2(

...

ˆ

Factorization v4g38

Measured 2D features (translation eliminated)

W

Intro.| Perspective Cam. | Affine Cam.| Ortho. Cam | SFM | Factorization| Upgrade Exercise 7: indicate in the lower diagram where are the

vectors: u1,u2,u3, v1,v2,v3 thatenforce the rank3 constraint

• )(

.....

.....

.....

.....

.....

0

.

0

.....

.....

.....

.....

.....

23

2

1

n

n

Wsvd

.....

.....

.....

.....

.....

00

0

0

.....

.....

.....

.....

.....

~3

2

1

w


0...Set 54 n

useless are parts unshaded rank3, be would~w

VTU

VT

U


The Factorization Algorithm (part 2): Metric upgrade• The M and X obtained previous are only approximations because

they do not satisfy the metric transform constraint, that means


.,0

~~~~~~

,1~~~~~~

,1~~~~~~

~~~,~~~ rows two theexample,For

.constraintion normalizat estaisfy thmust of ows

,~~~~~~

is at time pose theIf

321321

321321

321321

321321

321

321

etcbbbaaa

bbbbbb

aaaaaa

bbbaaa

MR

bbb

aaaMi

Tiiiiii

Tiiiiii

Tiiiiii

iiiiii

itimemetric

iii

iiiitime

metric


•


LQ

III

III

III

QQL

X

qqq

qqq

qqq

XQX

qqq

qqq

qqq

MQMM

soQXM

T

n

T

nmetric

metric

or find tois hereour task So

matrix symmetric a is which Set

,matrix 3x3 a using earlier) (found and upgradecan We

653

542

321

3

333231

232221

131211

31

333231

232221

131211

33)32()32(

33

Ref: Morita and Kanade,"A Sequential Factorization Method for Recovering Shape and Motion From Image Streams", IEEE PAMI, VOL. 19, NO. 8, AUGUST 1997


•


32321

321

32321

13

12

11

321

13

12

11

is at time pose eextract thcan we, From :Note

iii

iiii

Γ

ΓiΓiΓi

iii

ΓiΓiΓi

iii

bbb

aaaM

iM

bbb

:::

bbb

aaa

:::

aaa

M

Ref: Morita and Kanade,"A Sequential Factorization Method for Recovering Shape and Motion From Image Streams", IEEE PAMI, VOL. 19, NO. 8, AUGUST 1997


•


33

2332

22

1331

1221

11

1

12

6

1

11

11

11

0

:

0

1

:

1

yx

yxyx

yx

yxyx

yxyx

yx

g(x,y)

,c

I

:

I

,I

),b(ag

:

),b(ag

),b(bg

:

),b(bg

),a(ag

:

),a(ag

G

cGI

iiT

iiT

iiT

iiT

iiT

iiT

)(

Hence

found. be alsocan and

definitionby since and found is After

)(_

31

33)32()32(

653

542

321

Done

XQX

QMM

QL

III

III

III

QQL

I

cGinversepseudoI

nmetric

metric

T


Reconstruction result• Structure From Motion

-- factorization method demo

• --Input: 18 frames• --Output: VRML file • Mat lab implementation• http://www.youtube.com/watch?v=azl-

DGK6e1U• Python implementation• https://www.youtube.com/watch?v=Teakx

TW20mI


Input

Wireframe output

Model output

https://www.youtube.com/watch?v=TeakxTW20mI

https://www.youtube.com/watch?v=TeakxTW20mI


Bundle adjustment

• Non linear iterative method is more accurate than linear method, require first guess.

• Usually factorization gives the first guess for the bundle adjustment method.



Summary

• Studied the factorization structure-from-motion method for single camera multiple views


Reference

• [1] Hartley and Zisserman, Multiple geometry in computer vision, Cambridge, University Press. 2002.

• [2] Morita and Kanade,"A Sequential Factorization Method for Recovering Shape and Motion From Image Streams", IEEE PAMI, VOL. 19, NO. 8, AUGUST 1997


Appendices


Appendix 1More properties of SVD

iiiT

iiiT

ni

i

mi

i

nT

vvAA

uuAA

vvvV

v

vuuU

u

Define

AA

2

2

1

1

222

21

i

)(

)(

....

ectorssingular vright V of vectorsof Columns

....

ectorssingular vleft Uof vectorsof Columns

.,..,, of valuesEigen

is of Meaning


Appendix 2 Frobenius norm

• A is an m-by-n matrix Frobenius norm ||A||F

• See http://mathworld.wolfram.com/FrobeniusNorm.html

m

i

n

jijF

aA1 1

2


Appendix 3Rank of matrix

http://en.wikipedia.org/wiki/Rank_(linear_algebra)

• If A is of size m x n, Rank(A)<min{m,n}• Rank(AB)< min{rank(A), rank(B)}• Rank(A)= number of non zero singular value

found using SVD.


Projective matrix

Compositions Matrix form Constrains Degree of freedom of M

perspectiveM

int3 3 3 3 3 1|x x xK R T =

11 12 13 1

21 22 23 2

31 32 33 3

0

0 0

0 0 1

Tx c

Ty c

Tc

f s r r r R T

f r r r R T

r r r R T

11 12 13 14

21 22 23 24

31 32 33 34

m m m m

m m m m

m m m m

11 12 13

21 22 23

31 32 33

m m m

m m m

m m m

= orthogonal

9

orthographicM

11 12 13 1

21 22 23 2

1 0 0

0 1 0

0 0 1 0 0 0 1

Tc

Tc

r r r R T

r r r R T

11 12 13 1

21 22 23 2

0 0 0 1

m m m t

m m m t

11 12 13[ , , ]m m m

and

21 22 23[ , , ]m m m

are orthogonal and of unit norm

5

_scaled orthoM

11 12 13 1

21 22 23 2

0 0

0 0

0 0 1 0 0 0 1

Tc

Tc

k r r r R T

k r r r R T

11 12 13 1

21 22 23 2

0 0 0 1/

m m m t

m m m t

k

11 12 13[ , , ]m m m

and

21 22 23[ , , ]m m m

are orthogonal and equal norm.

6

_week perspM

11 12 13 1

21 22 23 2

0 0

0 0

0 0 1 0 0 0 1

Tx c

Ty c

k r r r R T

k r r r R T

11 12 13 1

21 22 23 2

0 0 0

m m m t

m m m t

d

11 12 13[ , , ]m m m

and

21 22 23[ , , ]m m m

are orthogonal and need have same norm

7

affineM 11 12 13 1

21 22 23 2

0

0 0

0 0 0 0 0 0 1

Tx c

Ty c

k s r r r R T

k r r r R T

11 12 13 1

21 22 23 2

0 0 0 1

m m m t

m m m t

rank of

11 12 13

21 22 23

m m m

m m m

is 2.

8

Appendix 4Different approximate camera models


Appendix 5


Answer1: Camera and world coordinates

•

(*)|

10

1

10

scoordinate homogenuos Use

)points 3D are ,( ),(

scoordinate osinhomogenu

. 3D tod transfomeis point 3D A

int

'

''

'

XCIRKx

XRCR

Z

Y

X

RCRX

RCRX

XXCXRX

XX

cam

camcam

cam

1

:)(

)(:

., using explicitly (*) Rewrite

(*)| (2)

is What (i)

,

Given :Exercise

333232131333231

323222121232221

313212111131211

int

333232131

323222121

313212111

3

2

1

333231

232221

131211

321int

int

3

2

1

333231

232221

131211

Z

Y

X

crcrcrrrr

crcrcrrrr

crcrcrrrr

K

s

sv

su

ii

crcrcr

crcrcr

crcrcr

c

c

c

rrr

rrr

rrr

RC

iAnswer

,c,c,cs,u,v,rK

XCIRKx

RC?

c

c

c

and C[su,sv,s],x

rrr

rrr

rrr

R

ij

World coordinates

Camera coordinates

C

Ycam

Xcam

Zcam

Zcam

Xcam

Ycam


Answer2: Exercise 2: More interesting fact

• The principal plane is the vector P3 (3th row of P)

T

T

TT

ext

T

ZYXplaneprinciple,PP

PP

pppp

pZpYpXp

],[,y/x/

Z

Y

X

pppp

pppp

pppp

KKy

x

],[X,Y,Z

315axis principal

0435_

4315

8762

0491

e.g

). of row 3rd theof vector the(or

tion representa in vector plane The

above. as formula thehas plane Principal theSo

equation) plane a is(it 0 is side handright The

00 isit scoordinate ousinhomogeneIn

10

possible) are values ZY, X,any so center, at worldnot is camera (The

infinity at is image its plane, Principal on the is point a If

3

3

34333231

34333231

34333231

24232221

14131211

int

Principal plane

Principal axis

Cameracenter

Image center

Z

Y

X

At the principal Plane Z=0


Answer 3

•

correct. isrelation theseecan you and

..etc) , (e.g.out termsall writedoubt,in If

.set read, easier toit make To

...0,1 E.g.

. 1,0 then if so ,orthogonal is Note

)(

)(

)(

)(

'

'

'

'|

in )('Cput

3332313

3

3111

3

22

11

int33

22

11

int333

322

311

int

3333231

2232221

1131211

intint43

3

rrrr

tCrd

etcrrrr

else*rrjiR

dr

Crr

Crr

K

tCrr

Crr

Crr

K

rtCrr

rtCrr

rtCrr

KtP

Crrrr

Crrrr

Crrrr

KCIRKP

rtC

T

Tt

TTTT

jiT

tT

TT

TT

TT

TT

TT

TT

TT

TT

T

T

T

Factorization v4g57

Answer 4:Perspective to Affine

• Of course the object will become smaller but if we zoom the camera simultaneously by a magnification factor of (dt/d0) the object will maintain the same size on screen.

camera. affinean is So

1000is of rowlast but the unchanged, rows first two The

0000

*gain

large, is when 3), exercisein (defined Since

0)( of rowlast in the termthe,gain,When

,,,, is( of rowlast The

1

)/(

)/(

)(

0

0

24232221

14131211

0

22

11

intlarge

3

large0

033323103

03

22

11

int03

22

11

int

)P(

]d [)P(

d

pppp

pppp

d

Crr

Crr

K)P(

dttCrd

tPdd

d

dt

dddrd

drddrdd

drt) P

dddr

Crr

Crr

Kd

d

dr

Crr

Crr

dd

dd

KtP

TT

TT

tT

t

t

ot

t

o

t

o

t

o

t

oT

t

oT

TT

TT

t

tT

TT

TT

ot

ot


Ans:Exercise 5 : Structure from motion for affine camerasa) Discuss differences between the affine and perspective camera.Ans: Affine: last row is [0 0 0 1]b) Which terms in an affine camera matrix govern affine rotation?Ans: m11,m12,m13,m21,m21,m22c) Which terms in an affine camera matrix govern affine translation? ans:T1 t2

• Input a sequence of image features xj(j=1,..n) of an object, for time i=1,2,…,,.

• Find – Structure of the object, Xj=(X,Y,Z)j for all features points,

j=1,2,…N– camera motion, PAffine for each i

ii

affine

i

affine

tMtmmm

tmmm

iP

pppp

pppp

iP

101000

)(

notations ofset another useor

1000

~~~~

~~~~~

12322222121

1131211

24232221

14131211

Crrrr

Crrrr

Crrrr

PT

T

T

eperspectiv3

333231

2232221

1131211

)43(

camera eperspectiv tocompared As


Answer: Exercise 6ii

jij xxx

• At time i, features are xj• x1=[4, 5] T

• x2=[6,7] T

• x3=[1, 8] T

• Mean is• =[(4+6+1)/3, (5+7+8)/3] T=[3.67,6.67]T

• The new (translation eliminated) xj is xj’• x1’=[4-3.67,5-6.67] T=[0.33,0.33] T

• x2’=[6-3.67,7-6.67] T=[2.33,0.33] T

• x3’=[1-3.67,8-6.67] T=[-2,67,1.33] T

• So translation is found• t(x,y)=[3.67,6.67] T


After translation (t) is foundProcess input features to eliminate translation

Wn

x

xxx

i

iij

ij

data input 2

theconstruct

centroid

effect ntranslatio

eliminate data toCenter

• • At time i, features are xj• x1=[4, 5]• x2=[6,7]• x3=[1, 8]• Mean is [(4+6+1)/3,

(5+7+8)/3]=[3.67,6.67]• The new xj is xj’• x1’=[4-3.67,5-6.67]=[0.33,0.33]• x2’=[6-3.67,7-6.67]=[2.33,0.33]• x3’=[1-3.67,8-6.67]=[-2,67,1.33]

• So xj’ is now the input (instead of xj)


Answer7:Exercise: indicate in the lower diagram where are the vectors: u1,u2,u3, v1,v2,v3

Enforce rank3 constraint

• )(

.....

.....

.....

.....

.....

0

.

0

.....

.....

.....

.....

.....

22

2

1

n

n

Wsvd

.....

.....

.....

.....

.....

00

0

0

.....

.....

.....

.....

.....

~3

2

1

w

0...Set 54 n

useless are parts unshaded rank3, be would~w3

3

VTU

v1,v2,v3

u1,u2,u3


• >>running result• rank(a) is 4• find svd : [u,s,v]=svd(a)

• u=-0.2861 -0.4365 0.3907 0.7582• -0.5835 -0.5352 0.1321 -0.5964• -0.7075 0.6956 0.0888 0.0877• -0.2776 -0.1980 -0.9066 0.2485• s = 20.6985 0 0 0• 0 9.6845 0 0• 0 0 3.9981 0• 0 0 0 1.6733• v = -0.7078 0.6946 -0.0844 -0.0972• -0.2578 -0.3908 -0.0377 -0.8828• -0.4356 -0.4743 -0.6719 0.3658• -0.4928 -0.3739 0.7349 0.2780• a_same =1.0000 2.0000 4.0000 6.0000• 5.0000 6.0000 7.0000 8.0000• 15.0000 1.0000 3.0000 5.0000• 3.0000 2.0000 6.0000 1.0000• a = 1 2 4 6• 5 6 7 8• 15 1 3 5• 3 2 6 1• make it rank3 by setting ss(4,4)=0• ss = 20.6985 0 0 0• 0 9.6845 0 0• 0 0 3.9981 0• 0 0 0 0• using the ss,find aa=u*ss*vT• aa =1.1233 3.1201 3.5359 5.6473• 4.9030 5.1190 7.3651 8.2775• 15.0143 1.1296 2.9463 4.9592• 3.0404 2.3671 5.8479 0.8844• rank(aa), should be 3

• %svd_demo1.m• a=[1 2 4 6 ;5 6 7 8;…• 15 1 3 5; 3 2 6 1]• 'rank(a) is '• rank(a)• 'find svd : [u,s,v]=svd(a)'• [u,s,v]=svd(a)• a_same=u*s*v'• a• 'make it rank3 by ss(4,4)=0‘• ss=s, ss(4,4)=0• 'using the ss,find aa=u*ss*vT'• aa=u*ss*v'• 'rank(aa), should be 3'• rank(aa)

Documents

Image processing and computer vision Chapter 9: 3D reconstruction and pose estimation from N-frames using Factorization (Structure From Motion SFM) for