Structure from MotionStructure from Motion
• For now, static scene and moving cameraFor now, static scene and moving camera– Equivalently, rigidly moving scene andEquivalently, rigidly moving scene and
static camerastatic camera
• Limiting case of stereo with many Limiting case of stereo with many camerascameras
• Limiting case of multiview camera Limiting case of multiview camera calibration with unknown targetcalibration with unknown target
• Given Given nn points and points and NN camera positions, camera positions, have 2have 2nNnN equations and 3 equations and 3nn+6+6NN unknowns unknowns
ApproachesApproaches
• Obtaining point correspondencesObtaining point correspondences– Optical flowOptical flow– Stereo methods: correlation, feature Stereo methods: correlation, feature
matchingmatching
• Solving for points and camera motionSolving for points and camera motion– Nonlinear minimization (bundle Nonlinear minimization (bundle
adjustment)adjustment)– Various approximations…Various approximations…
OrthographicOrthographic ApproximationApproximation
• Simplest SFM case: camera Simplest SFM case: camera approximated by orthographic approximated by orthographic projectionprojection
PerspectivePerspective OrthographicOrthographic
Weak PerspectiveWeak Perspective
• An orthographic assumption is An orthographic assumption is sometimes well approximated by a sometimes well approximated by a telephoto lenstelephoto lens
Weak PerspectiveWeak Perspective
Consequences ofConsequences ofOrthographic ProjectionOrthographic Projection
• Scene can be recovered up to scaleScene can be recovered up to scale
• Translation perpendicular to image Translation perpendicular to image planeplanecan never be recoveredcan never be recovered
Orthographic Structure from Orthographic Structure from MotionMotion
• Method due to Tomasi & Kanade, Method due to Tomasi & Kanade, 19921992
• Assume Assume nn points in space points in space pp11 … … ppnn
• Observed at Observed at NN points in time at points in time at image coordinates (image coordinates (xxijij, , yyijij))– Feature tracking, optical flow, etc.Feature tracking, optical flow, etc.
Orthographic Structure from Orthographic Structure from MotionMotion
• Write down matrix of dataWrite down matrix of data
NnN
n
NnN
n
yy
yy
xx
xx
1
111
1
111
D
NnN
n
NnN
n
yy
yy
xx
xx
1
111
1
111
D
Points Points Fra
mes
Fram
es
Orthographic Structure from Orthographic Structure from MotionMotion
• Step 1: find translationStep 1: find translation
• Translation parallel to viewingTranslation parallel to viewingdirection can not be obtaineddirection can not be obtained
• Translation perpendicular to viewing Translation perpendicular to viewing direction equals motion of average direction equals motion of average position of all pointsposition of all points
Orthographic Structure from Orthographic Structure from MotionMotion
• Subtract average of each rowSubtract average of each row
NNnNN
n
NNnNN
n
yyyy
yyyy
xxxx
xxxx
1
11111
1
11111
~D
NNnNN
n
NNnNN
n
yyyy
yyyy
xxxx
xxxx
1
11111
1
11111
~D
Orthographic Structure from Orthographic Structure from MotionMotion
• Step 2: try to find rotationStep 2: try to find rotation
• Rotation at each frame defines local Rotation at each frame defines local coordinate axes , , andcoordinate axes , , and
• ThenThen
ii jj kk
jiijjiij yx pjpi ~ˆ~,~ˆ~ jiijjiij yx pjpi ~ˆ~,~ˆ~
Orthographic Structure from Orthographic Structure from MotionMotion
• So, can write where R is a So, can write where R is a “rotation” matrix and S is a “shape” “rotation” matrix and S is a “shape” matrixmatrix
RSD~ RSD~
n
N
N ppS
j
j
i
i
R ~~
ˆ
ˆ
ˆ
ˆ
1
T
T1
T
T1
n
N
N ppS
j
j
i
i
R ~~
ˆ
ˆ
ˆ
ˆ
1
T
T1
T
T1
Orthographic Structure from Orthographic Structure from MotionMotion
• Goal is to factor Goal is to factor
• Before we do, observe that Before we do, observe that rankrank( ) = 3( ) = 3(in ideal case with no noise)(in ideal case with no noise)
• Proof:Proof:– Rank of Rank of RR is 3 unless no rotation is 3 unless no rotation– Rank of Rank of SS is 3 iff have noncoplanar points is 3 iff have noncoplanar points– Product of 2 matrices of rank 3 has rank 3Product of 2 matrices of rank 3 has rank 3
• With noise, With noise, rankrank( ) might be > 3( ) might be > 3
D~D~
D~D~
D~D~
Digression:Digression:Singular Value Decomposition Singular Value Decomposition
(SVD)(SVD)• Handy mathematical technique that Handy mathematical technique that
has application to many problemshas application to many problems
• Given any Given any mmnn matrix matrix AA, algorithm to , algorithm to find matrices find matrices UU, , VV, and , and WW such that such that
AA = = UU WW VVTT
UU is is mmnn and orthonormal and orthonormalV V is is nnnn and orthonormal and orthonormalWW is is nnnn and diagonal and diagonal
SVDSVD
• Treat as black box: code widely Treat as black box: code widely available (available (svd(A,0)svd(A,0) in Matlab) in Matlab)
T
1
00
00
00
VUA
nw
w
T
1
00
00
00
VUA
nw
w
SVDSVD
• The The wwii are called the are called the singular valuessingular values of of
AA
• If If AA is singular, some of the is singular, some of the wwii will be 0will be 0
• In general In general rankrank((AA) = number of nonzero ) = number of nonzero wwii
• SVD is mostly unique (up to SVD is mostly unique (up to permutation of singular values, or if permutation of singular values, or if some some wwii are equal) are equal)
SVD and InversesSVD and Inverses
• Why is SVD so useful?Why is SVD so useful?
• Application #1: inversesApplication #1: inverses
• AA-1-1=(=(VVTT))-1-1 WW-1-1 UU-1-1 = = VV WW-1-1 UUTT
• This fails when some This fails when some wwii are 0 are 0– It’s It’s supposedsupposed to fail – singular matrix to fail – singular matrix
• Pseudoinverse: if Pseudoinverse: if wwii=0, set 1/=0, set 1/wwii to 0 (!) to 0 (!)– ““Closest” matrix to inverseClosest” matrix to inverse– Defined for all (even non-square) Defined for all (even non-square)
matricesmatrices
SVD and Least SquaresSVD and Least Squares
• Solving Solving AxAx==bb by least squares by least squares
• xx=pseudoinverse(=pseudoinverse(AA) times ) times bb
• Compute pseudoinverse using SVDCompute pseudoinverse using SVD– Lets you see if data is singularLets you see if data is singular– Even if not singular, ratio of max to min Even if not singular, ratio of max to min
singular values (condition number) tells singular values (condition number) tells you how stable the solution will beyou how stable the solution will be
– Set 1/Set 1/wwii to 0 if to 0 if wwii is small (even if not is small (even if not
exactly 0)exactly 0)
SVD and EigenvectorsSVD and Eigenvectors
• Let Let AA==UWVUWVTT, and let , and let xxii be be iithth column of column of VV
• Consider Consider AATTAA xxii::
• So elements of So elements of WW are squared are squared eigenvalues and columns of eigenvalues and columns of VV are are eigenvectors of eigenvectors of AATTAA
iiiii xwwxx 222T2TTTT
0
0
0
1
0
VVWVVWUWVUVWAA iiiii xwwxx 222T2TTTT
0
0
0
1
0
VVWVVWUWVUVWAA
SVD and EigenvectorsSVD and Eigenvectors
• Eigenvectors of Eigenvectors of AATTAA turned up in turned up in finding least-squares solution to finding least-squares solution to AxAx=0=0
• Solution is eigenvector corresponding Solution is eigenvector corresponding to smallest eigenvalueto smallest eigenvalue
SVD and Matrix SimilaritySVD and Matrix Similarity
• One common definition for the norm of a One common definition for the norm of a matrix is the Frobenius norm:matrix is the Frobenius norm:
• Frobenius norm can be computed from Frobenius norm can be computed from SVDSVD
• So changes to a matrix can be evaluated So changes to a matrix can be evaluated by looking at changes to singular valuesby looking at changes to singular values
i j
ija2
FA
i jija2
FA
i
iw2
FA
iiw2
FA
SVD and Matrix SimilaritySVD and Matrix Similarity
• Suppose you want to find best rank-Suppose you want to find best rank-kk approximation to approximation to AA
• Answer: set all but the largest Answer: set all but the largest kk singular values to zerosingular values to zero
• Can form compact representation by Can form compact representation by eliminating columns of eliminating columns of UU and and VV corresponding to zeroed corresponding to zeroed wwii
SVD and PCASVD and PCA
• Principal Components Analysis (PCA): Principal Components Analysis (PCA): approximating a high-dimensional approximating a high-dimensional data set with a lower-dimensional data set with a lower-dimensional subspacesubspace
Original axesOriginal axes
****
******
**** **
**
********
**
**
****** **
**** ******
Data pointsData points
First principal componentFirst principal componentSecond principal componentSecond principal component
SVD and PCASVD and PCA
• Data matrix with points as rows, take Data matrix with points as rows, take SVDSVD
• Columns of Columns of VVkk are principal are principal
componentscomponents
• Value of Value of wwii gives importance of each gives importance of each
componentcomponent
SVD and OrthogonalizationSVD and Orthogonalization
• The matrix The matrix UU is the “closest” is the “closest” orthonormal matrix to orthonormal matrix to AA
• Yet another useful application of the Yet another useful application of the matrix-approximation properties of SVDmatrix-approximation properties of SVD
• Much more stable numerically thanMuch more stable numerically thanGraham-Schmidt orthogonalizationGraham-Schmidt orthogonalization
• Find rotation given general affine matrixFind rotation given general affine matrix
End of DigressionEnd of Digression
• Goal is to factor into Goal is to factor into RR and and SS
• Let’s apply SVD!Let’s apply SVD!
• But should have rank 3 But should have rank 3 all but 3 of the all but 3 of the wwii should be 0 should be 0
• Extract the top 3 Extract the top 3 wwii, together with , together with
the corresponding columns of the corresponding columns of UU and and VV
D~D~
T~UWVD T~UWVD
D~D~
Factoring for Factoring for Orthographic Structure from Orthographic Structure from
MotionMotion• After extracting columns, After extracting columns, UU33 has has
dimensions 2dimensions 2NN3 (just what we 3 (just what we wanted for wanted for RR))
• WW33VV33TT has dimensions 3 has dimensions 3nn (just what (just what
we wanted for we wanted for SS))
• So, let So, let RR**==UU33, , SS**==WW33VV33TT
Affine Structure from MotionAffine Structure from Motion
• The The ii and and jj entries of entries of RR** are not, in are not, in general, general, unit length and perpendicularunit length and perpendicular
• We have found motion (and therefore We have found motion (and therefore shape)shape)up to an affine transformationup to an affine transformation
• This is the best we could do if we didn’tThis is the best we could do if we didn’tassume orthographic cameraassume orthographic camera
Ensuring OrthogonalityEnsuring Orthogonality
• Since can be factored as Since can be factored as RR** SS**, it , it can also be factored as (can also be factored as (RR**QQ)()(QQ-1-1SS**), ), for any for any QQ
• So, search for So, search for QQ such that such that RR == RR** QQ has the properties we wanthas the properties we want
D~D~
Ensuring OrthogonalityEnsuring Orthogonality
• Want orWant or
• Let Let TT = = QQQQTT
• Equations for elements of Equations for elements of TT – solve by – solve byleast squaresleast squares
• Ambiguity – add constraints Ambiguity – add constraints
1ˆˆ T*T* QiQi ii 1ˆˆ T*T* QiQi ii
0ˆˆ
1ˆˆ
1ˆˆ
*TT*
*TT*
*TT*
ii
ii
ii
jQQi
jQQj
iQQi
0ˆˆ
1ˆˆ
1ˆˆ
*TT*
*TT*
*TT*
ii
ii
ii
jQQi
jQQj
iQQi
0
1
0ˆ,
0
0
1ˆ *
1T*
1T jQiQ
0
1
0ˆ,
0
0
1ˆ *
1T*
1T jQiQ
Ensuring OrthogonalityEnsuring Orthogonality
• Have found Have found TT = = QQQQTT
• Find Find Q Q by taking “square root” of by taking “square root” of TT– Cholesky decomposition if Cholesky decomposition if TT is positive is positive
definitedefinite– General algorithms (e.g. General algorithms (e.g. sqrtmsqrtm in in
Matlab)Matlab)
Orthogonal Structure from Orthogonal Structure from MotionMotion
• Let’s recap:Let’s recap:– Write down matrix of observationsWrite down matrix of observations– Find translation from avg. positionFind translation from avg. position– Subtract translationSubtract translation– Factor matrix using SVDFactor matrix using SVD– Write down equations for orthogonalizationWrite down equations for orthogonalization– Solve using least squares, square rootSolve using least squares, square root
• At end, get matrix At end, get matrix RR == RR** QQ of camera of camera positionspositionsand matrix and matrix SS == QQ-1-1SS** of 3D points of 3D points