View
0
Download
0
Category
Preview:
Citation preview
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Computer Vision – Lecture 17
Uncalibrated Reconstruction & SfM
08.07.2019
Bastian Leibe
Visual Computing Institute
RWTH Aachen University
http://www.vision.rwth-aachen.de/
leibe@vision.rwth-aachen.de
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Announcements
• No lecture tomorrow (Tuesday)
Due to a schedule conflict
• Last exercise will be offered on Monday, 15.07.
Optional, but recommended
Time slot & room to be announced…
• Repetition slides
I will provide a slide set (pdf) with summary slides for the
entire lecture
Idea: you can use this as an index to the lecture
2B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Course Outline
• Image Processing Basics
• Segmentation & Grouping
• Object Recognition
• Local Features & Matching
• Deep Learning
• 3D Reconstruction
Epipolar Geometry and Stereo Basics
Camera calibration & Triangulation
Uncalibrated Reconstruction & Active Stereo
Structure-from-Motion
3
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recap: A General Point
• Equations of the form
• How do we solve them? (always!)
Apply SVD
Singular values of A = square roots of the eigenvalues of ATA.
The solution of Ax=0 is the nullspace vector of A.
This corresponds to the smallest singular vector of A.4
B. Leibe
Ax 0
11 11 1
1
T
N
T
NN N NN
d v v
d v v
A UDV U
SVD
Singular values Singular vectors
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recap: Camera Parameters
• Intrinsic parameters
Principal point coordinates
Focal length
Pixel magnification factors
Skew (non-rectangular pixels)
Radial distortion
• Extrinsic parameters
Rotation 𝐑
Translation 𝐭(both relative to world coordinate system)
• Camera projection matrix
General pinhole camera: 9 DoF
CCD Camera with square pixels: 10 DoF
General camera: 11 DoF 5B. Leibe
0
0
1 1 1
x x x
y y y
m f p x
K m f p y
s s
P K R | t
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recap: Calibrating a Camera
• Goal
Compute intrinsic and extrinsic parameters
using observed camera data.
• Main idea
Place “calibration object” with known
geometry in the scene
Get correspondences
Solve for mapping from scene to image:
estimate P=PintPext
6B. LeibeSlide credit: Kristen Grauman
? P
Xi
xi
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recap: Camera Calibration (DLT Algorithm)
• P has 11 degrees of freedom.
• Two linearly independent equations per independent 2D/3D
correspondence.
• Solve with SVD (similar to homography estimation)
Solution corresponds to smallest singular vector.
• 5 ½ correspondences needed for a minimal solution.7
B. Leibe
• Solve with …?
Slide adapted from Svetlana Lazebnik
0pA 0
P
P
P
X0X
XX0
X0X
XX0
3
2
1111
111
T
nn
TT
n
T
nn
T
n
T
TTT
TTT
x
y
x
y
? P
Xi
xi
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
• Stack equations and solve with …?
Recap: Triangulation – Linear Algebraic Approach
• Two independent equations each in terms of
three unknown entries of X.
• Stack equations and solve with SVD.
• This approach nicely generalizes to multiple cameras.8
B. Leibe
XPx
XPx
222
111
0XPx
0XPx
22
11
0XP][x
0XP][x
22
11
Slide credit: Svetlana Lazebnik
O1O2
x1x2
X?
R1R2
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Topics of This Lecture
• Revisiting Epipolar Geometry Calibrated case: Essential matrix
Uncalibrated case: Fundamental matrix
Weak calibration
Epipolar Transfer
• Active Stereo Kinect sensor
Structured Light sensing
Laser scanning
• Structure from Motion (SfM) Motivation
Ambiguity
Projective factorization
Bundle adjustment
9B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recap: Epipolar Geometry – Calibrated Case
10B. Leibe
X
x x’
Slide credit: Svetlana Lazebnik
Essential Matrix
(Longuet-Higgins, 1981)
0)]([ xRtx RtExExT ][with0
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Epipolar Geometry: Calibrated Case
• E x’ is the epipolar line associated with x’ (l = E x’)
• ETx is the epipolar line associated with x (l’ = ETx)
• E e’ = 0 and ETe = 0
• E is singular (rank two)
• E has five degrees of freedom (up to scale)12
B. Leibe
X
x x’
Slide credit: Svetlana Lazebnik
0)]([ xRtx RtExExT ][with0
Why?
Why?
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Epipolar Geometry: Uncalibrated Case
• The calibration matrices K and K’ of the two cameras are unknown
• We can write the epipolar constraint in terms of unknownnormalized coordinates:
15B. Leibe
X
x x’
Slide credit: Svetlana Lazebnik
0ˆˆ xExT xKxxKx ˆ,ˆ
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Epipolar Geometry: Uncalibrated Case
16B. Leibe
X
x x’
Slide credit: Svetlana Lazebnik
Fundamental Matrix(Faugeras and Luong, 1992)
0ˆˆ xExT
xKx
xKx
ˆ
ˆ
1with0 KEKFxFx TT
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Epipolar Geometry: Uncalibrated Case
• F x’ is the epipolar line associated with x’ (l = F x’)
• FTx is the epipolar line associated with x (l’ = FTx)• F e’ = 0 and FTe = 0
• F is singular (rank two)
• F has seven degrees of freedom17
B. Leibe
X
x x’
Slide credit: Svetlana Lazebnik
0ˆˆ xExT 1with0 KEKFxFx TT
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Estimating the Fundamental Matrix
• The Fundamental matrix defines the epipolar geometry
between two uncalibrated cameras.
• How can we estimate F from an image pair?
We need correspondences…
18B. Leibe
xi
Xi
x0i
P? P0?
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
The Eight-Point Algorithm
• Taking 8 correspondences:
19B. Leibe
x = (u, v, 1)T, x’ = (u’, v’, 1)T
This minimizes:
2
1
)( i
N
i
T
i xFx
Slide adapted from Svetlana Lazebnik
[u0u; u0v; u0; uv0; vv0; v0; u; v; 1]
26666666666664
F11
F12
F13
F21
F22
F23
F31
F32
F33
37777777777775
= 0
Af = 0
Solve using… ?Solve using… SVD!
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Excursion: Properties of SVD
• Frobenius norm
Generalization of the Euclidean norm to matrices
• Partial reconstruction property of SVD
Let i i=1,…,N be the singular values of A.
Let Ap = UpDpVpT be the reconstruction of A when we set
p+1,…, N to zero.
Then Ap = UpDpVpT is the best rank-p approximation of A in the
sense of the Frobenius norm
(i.e. the best least-squares approximation).20
B. Leibe
2
1 1
m n
ijFi j
A a
min( , )
2
1
m n
i
i
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
The Eight-Point Algorithm
• Problem with noisy data
The solution will usually not fulfill the constraint that F only has
rank 2.
There will be no epipoles through which all epipolar lines pass!
• Enforce the rank-2 constraint using SVD
• As we have just seen, this provides the best least-squares
approximation to the rank-2 solution.
21B. Leibe
11 11 13
22
33 31 33
T
T
d v v
F d
d v v
UDV U
SVD
Set d33 to
zero and
reconstruct F
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Problem with the Eight-Point Algorithm
• In practice, this often looks as follows:
22B. Leibe
266666666664
u01u1 u01v1 u01 u1v01 v1v
01 v01 u1 v1 1
u02u2 u02v2 u02 u2v02 v2v
02 v02 u2 v2 1
u03u3 u03v3 u03 u3v03 v3v
03 v03 u3 v3 1
u04u4 u04v4 u04 u4v04 v4v
04 v04 u4 v4 1
u05u5 u05v5 u05 u5v05 v5v
05 v05 u5 v5 1
u06u6 u06v6 u06 u6v06 v6v
06 v06 u6 v6 1
u07u7 u07v7 u07 u7v07 v7v
07 v07 u7 v7 1
u08u8 u08v8 u08 u8v08 v8v
08 v08 u8 v8 1
377777777775
26666666666664
F11
F12
F13
F21
F22
F23
F31
F32
F33
37777777777775
=
266666666664
0
0
0
0
0
0
0
0
377777777775
Slide adapted from Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Problem with the Eight-Point Algorithm
• In practice, this often looks as follows:
Poor numerical conditioning
Can be fixed by rescaling the data
23B. Leibe
266666666664
u01u1 u01v1 u01 u1v01 v1v
01 v01 u1 v1 1
u02u2 u02v2 u02 u2v02 v2v
02 v02 u2 v2 1
u03u3 u03v3 u03 u3v03 v3v
03 v03 u3 v3 1
u04u4 u04v4 u04 u4v04 v4v
04 v04 u4 v4 1
u05u5 u05v5 u05 u5v05 v5v
05 v05 u5 v5 1
u06u6 u06v6 u06 u6v06 v6v
06 v06 u6 v6 1
u07u7 u07v7 u07 u7v07 v7v
07 v07 u7 v7 1
u08u8 u08v8 u08 u8v08 v8v
08 v08 u8 v8 1
377777777775
26666666666664
F11
F12
F13
F21
F22
F23
F31
F32
F33
37777777777775
=
266666666664
0
0
0
0
0
0
0
0
377777777775
Slide adapted from Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
The Normalized Eight-Point Algorithm
1. Center the image data at the origin, and scale it so the
mean squared distance between the origin and the data
points is 2 pixels.
2. Use the eight-point algorithm to compute F from the
normalized points.
3. Enforce the rank-2 constraint using SVD.
4. Transform fundamental matrix back to original units: if T
and T’ are the normalizing transformations in the two
images, then the fundamental matrix in original coordinates
is TT F T’.24
B. Leibe [Hartley, 1995]Slide credit: Svetlana Lazebnik
11 11 13
22
33 31 33
T
T
d v v
F d
d v v
UDV U
SVD
Set d33 to
zero and
reconstruct F
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
The Eight-Point Algorithm
• Meaning of error
Sum of Euclidean distances between points xi and epipolar
lines Fx’i (or points x’i and epipolar lines FTxi), multiplied by
a scale factor
• Nonlinear approach for refining the solution: minimize
Similar to nonlinear minimization
approach for triangulation.
Iterative approach (Gauss-Newton,
Levenberg-Marquardt,…)25
B. Leibe
:)( 2
1
i
N
i
T
i xFx
N
i
i
T
iii xFxxFx1
22 ),(d),(d
O1O2
x1x2
X?
x’1
x’2
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Comparison of Estimation Algorithms
26B. Leibe
8-point Normalized 8-point Nonlinear least squares
Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel
Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
3D Reconstruction with Weak Calibration
• Want to estimate world geometry without requiring calibrated cameras.
• Many applications: Archival videos
Photos from multiple unrelated users
Dynamic camera system
• Main idea:
Estimate epipolar geometry from a (redundant) set of point correspondences between two uncalibrated cameras.
27B. LeibeSlide credit: Kristen Grauman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Stereo Pipeline with Weak Calibration
• So, where to start with uncalibrated cameras?
Need to find fundamental matrix F and the correspondences
(pairs of points (u’,v’) ↔ (u,v)).
• Procedure
1. Find interest points in both images
2. Compute correspondences
3. Compute epipolar geometry
4. Refine28
B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Stereo Pipeline with Weak Calibration
1. Find interest points (e.g. Harris corners)
29B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Stereo Pipeline with Weak Calibration
2. Match points using only proximity
30B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Putative Matches based on Correlation Search
31B. Leibe Example from Andrew Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
RANSAC for Robust Estimation of F
• Select random sample of correspondences
• Compute F using them
This determines epipolar constraint
• Evaluate amount of support – number of inliers
within threshold distance of epipolar line
• Iterate until a solution with sufficient support
has been found (or for max #iterations)
• Choose F with most support (#inliers)
32B. LeibeSlide credit: Kristen Grauman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Putative Matches based on Correlation Search
33B. Leibe Example from Andrew Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Pruned Matches
• Correspondences consistent with epipolar geometry
34B. Leibe Example from Andrew Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Resulting Epipolar Geometry
35B. Leibe Example from Andrew Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Epipolar Transfer
• Assume the epipolar geometry is known
• Given projections of the same point in two images, how can
we compute the projection of that point in a third image?
36B. Leibe
x1 x2
? x3
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Extension: Epipolar Transfer
• Assume the epipolar geometry is known
• Given projections of the same point in two images, how can
we compute the projection of that point in a third image?
37B. Leibe
x1 x2 x3l32l31
l31 = FT13 x1
l32 = FT23 x2
When does epipolar transfer fail?
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Topics of This Lecture
• Revisiting Epipolar Geometry Calibrated case: Essential matrix
Uncalibrated case: Fundamental matrix
Weak calibration
Epipolar Transfer
• Active Stereo Kinect sensor
Structured Light sensing
Laser scanning
• Structure from Motion (SfM) Motivation
Ambiguity
Projective factorization
Bundle adjustment
38B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Microsoft Kinect – How Does It Work?
• Built-in IR
projector
• IR camera for
depth
• Regular camera
for color
39B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recall: Optical Triangulation
40B. Leibe
O1
x1
X?
Image plane
Camera center
3D Scene point
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Recall: Optical Triangulation
• Principle: 3D point given by intersection of two rays.
Crucial information: point correspondence
Most expensive and error-prone step in the pipeline…
41B. Leibe
O1 O2
x1 x2
X
Image plane
Camera center
3D Scene point
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Active Stereo with Structured Light
• Idea: Replace one camera by a projector.
Project “structured” light patterns onto the object
Simplifies the correspondence problem
42B. Leibe
O1 O2
x1 x2
X
Image plane
Camera center
3D Scene point
Projector
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
What the Kinect Sees…
43B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
3D Reconstruction with the Kinect
44B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Laser Scanning
• Optical triangulation
Project a single stripe of laser light
Scan it across the surface of the object
This is a very precise version of structured light scanning
46B. Leibe
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Slide credit: Steve Seitz
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Laser Scanned Models
47B. Leibe
The Digital Michelangelo Project, Levoy et al.
Slide credit: Steve Seitz
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Laser Scanned Models
48B. Leibe
The Digital Michelangelo Project, Levoy et al.
Slide credit: Steve Seitz
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Laser Scanned Models
49B. Leibe
The Digital Michelangelo Project, Levoy et al.
Slide credit: Steve Seitz
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Laser Scanned Models
50B. Leibe
The Digital Michelangelo Project, Levoy et al.
Slide credit: Steve Seitz
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Laser Scanned Models
51B. Leibe
The Digital Michelangelo Project, Levoy et al.
Slide credit: Steve Seitz
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Poor Man’s Scanner
52Bouget and Perona, ICCV’98
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Slightly More Elaborate (But Still Cheap)
53B. Leibe
http://www.david-laserscanner.com/
Software freely available from Robotics Institute TU Braunschweig
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Topics of This Lecture
• Revisiting Epipolar Geometry Calibrated case: Essential matrix
Uncalibrated case: Fundamental matrix
Weak calibration
Epipolar Transfer
• Active Stereo Kinect sensor
Structured Light sensing
Laser scanning
• Structure from Motion (SfM) Motivation
Ambiguity
Projective factorization
Bundle adjustment
54B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Structure from Motion
• Given: m images of n fixed 3D points
xij = Pi Xj , i = 1, … , m, j = 1, … , n
• Problem: estimate m projection matrices Pi and
n 3D points Xj from the mn correspondences xij55
B. Leibe
x1j
x2j
x3j
Xj
P1
P2
P3
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Applications
• E.g., movie special effects
Video
56B. Leibe Video Credit: Stefan Hafeneger
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Structure from Motion Ambiguity
• If we scale the entire scene by some factor k and, at the
same time, scale the camera matrices by the factor of 1/k,
the projections of the scene points in the image remain
exactly the same:
It is impossible to recover the absolute scale of the scene!
57B. LeibeSlide credit: Svetlana Lazebnik
x = PX =
µ1
kP
¶(kX)
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Structure from Motion Ambiguity
• If we scale the entire scene by some factor k and, at the
same time, scale the camera matrices by the factor of 1/k,
the projections of the scene points in the image remain
exactly the same.
• More generally: if we transform the scene using a
transformation Q and apply the inverse transformation to the
camera matrices, then the images do not change
58B. LeibeSlide credit: Svetlana Lazebnik
x =PX= (PQ¡1)QX
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Reconstruction Ambiguity: Similarity
59B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman
x =PX= (PQ¡1S )QSX
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Reconstruction Ambiguity: Affine
60B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman
Affine
x =PX= (PQ¡1A )QAX
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Reconstruction Ambiguity: Projective
61B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman
x=PX= (PQ¡1P )QPX
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Projective Ambiguity
62B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
From Projective to Affine
63B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
From Affine to Similarity
64B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Hierarchy of 3D Transformations
• With no constraints on the camera calibration matrix or on the scene, we get a projective reconstruction.
• Need additional information to upgrade the reconstruction to affine, similarity, or Euclidean. 65
B. Leibe
vTv
tAProjective
15dof
Affine
12dof
Similarity
7dof
Euclidean
6dof
Preserves intersection
and tangency
Preserves parallellism,
volume ratios
Preserves angles, ratios
of length
10
tAT
10
tRT
s
10
tRT
Preserves angles,
lengths
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Structure from Motion
• Given: m images of n fixed 3D points
xij = Pi Xj , i = 1, … , m, j = 1, … , n
• Problem: estimate m projection matrices Pi and
n 3D points Xj from the mn correspondences xij66
B. Leibe
x1j
x2j
x3j
Xj
P1
P2
P3
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Projective Structure from Motion
• Given: m images of n fixed 3D points
• zij xij = Pi Xj , i = 1,… , m, j = 1, … , n
• Problem: estimate m projection matrices Pi and n 3D points
Xj from the mn correspondences xij
• With no calibration info, cameras and points can only be recovered up to a 4£4 projective transformation Q:
X → QX, P → PQ-1
• We can solve for structure and motion when
2mn >= 11m +3n – 15
• For two cameras, at least 7 points are needed.
67B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Projective SfM: Two-Camera Case
• Assume fundamental matrix 𝐅 between the two views
First camera matrix: [I|0]Q-1
Second camera matrix: [A|b]Q-1
• Let , then
• And
• So we have
68B. Leibe
[ ]z z x A I | 0 X b Ax b
QXX ~
z z x b Ax b
( ) ( )z z x b x Ax b x
0][T Axbx
AbF ][
[ ] , [ | ]z z x I | 0 X x Ab X
b: epipole (FTb = 0), A = –[b×]F
Slide adapted from Svetlana Lazebnik
0 ( )z Ax b x
F&P sec. 13.3.1
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Projective SfM: Two-Camera Case
• Decomposing the Fundamental Matrix
This means that if we can compute the fundamental matrix between
two cameras, we can directly estimate the two projection matrices
from F.
Once we have the projection matrices, we can compute the 3D
position of any point X by triangulation.
• How can we obtain both kinds of information at the same
time?
69B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Projective Factorization
• If we knew the depths z, we could factorize D to estimate
M and S.
• If we knew M and S, we could solve for z.
• Solution: iterative approach
(alternate between above two steps).70
B. Leibe
n
mmnmnmmmm
nn
nn
zzz
zzz
zzz
XXX
P
P
P
xxx
xxx
xxx
D
21
2
1
2211
2222222121
1112121111
Cameras
(3m × 4)
Points (4 × n)
D = MS has rank 4
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Sequential Structure from Motion
• Initialize motion from two images
using fundamental matrix
• Initialize structure
• For each additional view:
Determine projection matrix
of new camera using all the
known 3D points that are
visible in its image –
calibration
71B. Leibe
Ca
me
ras
Points
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Sequential Structure from Motion
• Initialize motion from two images
using fundamental matrix
• Initialize structure
• For each additional view:
Determine projection matrix
of new camera using all the
known 3D points that are
visible in its image –
calibration
Refine and extend structure:
compute new 3D points,
re-optimize existing points
that are also seen by this camera –
triangulation
72B. Leibe
Ca
me
ras
Points
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Sequential Structure from Motion
• Initialize motion from two images
using fundamental matrix
• Initialize structure
• For each additional view:
Determine projection matrix
of new camera using all the
known 3D points that are
visible in its image –
calibration
Refine and extend structure:
compute new 3D points,
re-optimize existing points
that are also seen by this camera –
triangulation
• Refine structure and motion: bundle adjustment73
B. Leibe
Ca
me
ras
Points
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Bundle Adjustment
• Non-linear method for refining structure and motion
• Minimizing mean-square reprojection error
74B. Leibe
2
1 1
,),(
m
i
n
j
jiijDE XPxXP
x1j
x2j
x3
j
Xj
P1
P2
P3
P1Xj
P2Xj
P3Xj
Slide credit: Svetlana Lazebnik
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
Bundle Adjustment
• Idea
Seek the Maximum Likelihood (ML) solution assuming the
measurement noise is Gaussian.
It involves adjusting the bundle of rays between each camera center
and the set of 3D points.
Bundle adjustment should generally be used as the final step
of any multi-view reconstruction algorithm.
– Considerably improves the results.
– Allows assignment of individual covariances to each measurement.
• However…
It needs a good initialization.
It can become an extremely large minimization problem.
• Very efficient algorithms available.
75B. Leibe
Perc
ep
tual
an
d S
en
so
ry A
ug
me
nte
d C
om
pu
tin
gC
om
pute
r V
isio
n S
um
mer‘
19
References and Further Reading
• Background information on camera models and calibration
algorithms can be found in Chapters 6 and 7 of
• Also recommended: Chapter 9 of the same book on
Epipolar geometry and the Fundamental Matrix and
Chapter 11.1-11.6 on automatic computation of F.
76B. Leibe
R. Hartley, A. Zisserman
Multiple View Geometry in Computer Vision
2nd Ed., Cambridge Univ. Press, 2004
Recommended