View
225
Download
0
Category
Preview:
Citation preview
7/31/2019 Lecture 9h
1/13
COMPUTER VISION: TWO-V IE W GEOMETRY
IIT Kharagpur
Computer Science and Engineering,
Indian Institute of Technology
Kharagpur.
1 77
Two View Geometry
Epipolar Geometry: is the intrinsic projective geometry betwe
two views.
Fundamental Matrix: F is a 3 3 matrix of rank 2.
Internal parameters of cameras
Relative pose
Intrinsic Projective Geo
x image of X on image 1 x image of X on image 2
xTFx = 0
ipolar Geometry Two-View Geometry
OMETRY COMPONENTS:Baseline: is the line joining the two camera centres.
Image planes of the two cameras P P.
Pencil of planes having baseline as the axis.
The 3D point X which gets projected as x and x on the two
cameras
Plane passing through x, x and the 3D point X.
3 77
Epipolar Geometry Two-View Geom
GEOMETRY COMPONENTS:
5 77
Epipolar Geometry Two-View Geom
GEOMETRY COMPONENTS:
Rays back projected from x and x are coplanar lie on ) and inte
at X
7/31/2019 Lecture 9h
2/13
ipolar Geometry Two-View Geometry
OMETRY COMPONENTS:
x x are the corresponding points.
Plane : can be specified by the baseline and the ray
back-projected from x.
The line of intersection of with the second image is l
l is the epipolar line corresponding to the point x.
The corresponding point x lies on this epipolar line l.
7 77
Epipolar Geometry Two-View Geom
GEOMETRY COMPONENTS:
ipolar Geometry Two-View Geometry
OMETRY COMPONENTS:
9 77
Epipolar Geometry Two-View Geom
GEOMETRY COMPONENTS:Epipole: is the pointof intersection of the line joining the cam
centres the baseline) with the image plane.
Epipole: is the imageof the camera centre of the other view.
Epipolar plane: is the plane containing the baseline. There is
one-parameter family a pencil) of epipolar planes.
Epipolar line: is the line of intersection of the epipolar plane w
the image plane.
ALL EPIPOLAR LINES INTERSECT AT THE EPIPOLE .
ipolar Geometry Two-View Geometry
OMETRY COMPONENTS:
11 77
Epipolar Geometry Two-View Geom
GEOMETRY COMPONENTS:
7/31/2019 Lecture 9h
3/13
ipolar Geometry Two-View Geometry
OMETRY COMPONENTS:
tion parallel to the image plane
13 77
Fundamental Matrix Epipolar Geom
FUNDAMENTAL MATRIX: F is the algebraic representation of the
epipolar geometry.
Point to line mapping: A point x has a corresponding epipola
l in the second image.
x l
This mapping is the fundamental matrix F. It is a projective
mapping from points to lines.
The corresponding point x which matches to x must lie on l.
ndamental Matrix Epipolar Geometry
GEOMETRIC DERIVATION :
Consider a plane not passing through either of the two camera
centres.
The ray back-projected from point x intersects plane at point X.
The point X gets projected to point x in the second image.
The projected point x lies on the epipolar line l.
15 77
Fundamental Matrix Epipolar Geom
GEOMETRIC DERIVATION :
The set of all points xi in the first image and the correspondin
points xi in the second image are projectively equivalent, sin
they are each projectively equivalent to the planar point set X
There is a 2-D homography H mapping each xi to x
i
H is the transfer mapping from image 1 to image 2 via plane
ndamental Matrix Epipolar Geometry
GEOMETRIC DERIVATION :
Given the point x the epipolar line l passes through x and
epipole e
l = [e]Hx = Fx
Fundamental matrix F = [e]H
17 77
Fundamental Matrix Epipolar Geom
Cross product matrix: e = e1 e2 e3)
[e] =
0 e3 e2e3 0 e1
e2 e1 0
Any skew symmetric 3 3 matrix may be written in the form [
for a suitable vector e.
Matrix [e] is singular, and e is its null vector right or left).
The cross product of two 3-vectors a b
a b = [a] b = aT [b]
Fundamental matrix F = [e]H
7/31/2019 Lecture 9h
4/13
ndamental Matrix Epipolar Geometry
GEOMETRIC DERIVATION :
Fundamental matrix F = [e]H
[e] has rank 2, H has rank 3, F is a matrix of rank 2.
F is a mapping from IP2
onto a IP1
.F is a point map. It maps x l.
The pencil of epipolar lines through e forms IP1.
19 77
Fundamental Matrix Epipolar Geom
ALGEBRAIC DERIVATION:
The ray back-projected from x by P is obtained by solving P X
The ray is parametrized by the scalar .
X) = Px + C
P is the pseudo inverse of P, i.e. PP = I , C is the cameracentre given by PC = 0
Two points on the ray are Px at = 0) and camera centre C = ).
These two points are imaged by the second camera P
atPx PPx
C PC
ndamental Matrix Epipolar Geometry
GEBRAIC DERIVATION:The epipolar line joins these two projected points:
l = PC) PPx)
The epipole e = PC, we have l = e PP)x = Fx
F = [e]PP
Comparing this with the previously derived formula F = [e]H we
have H = PP.
21 77
Fundamental Matrix Epipolar Geom
IN TERMS OF CAMERA MATRICES:
P = K[ I 0] P = K[R t]
P =
K1
0
C =
0
1
F = [PC] PP
= [Kt] KRK1
= K [t] RK1
= KRRt
K1
= KRKKRt
Using result:
[t] M = M
M1t
= M
M1t
up to s
t is any vector
M non-singular matrix
M = detM)M
ndamental Matrix Epipolar Geometry
TERMS OF CAMERA MATRICES:
poles are given by images of the camera centres:
e= P
R
t
1
= KR
Tt e
= P
01
= K
t
F = [PC] PP
= [Kt] KRK1
= K [t] RK1
= KR
Rt
K1
= KRKKRt
F = [PC] PP
= [e]KRK1
= K [t] RK1
= KRRt
K1
= KRK [e]
23 77
Fundamental Matrix Epipolar Geom
CORRESPONDENCE CONDITION:
The epipolar line l = Fx. Since point x lies on this line, we havex
l = 0. This gives xFx = 0.
The fundamental matrix satisfies the condition that for any
pair of corresponding points x x in the two images xFx = 0F can be characterized without reference to camera matrix, o
terms of x x) point correspondences.
F can be computed from image correspondences.
At least 7 point correspondences are required to compute F.
7/31/2019 Lecture 9h
5/13
ndamental Matrix Epipolar Geometry
OPERTIES:
F is unique for two views.
F is 3 3 homogeneous matrix with rank 2.
If F is the fundamental matrix of the pair of cameras P P), then
F is the fundamental matrix of the pair in opposite order P P).
Epipolar line l = Fx contains the epipole e.
eFx) = eF)x = 0 for all x eF = 0
Epipolar line l = F
x
contains the epipole e.
eFx) = eF)x = 0 for all x Fe = 0
25 77
Fundamental Matrix Epipolar Geom
PROPERTIES:
F has 7 degrees of freedom. A 3 3 homogeneous matrix ha
independent ratios. F also satisfies the constraint detF = 0 w
removes one degree of freedom.
F is a correlation: a projective map taking point to a line. l =
Any point x on l is mapped to the same epipolar line l. This
means there is no inverse mapping, and F is not of full rank.
F is not invertible. Hence F is not a proper correlation.
ndamental Matrix Epipolar Geometry
POLAR LIN E HOMOGRAPHY:The set of epipolar lines in each of the images forms a pencil of
lines passing through the epipoles.
Such pencil of lines may be considered as a 1-D projective space.
27 77
Fundamental Matrix Epipolar Geom
EPIPOLAR LIN E HOMOGRAPHY:The set of epipolar lines in each of the images forms a pencil
lines passing through the epipoles.
Such pencil of lines may be considered as a 1-D projective sp
The corresponding epipolar lines are perspectively related.
There is a homography between the pencil of lines centered a
the 1st view and the pencil of lines centered at e in the 2nd vie
A homography between two such 1-D projective spaces has 3
degrees of freedom.
Degrees of freedom
for F
2 for e
2 for e
3 for epipolar line homography
= 7
ndamental Matrix Epipolar Geometry
POLAR LIN E HOMOGRAPHY:
Suppose l and l are corresponding epipolar lines.
Suppose k is any line passing through epipole e.
The point of intersection of two lines l and k is x = [k]l = k l.This point lies on the epipolar line l.
The epipolar line corresponding to x is l = Fx = F[k]l
Likewise we have l = Fx = F[k]l
29 77
Fundamental Matrix Epipolar Geom
Next
What is F for special special motions between twoviews.
7/31/2019 Lecture 9h
6/13
ndamental Matrix Epipolar Geometry
ECIAL MOTIONS BETWEEN VIEWS:
Pure translation between the two views.
Pure planar motion between the two views: the translation t is
orthogonal to the direction of rotation axis a
We assume there is no change in the internal parameters of the
camera viewing the scene.
31 77
Fundamental Matrix Epipolar Geom
SPECIAL MOTIONS BETWEEN VIEWS:
Pure translation
The camera undergoes a translation t.
Equivalently, the camera is assumed stationary and the world
points undergo translation t.
Points in 3-space move on straight lines parallel to t.
On the image plane these parallel lines appear to intersect at
vanishing point v in the direction of t.
Both the views have a common epipole v.
The imaged parallel lines are the epipolar lines.
33 77
Fundamental Matrix Epipolar Geom
SPECIAL MOTIONS BETWEEN VIEWS:
Pure translation
Camera translating along principal axis
mera translating along principal axis
35 77
Fundamental Matrix Epipolar Geom
Pure translation
The two cameras can be chosen as:
P = K[ I 0] P = K[ I t]
Given that the camera coordinate system is aligned with the world
coordinate system and the camera is looking at the Z axis.
7/31/2019 Lecture 9h
7/13
jection on the 1st camera Pure Translation
e inhomogeneous space point X gets projected to the
homogeneous) image point x.
X =
X
Y
Z
1
x =
x
y
1
Zx = PX = K[ I 0] X
ZK1x =
X
Y
Z
37 77
Projection on the 2nd camera Pure Trans
X =
X
Y
Z
1
x =
x
y
1
Zx = PX = K[ I t] X
Zx = [K Kt] X Zx = K
X
Y
Z
+ Kt
Zx = KZK1x) + Kt Zx = ZKK1x) + Kt
x = x + Kt/Z
The epipoles e e are the same in both the views and they are
vanishing points of the imaged parallel lines in the direction t.
e translation Fundamental Matrix
The situation whenthe object translates
by t is the same as
camera translating
by t
The epipoles e e are
the same in both the
views and they are
the vanishing points
of the imaged
parallel lines in the
direction t.
39 77
Pure translation Fundamental M
SOME OBSERVATIONS:
x = x + Kt/Z
The extent of motion depends on the magnitude of translation
and the inverse depth Z.
In the case of pure translation:
P = K[ I 0] P = K[ I t]
F = [PC] PP = [e]K
RK1 = [e]KK1 = [e]
F = [e]
e translation Fundamental Matrix
ME OBSERVATIONS: x = x + Kt/Z F = [e]
For camera translating parallel to x axis:
e =
1
0
0
F =
0 0 0
0 0 1
0 1 0
xFx = 0 and thus y = y
The fundamental matrix has 2 dofs which correspond to the
position of the epipole.
l = Fx = [e]x and x
[e]x = 0. Hence x lies on line [e]x = l
.
Implying that x x e = e are collinear.This collinearity property is termed as auto-epipolarand does not
hold for general motion.
y x[e]x = 0 ? Verify: x
0 0 0
0 0 1
0 1 0
x
41 77
Fundamental Matrix Epipolar Geom
SPECIAL MOTIONS BETWEEN VIEWS:
General Motion
We are given two arbitrary views:
Correction 1: Rotate the camera used for the first image so th
is aligned with the second camera. This rotation may be simu
by applying a projective transformation to the first image.
Correction 2: Apply further correction can be applied to the fir
image to account for any difference in the calibration matrices
K K of the two cameras.
The result of the two corrections is a projective transformation
the first image.
Now the two cameras are related by a pure translation.
7/31/2019 Lecture 9h
8/13
neral Motion Fundamental Matrix
43 77
General Motion Fundamental M
After applying the two corrections we have F as the fundamen
matrix between the corrected first image x and the second im
i.e. x x
F = [e] x = x xFx = 0
x[e]x = 0
Hence the fundamental matrix corresponding to the initial poi
correspondences x x is
F = [e]
trieving the camera matrices Fundamental matrix
The fundamental matrix F can be used to determine the cameramatrices of the two views.
The relations l = Fx and xFx = 0 are projectiverelationships.They make use of the projective coordinates in the image.
Euclidean measurements such as angles are not used.
If the images undergo a projective transformation,
x = x x = Hx
there is a corresponding map
l
= Fx F = H FH1
Fx = Hx)Fx) = xHFx = xFx
HF = F hence F = HFH1
45 77
Retrieving the camera matrices Fundamental m
The fundamental matrix F only depends on the projectiveproperties of the cameras P P.
F does not depend on the choice of the world coordinate fram
Rotation of world coordinates changes P P and not F.
If the 3-space undergoes a projective transformation using a
4 4 H1)
X = H1X
then the fundamental matrices corresponding to the pairs of
cameras P P) and P P) are the same.
PX = P)H1X) PX = P)H1X)
F = [PC] PP = P
H1C P
)H1P)
Fundamental matrix remains unchanged.
trieving the camera matrices Fundamental matrix
A pair of cameras can uniquely determine F.
A fundamental matrix determines the two cameras at best up to a
right multiplication by a 3D projective transformation.
en two camera matrices P P), it is always possible to identify amography such that P P) will form a canonical camera pair.
P = [ I 0] P = [M m]
e fundamental matrix corresponding to a pair of camera matrices
= [ I 0] P = [M m] is equal to F = [m] M
call
F = [e]PP
47 77
Retrieving the camera matrices Fundamental m
A fundamental matrix determines the two cameras at best up
right multiplication by a 3D projective transformation.
It will now be shown that if two pairs of camera matrices P P
and P
P
) have the same fundamental matrix F, then the pacamera matrices are related up to a right multiplication by aprojective transformation .
There always exists a non-singular 4 4 matrix such that
P = P and P
= P.
We can assume that the two pairs of cameras P P) and P P)
provided in the canonical form.
P = [ I 0] P = [A a] P = [ I 0] P
= [A a]
7/31/2019 Lecture 9h
9/13
trieving the camera matrices Fundamental matrix
P = [ I 0] P = [A a] P = [ I 0] P
= [A a]
F = [a]A = [a]A
have
aF = a[a]A = 0 and a
F = a[a]A = 0
ce F is rank 2, it has a 1-D null space. Hence a = ka
ce [a]A = [a]A,
[a
]A = k[a
]A [
a]k
A A) =
0
re k is any constant.
49 77
Retrieving the camera matrices Fundamental m
[a]A = k[a]A [a]kA A) = 0
Now, [a]kA A) is a 3 3 matrix.
If we substitute kA A) by a 3 3 matrix of form av then wethat [a]av
= 0
Hence kA A) = av where v is any 3-vector.Thus,
A = k1A + av)
trieving the camera matrices Fundamental matrix
en the two substitutions: a = ka and A = k1A + av)e camera matrices now become:
P = [ I 0]
P = [A a]
P = [ I 0]
P
= [A a]
P = [ I 0]
P
= [k1A + av ka]
here any which will now give
P = P and P = P
51 77
Retrieving the camera matrices Fundamental m
P = [ I 0]
P = [A a]
P = [ I 0]
P
= [k1A + av ka]
We choose =
k1 I 0
k1v k
P = [ I 0]
P = [A a]
P = k1P
P = [A a]
= [k1A + av) ka
= [A a] = P
Thus we have P = P and P = P
grees of Freedom Fundamental matrix
Each of the two camera matrices P P) have 11 degrees offreedom. Total: 22 dofs.
Specifying a projective world frame requires 15 dofs.
22-15 = 7.The fundamental matrix F has 7 degrees of freedom.
53 77
Computing camera matrices Fundamental m
F can determine the camera pair up to a projective transforma
of 3-space.
If any matrix, say M is skew symmetric, we have xMx = 0
Consider the composite matrix P
FP
we have XPFPX = 0 since xFx = 0
A non-zero matrix F is the fundamental matrix corresponding
pair of camera matrices P and P if and only if PFP is ske
symmetric.
7/31/2019 Lecture 9h
10/13
mputing camera matrices Fundamental matrix
Consider F to be the given Fundamental matrix.
Consider a pair of 3 4 matrices
P = [ I 0] P = [SF e] such that e
F = 0
Assume that P P have rank 3.
We need to verify if P P are indeed the camera matrices
corresponding to F. Following conditions need to be checked:
We need to verify that PFP is skew symmetric.
We need to choose a skew symmetric matrix S such that P has
rank 3.
55 77
Computing camera matrices Fundamental m
Check PFP is skew symmetric
[SF e] F [ I 0] =
FS
F 0
eF 0
=
FS
F 0
0 0
This is indeed skew symmetric if S is skew symmetric.
Choosing a suitablematrix S
S is skew symmetric. In terms of its null vector S = [s].
P = [SF e] = [ [s]F e]
mputing camera matrices Fundamental matrix
oosing a suitablematrix S
oose S = [s].P = [SF e] = [ [s]F e
]
need to verify that P = [ [s]F e] has rank 3. [ [s]F e] will have rank 3 provided se 0. Why?
57 77
Computing camera matrices Fundamental m
Choosing a suitablematrix S [ [s]F e] will have rank 3 provided se 0.[s]F has rank 2. The column space of [s]F is spanned by
cross product of s with the columns of F, and equals the pla
perpendicular to s.
If se 0 then e is not perpendicular to s, and hence it does
lie in this plane.
Thus [ [s]F e] has rank 3.
A suitable choice for s can be e since we have ee 0.
Thus we take S = [s] = [e]
mputing camera matrices Fundamental matrix
The camera matrices corresponding to the Fundamental matrix F
can be chosen as:
P = [ I 0] P = [[e]F e] such that e
F = 0
The left 3 3 sub-matrix of P i.e. [e]F has rank 2. Thiscorresponds to a camera with centre at .
59 77
Computing camera matrices Fundamental m
FAMILY OF CAMERAS WHICH HAVE THE SAME F
We can identify a family of cameras:
P = [ I 0] P = [[e]F e
v ke]
vk are parameters
v is any 3-vector and k is a non-zero scalar.
7/31/2019 Lecture 9h
11/13
Fundamental matrix
xt
After Fundamental matrix: .....
The Essential Matrix
61 77
Essential Matrix A special case
The Essential matrix is a special case of fundamental matrix f
pair of normalized cameras.
It has fewer degrees of freedom and additional properties.
It makes use of normalized image coordinates.
Normalized Coordinates
P = K[R t] x = PX x = K1x = [R t]X
The matrix K1P = [R t] is the normalized camera matrix.
We have the normalized camera pair: P = [ I 0] and P = [R
sential Matrix A special case of F
P = K[R t] x = PX x = K1x = [R t]X
The Fundamental matrix corresponding to the normalized camera
pair P = [ I 0] and P = [R t] is called as the Essential matrix:
E = [t]R = R
Rt
We have
xEx = 0
For the corresponding points x x, the normalized image
coordinates are x x
63 77
Essential Matrix A special case
xEx = 0
Substituting for x = K1x and x = K1x gives
xKEK1x = 0 F = KEK1 E = KFK
Properties of Essential Matrix
Has 5 dofs: 3 for R and 3 for t and -1 for overall scale.
A 3 3 matrix is an essential matrix if and only if two of itssingular values are equal and third is zero.
sential Matrix A special case of F
xt
We show that E has TWO singular values whichare equal and the third is zero
65 77
Essential Matrix A special case
Consider decomposition of E as
E = SR = [t]R
S is a skew-symmetric matrix which can be decomposed as
S = kUZU where U is orthogonal
Matrix Z is a block diagonal matrix of the form
Z =
0 1 0
1 0 0
0 0 0
as a matrix product =
1 0 0
0 1 0
0 0 0
0 1
1 0
0 0
7/31/2019 Lecture 9h
12/13
sential Matrix A special case of F
nsider decomposition of E as
E = SR = kUZU R = [t]R
s skew symmetric and
Z = diag1,1,0) W where W =
0 1 0
1 0 0
0 0 1
urns out to be an orthogonal matrix. S = UZU
S = U diag1,1,0) W U E = SR = U diag1,1,0) WUR)
67 77
Essential Matrix A special case
Consider decomposition of E as up to scale)
E = SR = UZU R = [t]R
SVD decomposition of
E = UDV = U diag1,1,0) WUR) where V = WUR)
Thus E has two singular values which are equal.
SVD of E is not unique. Alternate SVDs are given as:
E = UdiagR22 1)) diag1,1,0) diagR22 1)V)
where R22 is any rotation matrix.
sential Matrix A special case of F
xt
How to extract cameras from E
69 77
Extraction of Cameras Essential ma
Consider decomposition of E as up to scale)
E = SR = [t]R
The two cameras can be chosen as: P = [ I 0] P = [R t
The vector t has to be chosen such that S t = 0.
SVD of E is not unique. Alternate SVDs are given as:
E = UDV = U diag1,1,0) WUR) where V = WUR
W =
0 1 0
1 0 0
0 0 1
R = UW
V
raction of Cameras Essential matrix E
nsider decomposition of E as
E = SR = [t]R
e two cameras can be chosen as: P = [ I 0] P
= [R t]The vector t has to be chosen such that S t = 0.
We choose
t = U0 0 1) = u
It can be verified that St = 0
St = U diag1,1,0) W U) t
= U diag1,1,0) W U) u
= 0
71 77
Extraction of Cameras Essential ma
It can be verified that St = 0
St = U diag1,1,0) W U) t
= U diag1,1,0) W U) u
a1 a2 a3b1 b2 b3c1 c2 c3
1 0 0
0 1 0
0 0 0
0 1 0
1 0 0
0 0 1
a1 b1 c1a2 b2 c2a3 b3 c3
a
b
ca1 a2 a3b1 b2 b3c1 c2 c3
0 1 0
1 0 0
0 0 0
a1 b1 c1a2 b2 c2a3 b3 c3
a
b
ca2 a1 0
b2 b1 0
c2 c1 0
a1 b1 c1a2 b2 c2a3 b3 c3
a
b
c
7/31/2019 Lecture 9h
13/13
raction of Cameras Essential matrix Ea2a1 a1a2 a2b1 a1b2 a2c1 a1c2b2a1 b1a2 b2b1 b1b2 b2c1 b1c2c2a1 a2c1 c2b1 c1b2 c2c1 c1c2
a3b3c3
0 a2b1 a1b2 a2c1 a1c2b2a1 b1a2 0 b2c1 b1c2c2a1 a2c1 c2b1 c1b2 0
a3b3c3
a2b1b3 a1b2b3 + a2c1c3 a1c2c3a3b2a1 a3b1a2 + b2c1c3 b1c3c2c2a3a1 a2a3c1 + c2b1b3 c1b2b3
a2b1b3 + c1c3) a1b2b3 + c2c3)b2a3a1 + c1c3) b1a3a2 + c3c2)c2a3a1 + b1b3) c1a2a3 + b2b3)
=
a2a1a3 a1a2a3b2b1b3 b1b2b3c2c1c3 c1c2c3
= 0
73 77
Extraction of Cameras Essential ma
Consider decomposition of E as
E = SR = [t]R
The two cameras can be chosen as: P = [ I 0] P = [R t
We choose
t = U0 0 1) = u R = UWV
There are 4 possible pairs of cameras:
P
= [R t]= [UWV + u]
= [UWV u]
P
= [R t]= [UWV + u]
= [UWV u]
raction of Cameras Essential matrix E
There are 4 possible pairs of cameras:
P = [R t]
= [UWV + u]
= [UWV u]
P = [R t]
= [UWV + u]
= [UWV u]
W and W are related by a rotation througth 1800 about the
base-line.
75 77
mmary
Intrinsic projective geometry of 2-views.
Epipolar geometry
Fundamental matrix
Deriving the fundamental matrix from camera matrices.Deriving the fundamental matrix from point correspondences.
Deriving the camera matrices from the fundamental matrix.
Essential matrix
Deriving the camera matrices from the essential matrix.
77 77
Recommended