Embed Size (px)
Citation preview
Lecture 5 -Silvio Savarese 24-Jan-18
ProfessorSilvioSavareseComputationalVisionandGeometryLab
Lecture5Epipolar Geometry
Lecture 5 -Silvio Savarese 24-Jan-18
• Whyisstereouseful?• Epipolar constraints• Essentialandfundamentalmatrix• EstimatingF• Examples
Lecture5Epipolar Geometry
Reading:[AZ] Chapter:4“Estimation– 2DperspectivetransformationsChapter:9“Epipolar GeometryandtheFundamentalMatrixTransformation”Chapter:11“ComputationoftheFundamentalMatrixF”
[FP] Chapter:7“Stereopsis”Chapter:8“StructurefromMotion”
Pinhole perspective projectionRecovering structure from a single view
C
Ow
Pp
Calibration rigScene
Camera K
From calibration rig
From points and lines at infinity + orthogonal lines and planes ® structure of the scene, K
® location/pose of the rig, K
Knowledge about scene (point correspondences, geometry of lines & planes, etc…
Pinhole perspective projectionRecovering structure from a single view
C
Ow
Pp
Calibration rigScene
Camera K
Why is it so difficult?
Intrinsic ambiguity of the mapping from 3D to image (2D)
Recovering structure from a single view
Intrinsic ambiguity of the mapping from 3D to image (2D)
Courtesy slide S. Lazebnik
Two eyes help!
O1 O2
pp’
Twoeyeshelp!
Thisiscalledtriangulation
K’ =knownK =known
R, T
P = l × "lP
l 'l[Eq. 1]
• FindP*thatminimizesd(p,M P*)+ d(p ',M 'P*)
O1 O2
pp’
P
Triangulation
[Eq. 2]
MP*M’P*
P*
Multi(stereo)-viewgeometry
• Camerageometry:Givencorrespondingpointsintwoimages,findcameramatrices,positionandpose.
• Scenegeometry:Findcoordinatesof3Dpointfromitsprojectioninto2ormultipleimages.
• Correspondence:Givenapointp inoneimage,howcanIfindthecorrespondingpointp’ inanotherone?
• Epipolar Plane • Epipoles e, e’
• Epipolar Lines• Baseline
Epipolar geometry
O1 O2
p’
P
p
e e’
= intersections of baseline with image planes = projections of the other camera center
Example of epipolar lines
O1 O2
P
e’p p’e
Example: Parallel image planes
• Baseline intersects the image plane at infinity• Epipoles are at infinity• Epipolar lines are parallel to u axis
u u
v v
Example:ParallelImagePlanes
Example: Forward translation
• The epipoles have same position in both images• Epipole called FOE (focus of expansion)
O2
e
e’
O1
P
- Two views of the same object - Given a point on left image, how can I find the corresponding point on right image?
Epipolar Constraint
pWhere is p’?
Epipolar geometry
O1 O2
P
p p’
Epipolar line 2
Epipolar Constraint
p p’Epipolar line 2
M P =uv1
!
"
###
$
%
&&&= p
O1 O2
pp’
P
R, T
Epipolar Constraint
!M P =!u!v1
"
#
$$$
%
&
'''= p '[Eq. 3] [Eq. 4]
M = K I 0!"
#$
M ' = K ' RT −RTT"#$
%&'
u
v
M = K I 0!"
#$
O1 O2
pp’
P
R, T
Epipolar Constraint
K = K’ are known(canonical cameras)
[ ]0IM =
Kcanonical =1 0 00 1 00 0 1
!
"
###
$
%
&&&
M ' = K ' RT −RTT"#$
%&'
M ' = RT −RTT"#$
%&'[Eq. 5] [Eq. 6]
T
O1
O2
T =O2 inthecamera1referencesystemR istherotationmatrixsuchthatavectorp’ inthecamera2isequaltoRp’ incamera1.
ThecamerasarerelatedbyR,T
p’
Camera 1
Camera 2
O1 O2
pp’
P
R, T
[ ] 0)( =¢´×® pRTpT
Epipolar Constraint
is perpendicular to epipolar planeT × ((R "p )+T ) = T × (R "p )= R !p +Tp’ in first camera reference system is
[Eq. 7]
baba ][0
00
´=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
--
-=´
z
y
x
xy
xz
yz
bbb
aaaaaa
Cross product as matrix multiplication
O1 O2
pp’
P
R, T
[ ] 0)pR(TpT =¢´× [ ] 0pRTpT =¢××® ´
E = Essential matrix(Longuet-Higgins, 1981)
Epipolar Constraint
[Eq. 8] [Eq. 9]
• l = E p’ is the epipolar line associated with p’ • l’ = ET p is the epipolar line associated with p • E e’ = 0 and ET e = 0 • E is 3x3 matrix; 5 DOF• E is singular (rank two)
Epipolar Constraint
O1 O2
p’
P
p
e e’l l’
pT ⋅ E p ' = 0[Eq. 10]
Epipolar Constraint
O1 O2
pp’
P
[ ]0IKM =
R, T
M ' = K ' RT −RTT"#$
%&'
pc = K−1 p !pc = K '
−1 !p[Eq. 11] [Eq. 12]
O1 O2
pp’
P
Epipolar Constraint
pTc ⋅ T×[ ] ⋅R #pc = 0 [ ] 0pKRT)pK( 1T1 =¢¢××® -´
-
[ ] 0pKRTKp 1TT =¢¢×× -´
- 0pFpT =¢® [Eq. 13]
[Eq.9]
pc = K−1 p
!pc = K '−1 !p
[Eq. 11]
[Eq. 12]
O1 O2
pp’
P
Epipolar Constraint
0pFpT =¢
F = Fundamental Matrix(Faugeras and Luong, 1992)
[ ] 1-´
- ¢××= KRTKF T[Eq. 13]
[Eq. 14]
Epipolar Constraint
O1 O2
p’
P
p
e e’
• l = F p’ is the epipolar line associated with p’ • l’= FT p is the epipolar line associated with p • F e’ = 0 and FT e = 0• F is 3x3 matrix; 7 DOF • F is singular (rank two)
pT ⋅ F p ' = 0
l’l
Why F is useful?
- Suppose F is known- No additional information about the scene and camera is given- Given a point on left image, we can compute the corresponding epipolar line in the second image
l’ = FT pp p’
WhyFisuseful?
• Fcapturesinformationabouttheepipolargeometryof2views+cameraparameters
• MOREIMPORTANTLY: Fgivesconstraintsonhowthescenechangesunderviewpointtransformation(withoutreconstructingthescene!)
• Powerfultoolin:• 3Dreconstruction• Multi-viewobject/scenematching
O1 O2
pp’
P
Estimating FThe Eight-Point Algorithm
0pFpT =¢
(Hartley, 1995)(Longuet-Higgins, 1981)
0pFpT =¢
Estimating F
Let’s take 8 corresponding points
( )11 12 13
21 22 23
31 32 33
', ,1 ' 0
1
F F F uu v F F F v
F F F
æ öæ öç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷è øè ø
( )
11
12
13
21
22
23
31
32
33
', ', , ', ', , ', ',1 0
FFFF
uu uv u vu vv v u v FFFFF
æ öç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷è ø
[Eq. 13]
[Eq. 14]
!p =!u!v1
"
#
$$$
%
&
'''
p =uv1
!
"
###
$
%
&&&
Estimating F
Estimating F
uiu 'i ,uiv 'i ,ui ,viu 'i ,viv 'i ,vi ,u 'i ,vi ',1( )
F11F12F13F21F22F23F31F32F33
!
"
#############
$
%
&&&&&&&&&&&&&
= 0 [Eq. 14]
Estimating F
Lsq. solution by SVD!• Rank 8 A non-zero solution exists (unique)
• Homogeneous system
• If N>8 F̂
f
1=f
W 0=f
W
1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4
5 5 5 5 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 6 6 6 6
7 7 7 7 7 7
' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' '
u u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v
11
12
13
21
22
23
317 7 7 7 7 7
328 8 8 8 8 8 8 8 8 8 8 8
33
0
' ' ' ' 1' ' ' ' ' ' 1
FFFFFFF
u v v v u vF
u u u v u v u v v v u vF
æ öæ öç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷è øç ÷
è ø
[Eqs. 15]
0pF̂pT =¢and estimated F may have full rank (det(F) ≠0)
0F̂F =-Find F that minimizes
Subject to det(F)=0
But remember: fundamental matrix is Rank2
Frobenius norm (*)
SVD (again!) can be used to solve this problem
^ ^
(*) Sq. root of the sum of squares of all entries
satisfies:F̂
0F̂F =-Find F that minimizes
Subject to det(F)=0
Frobenius norm (*)
F =Us1 0 00 s2 00 0 0
!
"
####
$
%
&&&&
VT
Us1 0 00 s2 00 0 s3
!
"
####
$
%
&&&&
VT = SVD(F̂)
Where:
[HZ] pag 281, chapter 11, “Computation of F”
Data courtesy of R. Mohr and B. Boufama.
Mean errors:10.0pixel9.1pixel
Problems with the 8-Point Algorithm
- Recall the structure of W:- do we see any potential (numerical) issue?
F1=f,0=fW Lsq solution
by SVD
1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4
5 5 5 5 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 6 6 6 6
7 7 7 7 7 7
' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' '
u u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v
11
12
13
21
22
23
317 7 7 7 7 7
328 8 8 8 8 8 8 8 8 8 8 8
33
0
' ' ' ' 1' ' ' ' ' ' 1
FFFFFFF
u v v v u vF
u u u v u v u v v v u vF
æ öæ öç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷è øç ÷
è ø
Problems with the 8-Point Algorithm
• Highlyun-balanced(notwellconditioned)• ValuesofWmusthavesimilarmagnitude• ThiscreatesproblemsduringtheSVDdecomposition
0=Wf
HZpag
108
Normalization
IDEA: Transform image coordinates such that the matrix W becomes better conditioned (pre-conditioning)
For each image, apply a transformation T (translation and scaling) acting on image coordinates such that:
• Origin = centroid of image points• Mean square distance of the image points from origin is ~2 pixels
Coordinate system of the image before applying T
Coordinate system of the image after applying T
T
Example of normalization
• Origin = centroid of image points• Mean square distance of the image points from origin is ~2 pixels
Normalization
ii pTq = ii pTq ¢¢=¢
1. Normalize coordinates in images 1 and 2:
2. Use the eight-point algorithm to compute from thecorresponding points q and q’ .
1. Enforce the rank-2 constraint:
2. De-normalize Fq:
i i
ii pTq = ii pTq ¢¢=¢
0qFq qT =¢
The Normalized Eight-Point Algorithm
qF®
'TFTF qT= 0)Fdet( q =
0. Compute T and T’ for image 1 and 2, respectively
F̂q
such that:
With
nor
mal
izat
ion
With
out n
orm
aliz
atio
nMean errors:10.0pixel9.1pixel
Mean errors:1.0pixel0.9pixel
http://danielwedge.com/fmatrix/
The Fundamental Matrix Song
Lecture 5 -Silvio Savarese 24-Jan-18
Nextlecture:Stereosystems
O1 O2
P
e’p p’
e
Example: Parallel image planes
E=?K1=K2 = knownHint :
x
y
z
x parallel to O1O2
R = I T = (T, 0, 0)
u
v
E =
0 −Tz TyTz 0 −Tx−Ty Tx 0
"
#
$$$$
%
&
''''
R =0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
Essentialmatrixforparallelimages
T = [ T 0 0 ]R = I
[ ] RTE ×= ´
[Eq. 20]
O1
P
p p’
Example: Parallel image planes
x
y
z
horizontal!What are the directions of epipolar lines?
l = E p ' =0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
u 'v '1
"
#
$$$
%
&
'''=
0−TT v '
"
#
$$$
%
&
'''
l l’
u
v
e’e
u’
v’
Example: Parallel image planes
How are pand p’ related?
pT ⋅ E p ' = 0
P
e’p p’
e
y
z
xO1
u
v
u’
v’
Example: Parallel image planes
( ) ( )T
0 0 0 0p E p 0 1 0 0 0 1 0
0 0 1
uu v T v u v T Tv Tv
T Tv
¢é ù æ ö æ öç ÷ ç ÷ê ú¢ ¢ ¢= Þ - = Þ - = Þ =ç ÷ ç ÷ê úç ÷ ç ÷¢ê úë û è ø è ø
How are p and p’ related?
P
e’p p’
e
y
z
O1x
⇒ v = "v
u
v
u’
v’
Example: Parallel image planes
Rectification: making two images “parallel” Why it is useful? • Epipolar constraint ® v = v’
• New views can be synthesized by linear interpolation
P
e’p p’
e
y
z
O1x
u
v
u’
v’
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
Rectification
u
u
u
v
v
v
î2
î1
îS
I1
I2
Is
P
C2
Cs
C1
H1
H2
From its reflection!
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
