Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Lecture 6 -Silvio Savarese 2-Feb-18
ProfessorSilvioSavareseComputationalVisionandGeometryLab
Lecture6StereoSystemsMulti-viewgeometry
Lecture 6 -Silvio Savarese 2-Feb-18
• Stereosystems• Rectification• Correspondenceproblem
• Multi-viewgeometry• TheSFMproblem• AffineSFM
Reading: [AZ] Chapter:9“Epip.Geom.andtheFundam.MatrixTransf.”[AZ] Chapter:18“Nviewcomputationalmethods”[FP] Chapters:7“Stereopsis”[FP] Chapters:8“StructurefromMotion”
Lecture6StereoSystemsMulti-viewgeometry
• Epipolar Plane • Epipoles e, e’
• Epipolar Lines• Baseline
Epipolar geometry
p’
P
p
e e’
= intersections of baseline with image planes = projections of the other camera center
O1 O2
pp’
P
Epipolar Constraint
0=¢pEpT
E = Essential Matrix(Longuet-Higgins, 1981)
[ ] RTE ×= ´
O1 O2
E =
0 −Tz TyTz 0 −Tx−Ty Tx 0
"
#
$$$$
%
&
''''
R
Essentialmatrix
[ ] RTE ×= ´
pp’
P
Epipolar Constraint
0pFpT =¢
F = Fundamental Matrix(Faugeras and Luong, 1992)
[ ] 1-´
- ¢××= KRTKF T
O1 O2
Parallel image planes
• Epipolar lines are horizontal• Epipoles go to infinity• v-coordinates are equal
p =pupv1
!
"
####
$
%
&&&&
p ' =p 'upv1
!
"
####
$
%
&&&&
O1 O2
P
e’p p’
e
x
y
z
u
v
O1 O2
Parallel image planes
E=?K1=K2 = knownHint :
x parallel to O1O2
R = I T = (T, 0, 0)
P
e’p p’
e
x
y
z
u
v
E =
0 −Tz TyTz 0 −Tx−Ty Tx 0
"
#
$$$$
%
&
''''
R =0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
Essentialmatrixforparallelimages
T = [ T 0 0 ]R = I
[ ] RTE ×= ´
O1
P
p p’
Parallel image planes
x
y
z
horizontal!What are the directions of epipolar lines?
l = E p '
l l’
u
v
e’e
u’
v’
=0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
u 'v '1
"
#
$$$
%
&
'''=
0−TT v '
"
#
$$$
%
&
'''
Parallel image planes
How are pand p’ related?
pT ⋅ E p ' = 0
P
e’p p’
e
y
z
xO1
u
v
u’
v’
Parallel image planes
( ) ( )T
0 0 0 0p E p 0 1 0 0 0 1 0
0 0 1
uu v T v u v T Tv Tv
T Tv
¢é ù æ ö æ öç ÷ ç ÷ê ú¢ ¢ ¢= Þ - = Þ - = Þ =ç ÷ ç ÷ê úç ÷ ç ÷¢ê úë û è ø è ø
How are p and p’ related?
P
e’p p’
e
y
z
O1x
⇒ v = "v
u
v
u’
v’
Parallel image planes
Rectification: making two images “parallel” Why it is useful? • Epipolar constraint ® v = v’
• New views can be synthesized by linear interpolation
P
e’p p’
e
y
z
O1x
u
v
u’
v’
Rectification:makingtwoimages“parallel”
H
Courtesy figure S. Lazebnik
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
Rectification
u
u
u
v
v
v
î2
î1
îS
I1
I2
Is
P
C2
Cs
C1
H1
H2
From its reflection!
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
Whyareparallelimagesuseful?
• Makestriangulationeasy• Makesthecorrespondenceproblemeasier
Point triangulation
O O’
P
e’p p’e
x
y
z
u
v
u
v
p =pupv1
!
"
####
$
%
&&&&
p ' =p 'upv1
!
"
####
$
%
&&&&
pu − p 'udisparity = [Eq. 1]∝B ⋅ fz
B
z
Disparity is inversely proportional to depth z!
p'upuf
B
z
O O’
P
pu − p 'u
Disparity is inversely proportional to depth z!
Computing depth
disparity = [Eq. 1]
uu
(0, pv, 1)
∝B ⋅ fz
Disparity mapshttp://vision.middlebury.edu/stereo/
Stereo pair
Disparity map / depth map
pu p’u[Eq. 1]
pu − p 'u ∝B ⋅ fz
Whyareparallelimagesuseful?
• Makestriangulationeasy• Makesthecorrespondenceproblemeasier
Correspondence problem
p’
P
p
O O’
Given a point in 3D, discover corresponding observations in left and right images [also called binocular fusion problem]
p’ belongs to l’ = FT pp belongs to l = F p’
Correspondence problem
p’
P
p
O O’
When images are rectified, this problem is much easier!
•A Cooperative Model (Marr and Poggio, 1976)
•Correlation Methods (1970--)
•Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Correspondence problem
[FP] Chapters: 7
p
Correlation Methods (1970--)
image 1
Where is is ?p '
Image 2
p =uv1
!
"
###
$
%
&&&
p ' =u 'v1
!
"
###
$
%
&&&
p
image 1
Where is is ?p '
Image 2
Correlation Methods (1970--)
100
uu
p =uv1
!
"
###
$
%
&&&
p ' =u 'v1
!
"
###
$
%
&&&
u
p
image 1
Where is is ?p '
Image 2
Correlation Methods (1970--)
10030 170 100
Image 2
130
What’s the problem with this?
u
• PickupawindowW around
image 1
p = (u,v )• Buildvectorw
image 2
W
u u
Window-based correlation
u
vp
• PickupawindowW around
image 1
p = (u,v )• Buildvectorw
Example:W isa3x3windowinredw isa9x1vectorw=[100,100,100,90,100,20,150,150,145]T
• SlidethewindowWalongv=inimage2andcomputew’(u)foreachu
image 2
v
100 100100
90 100 20
150 150145
• ComputethedotproductwT w’(u)foreachuandretainthemaxvalue
uu
Window-based correlation
W
u u ' = u + d = u −1
v
Window-based correlation
image 1
Example:W isa3x3windowinredw isa9x1vectorw=[100,100,100,90,100,20,150,150,145]T
image 2
uu
What’s the problem with this?
u ' = u + d = u −1
v100 100100
90 100 20
150 150145
W
u
Changes of brightness/exposure
Changes in the mean and the variance of intensity values in corresponding windows!
Normalized cross-correlation
Find u that maximizes:(w−w)T ( "w (u)− "w )(w−w) ( "w (u)− "w )
[Eq. 2]
w = w '(u) =mean value within Wlocated at ubar in image 1
mean value within W located at u in image 2
image 1 image 2
uu
u ' = u + d = u +1
v100 100100
90 100 20
150 150145
W
u
Image 1 Image 2
scanline
Example
NCC
Credit slide S. Lazebnik
u
v
– Smaller window- More detail- More noise
– Larger window- Smoother disparity maps- Less prone to noise
Window size = 3 Window size = 20
Effect of the window’s size
Credit slide S. Lazebnik
Issues
•Fore shortening effect
•Occlusions
O O’
O O’
O O’
Issues
• However, when B/z is small, small errors in measurements imply large error in estimating depth
• To reduce the effect of foreshortening and occlusions, it is desirable to have small B / z ratio!
O O’
u u’ ue’ u u’ ue’
Large B / z ratio
Small B / z ratio
• Homogeneous regions
Issues
mismatch
•Repetitive patterns
Issues
Correspondence problem is difficult!
- Occlusions- Fore shortening- Baseline trade-off- Homogeneous regions- Repetitive patterns
Apply non-local constraints to help enforce the correspondences
• Uniqueness – For any point in one image, there should be at most
one matching point in the other image
• Ordering– Corresponding points should be in the same order in
both views
• Smoothness– Disparity is typically a smooth function of x (except
in occluding boundaries)
Non-local constraints
Lecture 5 -Silvio Savarese 2-Feb-18
• Stereosystems• Rectification• Correspondenceproblem
• Multi-viewgeometry• TheSFMproblem• AffineSFM
Lecture6StereoSystemsMulti-viewgeometry
Courtesy of Oxford Visual Geometry Group
Structure from motion problem
Structure from motion problem
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points
•xij = Mi Xj , i = 1, … , m, j = 1, … , n
From the mxn observations xij, estimate: •m projection matrices Mi
•n 3D points Xj
x1j
x2j
xmj
Xj
motionstructure
M1
M2
Mm
Structure from motion problem
Affine structure from motion(simpler problem)
Image 1
World point Xj
Image i
From the mxn observations xij, estimate: •m projection matrices Mi (affine cameras)•n 3D points Xj
xij
=
m1
m2
0 0 0 1
!
"
####
$
%
&&&&
=
m1
m2
m3
!
"
####
$
%
&&&&
X
xE = (m1X,m2X)T
x =M X=
A2 x3 b2 x101x3 1
!
"##
$
%&&
magnification
úúú
û
ù
êêê
ë
é=
3
2
1
mmm
x =M X M = A bv 1
!
"#
$
%&=
m1
m2
m3
!
"
####
$
%
&&&&
X =m1Xm2Xm3X
!
"
####
$
%
&&&&
xE = (m1Xm3X
, m2Xm3X
)T
Perspective
Affine
= A b!"
#$X = A b!
"#$
xyz1
!
"
%%%%
#
$
&&&&
=AXE+b
[Eq. 3]
M =
m1
m2
m3
!
"
####
$
%
&&&&=
m1Xm2X1
!
"
####
$
%
&&&&
XE =xyz
!
"
###
$
%
&&&
Xj
xij =AiX j +bi
Affine cameras
For the affine case (in Euclidean space)
xi j
x1j
Image i
Image 1
[Eq. 4]
2x1 2x3 2x13x1
The Affine Structure-from-Motion Problem
Given m images of n fixed points Xj we can write
Problem: estimate m matrices Ai, m matrices biand the n positions Xj from the m´n observations xij .
2m ´ n equations in 8m + 3n - 8 unknowns
How many equations and how many unknown?
N. of cameras N. of pointsxij =AiX j +bi for i = 1, …,m and j = 1, … ,n
The Affine Structure-from-Motion Problem
Two approaches:
- Algebraic approach (affine epipolar geometry; estimate F; cameras; points)
- Factorization method
Next lecture
Multiple view geometry:Affine and Perspective structure from Motion