Upload
matthew-boone
View
215
Download
0
Embed Size (px)
Citation preview
Robot Vision SS 2007 Matthias Rüther 1
ROBOT VISION Lesson 6a: Shape from Stereo, short summary
Matthias Rüther
Slides partial courtesy of Marc Pollefeys Department of Computer Science
University of North Carolina, Chapel Hill
Robot Vision SS 2007 Matthias Rüther 2
Shape from Stereo
What do we want to do?
Measure 3D point location from two images of the point (from different viewpoints).
Robot Vision SS 2007 Matthias Rüther 3
Shape from Stereo
How?
• Acquire two images (simultaneously, if the scene is not stationary)
• Determine camera geometry
• Find homologous (corresponding) points, i.e. points in both images which are an image of the same world point.
• Triangulate homologous points, i.e. determine world point from homologous points and camera geometry
Robot Vision SS 2007 Matthias Rüther 4
(i) We have the camera geometry and an image point, how does this constrain the location of the corresponding image point in the second image? (Correspondence geometry)
(ii) We have corresponding image points, how does this constrain camera geometry? (Motion)
(iii) We have the camera geometry and corresponding image points, what are the 3D point coordinates? (Correspondence geometry)
Questions
Robot Vision SS 2007 Matthias Rüther 5
What if only C, C’, x are known?
Question i: Epipolar Geometry and F
Robot Vision SS 2007 Matthias Rüther 6
Family of planes and lines l and l’ Intersection in e and e’
Epipolar Geometry
Robot Vision SS 2007 Matthias Rüther 7
algebraic representation of epipolar geometry
l'x
we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F
Question i: Epipolar Geometry and F
Robot Vision SS 2007 Matthias Rüther 8
geometric derivation
xHx' π
x'e'l' FxxHe' π
mapping from 2-D to 1-D family (rank 2)
The Fundamental Matrix (F)
0
0
0
12
13
23
aa
aa
aa
a
Robot Vision SS 2007 Matthias Rüther 9
algebraic derivation
λCxPλX IPP
PP'e'F
xPP'CP'l
(note: doesn’t work for C=C’ F=0)
xP
λX
The Fundamental Matrix (F)
Robot Vision SS 2007 Matthias Rüther 10
correspondence condition
0Fxx'T
The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images
0l'x'T
The Fundamental Matrix (F)
Robot Vision SS 2007 Matthias Rüther 11
Mapping of corresponding epipolar lines: derivation
l,l’ epipolar lines, k line not through e l’=F[k]xl and symmetrically l=FT[k’]xl’
lke
kl lFk
e'
(pick k=e, since eTe≠0)
leFl' l'e'Fl T
Epipolar Line Geometry
Robot Vision SS 2007 Matthias Rüther 12
-T -13x3 3x3 3x3 3x3
ˆˆ ˆx H x, x' H 'x' F H ' FH
Derivation based purely on projective concepts
1 -14 4 4 4 4 4 4 4
ˆ ˆx PX PH H X PXx x x xPH H X
F invariant to transformations of projective 3-space
1 14 4 4 4 4 4 4 4
ˆ ˆx' P'X ' ( ' )( ) P'Xx x x xP H H X P H H X
FP'P,
P'P,F
unique
not unique
canonical form
m]|[MP'0]|[IP
MmF
Invariance under projective transformation
without derivation
Robot Vision SS 2007 Matthias Rüther 13
F matrix corresponds to P,P’ iff P’TFP is skew-symmetric
X0,FPXP'X TT
Possible choice:
]e'|F][[e'P' 0]|[IP
Canonical representation:T
3x1P [I|0] P' [[e'] F e'v |λe']
Canonical cameras given F
Arbitrary 3-vector Arbitrary scalar
Robot Vision SS 2007 Matthias Rüther 15
C1
C2
l2
P
l1
e1
e2
0m m 1T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
P
l1
e1
e2
m1
L1
m2
L2
M
l2lT1
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
Epipolar Geometry
Robot Vision SS 2007 Matthias Rüther 16
given xi↔x‘i , compute P,P‘ and Xi
reconstruction problem:
ii PXx x P'Xi i for all i
without additional information possible up to projective ambiguity
3D reconstruction of cameras and structure
Robot Vision SS 2007 Matthias Rüther 17
(i) Compute F from correspondences(ii) Compute camera matrices from F(iii) Compute 3D point for each pair of
corresponding points
computation of Fuse x‘iFxi=0 equations, linear in coeff. F8 points (linear), 7 points (non-linear), 8+ (least-squares)
computation of camera matricesuse ]λe'|ve'F][[e'P' 0]|[IP T
triangulationcompute intersection of two backprojected rays
Outline of reconstruction
Robot Vision SS 2007 Matthias Rüther 18
iii XHPHPXx P-1
P
Reconstruction ambiguity: projective
Robot Vision SS 2007 Matthias Rüther 19
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
i111 X,'P,P i222 X,'P,Pii xx -1
12 HPP -112 HPP
12 HXX 0FxFx :except ii
iiiii 22111-1
112 XPxXPHXHPHXP along same ray of P2, idem for P‘2
two possibilities: X2i=HX1i, or points along baseline
key result: allows reconstruction from pair of uncalibrated images
The projectove reconstruction theorem
Robot Vision SS 2007 Matthias Rüther 20
Robot Vision SS 2007 Matthias Rüther 21
(i) Projective reconstruction
(ii) Affine reconstruction
(iii) Metric reconstruction
Stratified reconstruction
Robot Vision SS 2007 Matthias Rüther 22
ii HXXE
use control points XEi with known coordinatesto go from projective to metric
Eii XPHx -1(2 lin. eq. in H-1
per view, 3 for two views)
Direct metric reconstruction using ground truth
Robot Vision SS 2007 Matthias Rüther 23
0Fxx'T
separate known from unknown
0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx
0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx
(data) (unknowns)(linear)
0Af
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
Computation of F: basic equation
Robot Vision SS 2007 Matthias Rüther 24
0Fe'T 0Fe 0detF 2Frank
T333
T222
T111
T
3
2
1
VσUVσUVσUVσ
σσ
UF
SVD from linearly computed F matrix (rank 3)
T222
T111
T2
1
VσUVσUV0
σσ
UF'
FF'-FminCompute closest rank-2 approximation
Imposing the singularity constraint
Robot Vision SS 2007 Matthias Rüther 25
Robot Vision SS 2007 Matthias Rüther 26
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
yxyyyyxxxyxx
yxyyyyxxxyxx
yxyyyyxxxyxx
nnnnnnnnnnnn
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differenceBetween column of data matrix least-squares yields poor results
The NOT normalized 8-point algorithm
Robot Vision SS 2007 Matthias Rüther 27
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
Least squares yields good results (Hartley, PAMI´97)
The normalized 8-point algorithm
Robot Vision SS 2007 Matthias Rüther 28
Maximum Likelihood Estimation
i
iiii dd 22 'x̂,x'x̂,x
(= least-squares for Gaussian noise)
0x̂F'x̂ subject to T
iXt],|[MP'0],|[IP Parameterize:
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
iiii XP'x̂,PXx̂
Minimize cost using Levenberg-Marquardt(preferably sparse LM, see book)
(overparametrized)
The Gold Standard Algorithm
Robot Vision SS 2007 Matthias Rüther 29
Examples
Robot Vision SS 2007 Matthias Rüther 30
Examples
Robot Vision SS 2007 Matthias Rüther 31
Examples
Robot Vision SS 2007 Matthias Rüther 32
1. Do not use unnormalized algorithms
2. Quick and easy to implement: 8-point normalized
3. Better: enforce rank-2 constraint during minimization
4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)
Recommendations
Robot Vision SS 2007 Matthias Rüther 33
Extract feature points to relate images
Required properties:– Well-defined
(i.e. neigboring points should all be different)
– Stable across views
(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
The correspondence problem: feature points
Robot Vision SS 2007 Matthias Rüther 34
homogeneous
edge
corner
M should have large eigenvalues
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
Find points that differ as much as possible from all neighboring points
Feature = local maxima (subpixel) of F(1, 2)
Feature points
Robot Vision SS 2007 Matthias Rüther 35
Select strongest features (e.g. 1000/image)
Feature points
Robot Vision SS 2007 Matthias Rüther 36
Evaluate NCC for all features with
similar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
Robot Vision SS 2007 Matthias Rüther 37
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
1 5
24
3
1 5
24
3
Gives satisfying results for small image motions
Similarity Example
Robot Vision SS 2007 Matthias Rüther 38
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. doStep 3.1 select minimal sample (i.e. 8 matches)
Step 3.2 compute F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90% 80% 70% 60% 50%
#samples 5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
Robot Vision SS 2007 Matthias Rüther 39
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
Robot Vision SS 2007 Matthias Rüther 40
Degenerate cases– Planar scene
– Pure rotation
No unique solution– Remaining DOF filled by noise
– Use simpler model (e.g. homography)
Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
– Compare H and F according to expected residual error (compensate for model complexity)
Degenerate Cases
Robot Vision SS 2007 Matthias Rüther 41
Absence of sufficient features (no texture)
Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
More problems