Upload
baruch
View
27
Download
0
Embed Size (px)
DESCRIPTION
Computing F class 13. Multiple View Geometry Comp 290-089 Marc Pollefeys. Two-view geometry. Epipolar geometry 3D reconstruction F-matrix comp. Structure comp. p. p. L 2. L 2. m 1. m 1. m 1. C 1. C 1. C 1. M. M. L 1. L 1. l 1. l 1. e 1. e 1. l T 1. l 2. e 2. e 2. - PowerPoint PPT Presentation
Citation preview
Computing Fclass 13
Multiple View GeometryComp 290-089Marc Pollefeys
Two-view geometry
Epipolar geometry
3D reconstruction
F-matrix comp.
Structure comp.
C1
C2
l2
l1
e1
e20m m 1
T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong
matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
l1
e1
e2
m1
L1
m2
L2
M
l2lT1
Epipolar geometry
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
1T
1 lPπ
1T
1T
22 lPPl
0lx 2T
2
111 xel
0x][ePPx 1x1T
1T
2T2
The projective reconstruction theorem
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!
ObjectiveGiven two uncalibrated images compute (PM,P‘M,{XMi})(i.e. within similarity of original scene and cameras)
Algorithm(i) Compute projective reconstruction (P,P‘,{Xi})
(a) Compute F from xi↔x‘i(b) Compute P,P‘ from F(c) Triangulate Xi from xi↔x‘i
(ii) Rectify reconstruction from projective to metricDirect method: compute H from control points
Stratified method:(a) Affine reconstruction: compute ∞
(b) Metric reconstruction: compute IAC
ii HXXE -1
M PHP -1M HPP ii HXXM
π0|I
H
100AH
-1
1TT ωMMAA
Image information provided
View relations and projective objects
3-space objects
reconstruction ambiguity
point correspondences F projective
point correspondences including vanishing points F,H∞ ∞
affine
Points correspondences and internal camera calibration
F,H∞
’
∞
∞
metric
Epipolar geometry: basic equation
0Fxx'T
separate known from unknown
0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx
0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx
(data) (unknowns)(linear)
0Af
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
the singularity constraint
0Fe'T 0Fe 0detF 2Frank
T333
T222
T111
T
3
2
1
VσUVσUVσUVσ
σσ
UF
SVD from linearly computed F matrix (rank 3)
T222
T111
T2
1
VσUVσUV0
σσ
UF'
FF'-FminCompute closest rank-2 approximation
Frobenius distance
)()(real is
*
1 1
2 AAtraceAAtraceaA TAm
i
n
jijFnm
the minimum case – 7 point correspondences
0f1''''''
1''''''
777777777777
111111111111
yxyyyxyxyxxx
yxyyyxyxyxxx
T9x91717717x7 ]V0 ,0 ,σ,...,σ[diagUA
9x298 0]VA[V
1...70,)xλFF(x 21T iii
one parameter family of solutions
but F1+F2 not automatically rank 2
F1 F2
F
3
F7pts
0λλλ)λFFdet( 012
23
321 aaaa
(obtain 1 or 3 solutions)
(cubic equation)
0)λIFFdet(Fdet)λFFdet( 1-12221
the minimum case – impose rank 2
Compute possible as eigenvalues of (only real solutions are potential solutions)
1-12 FF
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
yxyyyyxxxyxx
yxyyyyxxxyxx
yxyyyyxxxyxx
nnnnnnnnnnnn
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differenceBetween column of data matrix least-squares yields poor results
the NOT normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
Least squares yields good results (Hartley, PAMI´97)
the normalized 8-point algorithm
algebraic minimization
possible to iteratively minimize algebraic distance subject to det F=0 (see book if interested)
Geometric distance
Gold standard
Sampson error
Symmetric epipolar distance
Gold standard
Maximum Likelihood Estimation
i
iiii dd 22 'x̂,x'x̂,x
(= least-squares for Gaussian noise)
0x̂F'x̂ subject to T
iXt],|[MP'0],|[IP Parameterize:
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
iiii XP'x̂,PXx̂
Minimize cost using Levenberg-Marquardt(preferably sparse LM, see book)
(overparametrized)
Parameterization of F
Alternative, minimal parametrization (with a=1)
(note (x,y,1) and (x‘,y‘,1) are epipoles)
problems: • a=0 pick largest of a,b,c,d to fix
• epipole at infinity
pick largest of x,y,w and of x’,y’,w’
4x3x3=36 parametrizations!
reparametrize at every iteration, to be sure
First-order geometric error (Sampson error)
ee
1TT JJ T
T
JJ
ee (one eq./pointJJT scalar)
0Fxx'T e
001
Fx'T
ix
e
22
21
2
2T2
1TT FxFxFx'Fx'JJ
T
T
JJ
ee
22
21
2
2T2
1T
T
FxFxFx'Fx'
Fxx'
(problem if some x is located at epipole)
advantage: no subsidiary variables required
Symmetric epipolar error
2
221
2
2T2
1T
T
FxFx
1
Fx'Fx'
1Fxx'
i
iiii dFd2T2 x'F,xx,x'
Some experiments:
Some experiments:
Some experiments:
Some experiments:
i
iiii dFd2T2 x'F,xx,x'
(for all points!)
Residual error:
Recommendations:
1. Do not use unnormalized algorithms
2. Quick and easy to implement: 8-point normalized
3. Better: enforce rank-2 constraint during minimization
4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)
Special case:
Enforce constraints for optimal results:Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof)
The envelope of epipolar lines
What happens to an epipolar line if there is noise?
Monte Carlo
n=10 n=15 n=25 n=50
l2Tll kC
Other entities?
Lines give no constraint for two view geometry(but will for three and more views)
Curves and surfaces yield some constraints related to tangency
Automatic computation of F
(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)
• Extract feature points to relate images
• Required properties:• Well-defined
(i.e. neigboring points should all be different)
• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
Feature points
homogeneous
edge
corner
dxdyyxwyI
xI
W yIxI
,
M
M should have large eigenvalues
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
MTSSD
Find points that differ as much as possible from all neighboring points
Feature = local maxima (subpixel) of F(1, 2)
Feature points
Select strongest features (e.g. 1000/image)
Feature points
Evaluate Normalized Cross Correlation for all features with similar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
1 5
24
3
1 5
24
3
Gives satisfying results for small image motions
Feature example
• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination
geometric transformations
photometric changes
Wide-baseline matching…
Wide baseline matching for two different region types
(Tuytelaars and Van Gool BMVC 2000)
Wide-baseline matching…
Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90%
80%
70% 60%
50%
#samples
5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
• Degenerate cases• Planar scene• Pure rotation
• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
• Compare H and F according to expected residual error (compensate for model complexity)
Degenerate cases:
• Absence of sufficient features (no texture)• Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
More problems:
geometric relations between two views is fully
described by recovered 3x3 matrix F
two-view geometry
Next class: image pair rectificationreconstructing points and lines