View
231
Download
3
Category
Preview:
Citation preview
Computer Visioncmput 498/613
3D Modeling from images3D Modeling from images
Martin JagersandMartin Jagersand
Pinhole camera• Central projectionCentral projection
• Principal point & aspectPrincipal point & aspect
10100
000
000
~
1
)/,/(),,(
Z
Y
X
f
f
Z
fY
fX
y
x
ZfYZfXZYX TT
1100
10
01
1
1
1
~
1
y
x
cp
cp
cyp
cxp
v
u
yy
xx
yy
xx c
px
py
The projection matrix:
cam
camyy
xx
IK
cp
f
cp
f
X0x
Xx
]|[
0100
00
00
Projective camera
• Camera rotation and translationCamera rotation and translation
• The projection matrixThe projection matrix XtXXtX TT
camcam RRR
Xtx IKRT
PIn general:
•P is a 3x4 matrix with 11 DOF
•Finite: left 3x3 matrix non-singular
•Infinite: left 3x3 matrix singular
Properties: P=[M p4]
•Center:
•Principal ray (projection direction)
0,01
0
41
dd
Cp
C
C
MM
P
3)det( mv M
• Infinite cameras where the last row of Infinite cameras where the last row of PP is is (0,0,0,1)(0,0,0,1)• Points at infinity are mapped to points at infinityPoints at infinity are mapped to points at infinity
Affine cameras
z
z
y
x
Ty
x
td
t
t
t
R
d
t
t
KP
0
0
t
k
j
i
0
j
i
td0
j
k
i{camera}
Z
X
Y{world}
x0
• ErrorError)( 0
0
xxxx
projperspaff d
Good approximation:
• small compared to d0
• point close to principal ray
X
X’
xaff
xpersp
Camera calibration
• 11 DOF => at least 6 points11 DOF => at least 6 points• Linear solutionLinear solution
– Normalization requiredNormalization required– Minimizes algebraic errorMinimizes algebraic error
• Nonlinear solutionNonlinear solution– Minimize geometric error (pixel re-projection)Minimize geometric error (pixel re-projection)
• Radial distortionRadial distortion– Small near the center, increase towards peripherySmall near the center, increase towards periphery
1
0min
p
pA
ii P Xx known?known
...1 221 rKrKr
Application: raysets
Gortler and al.; Microsoft Lumigraph
t
v
s
u (s,t)
(u,v)
H-Y Shum, L-W He; MicrosoftConcentric mosaics
Ck
Cl C0
Li
Lj
vi
vj i j
• Projection equationProjection equation
xxii=P=PiiXX
• Resection:Resection:– xxii,X P,X Pii
Multi-view geometry - resection
Given image points and 3D points calculate camera projection matrix.
• Projection equationProjection equation
xxii=P=PiiXX
• Intersection:Intersection:– xxii,P,Pi i XX
Multi-view geometry - intersection
Given image points and camera projections in at least 2 views calculate the 3D points (structure)
• Projection Projection equationequation
xxii=P=PiiXX
• Structure from Structure from motion (SFM)motion (SFM)– xxii P Pii,, XX
Multi-view geometry - SFM
Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion)
•Estimate projective structure
•Rectify the reconstruction to metric (autocalibration)
Depth from stereo
•Calibrated aligned camerasCalibrated aligned cameras
Z
X(0,0) (d,0)
Z=f
xl xrrl
rl
xx
dfZ
dXx
fX
x
fZ
)(
Disparity d
Trinocular Vision System
(Point Grey Research)
Application: depth based reprojection
3D warping, 3D warping, McMillanMcMillan
Plenoptic modeling, Plenoptic modeling, McMillan & BishopMcMillan & Bishop
Application: depth based reprojection
Layer depth images,Layer depth images, Shade et al. Shade et al.
Image based objects,Image based objects, Oliveira & Bishop Oliveira & Bishop
Affine camera factorization3D structure from many images
The affine projection equations are The affine projection equations are
1
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
10001
1
j
j
j
yi
xi
ij
ij
Z
Y
X
P
P
y
x
~
~
4
4
j
j
j
yi
xi
ij
ij
yiij
xiij
Z
Y
X
P
Py
x
Py
Px
Orthographic factorization
The ortographic projection equations are The ortographic projection equations are
where where njmijiij ,...,1,,...,1,Mm P
All equations can be collected for all All equations can be collected for all ii and and jj
wherewhere
n
mmnmm
n
n
M,...,M,M,,
mmm
mmm
mmm
212
1
21
22221
11211
M
P
P
P
Pm
MPm
M ~
~m
j
j
j
jyi
xi
iij
ijij
Z
Y
X
,P
P,
y
xP
Note that P and M are resp. 2mx3 and 3xn matrices and
therefore the rank of m is at most 3
(Tomasi Kanade’92)
Orthographic factorization
Factorize Factorize mm through singular value decomposition through singular value decomposition
An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows
TVUm
TVMUP ~,
~
(Tomasi Kanade’92)
n
mmnmm
n
n
M,...,M,M
mmm
mmmmmm
min 212
1
21
22221
11211
P
P
P
Closest rank-3 approximation yields MLE!
0~~
1~~
1~~
TT1
TT1
TT1
yi
xi
yi
yi
xi
xi
PP
PP
PP
0~~
1~~
1~~
T
T
T
yi
xi
yi
yi
xi
xi
PP
PP
PP
C
C
C
A metric reconstruction is obtained as followsA metric reconstruction is obtained as follows
Where A is computed from Where A is computed from
Orthographic factorization
Factorize Factorize mm through singular value decomposition through singular value decomposition
An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows
TVUm
TVMUP ~,
~
MQMQPP~
,~ 1
0
1
1
T
T
T
yi
xi
yi
yi
xi
xi
PP
PP
PP 3 linear equations per view on
symmetric matrix C (6DOF)
Q can be obtained from C through Cholesky factorisation
and inversion
(Tomasi Kanade’92)
N-view geometryAffine factorization
[Tomasi &Kanade ’92][Tomasi &Kanade ’92]
•Affine camera Affine camera
•ProjectionProjection
•nn points, points, mm views: measurement matrix views: measurement matrix
]|[ tMP M 2x3 matrix; t 2D vector
t
Z
Y
X
My
x
nmm
nm
n
M
M
W XX
xx
xx
1
1
1
111
~~
~~
W: Rank 3XMVDUW
UDVWT
nm
T
ˆˆˆ33332
Assuming isotropic zero-mean Gaussian noise, factorization achieves ML affine reconstruction.
txx ~
Weak perspective factorization
[D. Weinshall][D. Weinshall]
•Weak perspective cameraWeak perspective camera
•Affine ambiguityAffine ambiguity
•Metric constraintsMetric constraints
j
i
s
sM
)ˆ)(ˆ(ˆˆˆ 11 XQQMXQQMW
0ˆˆ
ˆˆˆˆ 2
ji
jjii
sQQs
ssQQssQQsTT
TTTT
Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles
Full perspective factorization
The camera equations The camera equations
for a fixed image for a fixed image ii can be written in matrix form can be written in matrix form asas
where where
njmijiijij ,...,1,,...,1,Mmλ P
MPm iii
imiii
mimiii mmm
λ,...,λ,λdiag
M,...,M,M,,...,,
21
2121
Mm
Perspective factorization
All equations can be collected for all All equations can be collected for all ii as as
wherewherePMm
mnn P
P
P
P
m
m
m
m...
,...
2
1
22
11
In these formulas In these formulas mm are known, but are known, but ii,,PP and and MM are unknown are unknown
Observe that Observe that PMPM is a product of a 3 is a product of a 3mmx4 matrix and a 4xx4 matrix and a 4xnn matrix, i.e. it is a rank 4 matrixmatrix, i.e. it is a rank 4 matrix
Perspective factorization algorithm
Assume that i are known, then PM is known.
Use the singular value decomposition PM=U VT
In the noise-free case
S=diag(1,2,3,4,0, … ,0)and a reconstruction can be obtained by setting:
P=the first four columns of U.M=the first four rows of V.
Iterative perspective factorization
When i are unknown the following algorithm can be used:
1. Set ij=1 (affine approximation).
2. Factorize PM and obtain an estimate of P and M. If 5 is sufficiently small then STOP.
3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM
4. Goto 2.
In general the algorithm minimizes the proximity measure P(,P,M)=5Note that structure and motion recovered
up to an arbitrary projective transformation
Projective factorizationsummary
[Sturm & Triggs’96][ Heyden ‘97 ][Sturm & Triggs’96][ Heyden ‘97 ]
•Measurement matrixMeasurement matrix
•Known projective depthKnown projective depth
– Projective ambiguity Projective ambiguity
• Iterative algorithmIterative algorithm– Reconstruct withReconstruct with– Reestimate depth and iterate Reestimate depth and iterate
nmm
nmn
mm
nn
P
P
W XX
xx
xx
1
1
11
1111
11
3mxn matrix
Rank 4
ij
XPVDUW
UDVWT
nm
T
ˆˆˆ44442
1ij
ij
Further Factorization work
Factorization with uncertaintyFactorization with uncertainty
Factorization for dynamic scenesFactorization for dynamic scenes
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Bregler et al. 2000, Brand 2001)
Sequential 3D structure from motionusing 2 and 3 view geom
• Initialize structure and motion from two viewsInitialize structure and motion from two views• For each additional viewFor each additional view
– Determine poseDetermine pose– Refine and extend structureRefine and extend structure
• Determine correspondences robustly by jointly Determine correspondences robustly by jointly estimating matches and epipolar geometry estimating matches and epipolar geometry
Images
2 view geometry (Epipolar geometry)
Fundamental matrix
• Algebraic representation of epipolar geometryAlgebraic representation of epipolar geometry
Step 1: X on a plane
Step 2: epipolar line l’xxe
xexel
xx
FH
H
]'[
']'['''
'
0' xx FT
F
•3x3, Rank 2, det(F)=0
•Linear sol. – 8 corr. Points (unique)
•Nonlinear sol. – 7 corr. points (3sol.)
•Very sensitive to noise & outliers
[Faugeras ’92, Hartley ’92 ]
Epipolar lines:
Epipoles:
Projection matrices:
'|']'['
]|[
0'0
''
evee
0
ee
xlxl
T
T
T
FP
IP
FF
FF
3,4,N view geometry
• Trifocal tensor (3 view geometry)Trifocal tensor (3 view geometry)
],,[: 321 TTTT 3x3x3 tensor;
27 params. (18 indep.)
0]"[)(]'[
"]3,2,1['
xxx
lll
ii
i
T
T
TTT lines
points
• Quadrifocal tensor (4 view geometry) Quadrifocal tensor (4 view geometry) [Triggs ’95]
•Multiview tensors Multiview tensors [Hartley’95][ Hayden ‘98]
There is no additional constraint between more than 4 images. All the constraints There is no additional constraint between more than 4 images. All the constraints can be expressed using F,triliear tensor or quadrifocal tensor. can be expressed using F,triliear tensor or quadrifocal tensor.
[Hartley ’97][Torr & Zisserman ’97][ Faugeras ’97]
Initial structure and motion
eeaFeP
0IPT
x2
1
Epipolar geometry Projective calibration
012 FmmT
compatible with F
Yields correct projective camera setup(Faugeras´92,Hartley´92)
Obtain structure through triangulation
Use reprojection error for minimizationAvoid measurements in projective space
Compute Pi+1 using robust approachFind additional matches using predicted projectionExtend, correct and refine reconstruction
2D-2D
2D-3D 2D-3D
mimi+1
M
new view
Determine pose towards existing structure
Non-sequential image collections
4.8im/pt64 images
3792
po
ints
Problem:Features are lost and reinitialized as new features
Solution:Match with other close views
For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure
Relating to more views
Problem: find close views in projective frame
For every view For every view iiExtract featuresExtract featuresCompute two view geometry Compute two view geometry ii-1/-1/ii and matches and matches Compute pose using robust algorithmCompute pose using robust algorithmFor all For all closeclose views views kk
Compute two view geometry Compute two view geometry kk//ii and matches and matchesInfer new 2D-3D matches and add to listInfer new 2D-3D matches and add to list
Refine pose using all 2D-3D matchesRefine pose using all 2D-3D matchesRefine existing structureRefine existing structureInitialize new structureInitialize new structure
Determining close views
• If viewpoints are If viewpoints are closeclose then most image changes then most image changes can be modelled through a can be modelled through a planar homographyplanar homography
• Qualitative distance measureQualitative distance measure is obtained by is obtained by looking at the looking at the residual errorresidual error on the on the best possible best possible planar homographyplanar homography
Distance = m´,mmedian min HD
9.8im/pt
4.8im/pt
64 images
64 images
3792
po
ints
2170
po
ints
Non-sequential image collections (2)
Refining structure and motion
• Minimize reprojection errorMinimize reprojection error
– Maximum Likelyhood Estimation Maximum Likelyhood Estimation (if (if error zero-mean Gaussian noise)error zero-mean Gaussian noise)
– Huge problem but can be solved efficientlyHuge problem but can be solved efficiently(Bundle adjustment)(Bundle adjustment)
m
k
n
iikD
ik 1 1
2
kiM̂,P̂
M̂P̂,mmin
Refining a captured model:Bundle adjustment
• Refine structure XRefine structure Xjj and motion P and motion Pii
• Minimize geometric errorMinimize geometric error• ML solution, assuming noise is GaussianML solution, assuming noise is Gaussian• Tolerant to missing dataTolerant to missing data
ji
ijj
iPd,
2),ˆˆ(min xX
Projective ambiguity andself-calibration
Given an uncalibrated image sequence with corresponding point it is possible to reconstruct the object up to an unknown projective transformation
• Autocalibration (self-calibration): Determine a projective transformation T that upgrades the projective reconstruction to a metric one.
T
´M´M))((Mm 1 PTPTP
Remember:Stratification of geometry
15 DOF 12 DOFplane at infinity
parallelism
More general
More structure
Projective Affine Metric
7 DOFabsolute conicangles, rel.dist.
Constraints ?
• Scene constraintsScene constraints– Parallellism, vanishing points, horizon, ...Parallellism, vanishing points, horizon, ...– Distances, positions, angles, ...Distances, positions, angles, ...
Unknown scene no constraints
• Camera extrinsics constraints–Pose, orientation, ...
Unknown camera motion no constraints • Camera intrinsics constraints
–Focal length, principal point, aspect ratio & skew
Perspective camera model too general some constraints
Euclidean projection matrix
tRRKP TT
1yy
xx
uf
usf
K
Factorization of Euclidean projection matrix
Intrinsics:
Extrinsics: t,R
Note: every projection matrix can be factorized,
but only meaningful for euclidean projection matrices
(camera geometry)
(camera motion)
Constraints on intrinsic parameters
Constant Constant e.g. fixed camera:e.g. fixed camera:
KnownKnowne.g. rectangular pixels:e.g. rectangular pixels:
square pixels:square pixels: principal point known:principal point known:
21 KK
0s
1yy
xx
uf
usf
K
0, sff yx
2,
2,
hwuu yx
Self-calibration
Upgrade from Upgrade from projectiveprojective structure to structure to metricmetric structure using structure using constraintsconstraints onon intrinsicintrinsic camera parameters camera parameters– Constant intrinsicsConstant intrinsics
– Some known intrinsics, others varyingSome known intrinsics, others varying
– Constraints on intrincs and restricted motionConstraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion)(e.g. pure translation, pure rotation, planar motion)
(Faugeras et al. ECCV´92, Hartley´93,
Triggs´97, Pollefeys et al. PAMI´98, ...)
(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)
(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)
A counting argument
• To go from projective (15DOF) to metric (7DOF) at least 8 To go from projective (15DOF) to metric (7DOF) at least 8 constraints are neededconstraints are needed
• Minimal sequence length should satisfyMinimal sequence length should satisfy
• Independent of algorithmIndependent of algorithm• Assumes general motion (i.e. not critical)Assumes general motion (i.e. not critical)
8#1# fixednknownn
Conics•Conic: Conic:
– Euclidean geometry: hyperbola, ellipse, parabola & degenerateEuclidean geometry: hyperbola, ellipse, parabola & degenerate– Projective geometry: equivalent under projective transformProjective geometry: equivalent under projective transform– Defined by 5 pointsDefined by 5 points
•TangentTangent
•Dual conic C*Dual conic C*
0
022
xx C
feydxcybxyaxT
fed
ecb
dba
C
2/2/
2/2/
2/2/
xl C
0* ll CT
Quadrics
Quadrics: Quadrics: QQ 4x4 symmetric matrix 4x4 symmetric matrix
9 DOF (defined by 9 points in general pose)9 DOF (defined by 9 points in general pose)
•Dual: Dual: Q*Q*Planes tangent to the quadricPlanes tangent to the quadric
0XX QT
0* ππ QT
Summary:Conics & Quadrics
0mmT C 0ll *T C
1* CC
conics
0MMT Q 0*T Q
1* QQ
quadrics
1T´~ CHHCC
T*** ´~ HHCCC T´ TTQ~QQ ***
1T´~ QTTQQ
transformations
T** ~ PPQC
projection
The absolute conic
• Absolute conic is a imaginary circle onAbsolute conic is a imaginary circle on• The absolute dual quadric (rim quadric)The absolute dual quadric (rim quadric)
• In a metric frameIn a metric frame
π
0),,(),,(
0
)1,0,0,0(
321321
4
23
22
21
TxxxIxxx
x
xxx
π
*
0
0*
*
T
T
I
0
0
Note: is the nullspace of
*
On : π
*
Fixed under similarity transf.
Self-calibration
•Theoretically formulated by Theoretically formulated by [Faugeras ’92][Faugeras ’92]
•2 basic approaches2 basic approaches– Stratified: recover Stratified: recover – Direct: recover Direct: recover [Triggs’97][Triggs’97]
•Constraints:Constraints:– Camera internal constraintsCamera internal constraints
– Constant parameters Constant parameters [Hartley’94][ Mohr’93][Hartley’94][ Mohr’93]
– Known skew and aspect ratioKnown skew and aspect ratio [Hayden&[Hayden&ÅÅstrströöm’98][Pollefeys’98]m’98][Pollefeys’98]
– Scene constraints (angles, ratios of length)Scene constraints (angles, ratios of length)
•Choice of H:Choice of H: Knowing camera Knowing camera KK and and
π
*
TTT K
KH )1,(,
1p
p
0
π
Absolute Dual Quadric and Self-calibration
Eliminate extrinsics from equationEliminate extrinsics from equation
Equivalent to projection of dual quadric
))(Ω)((Ω *1* TTTTT PTTTPTPPKK
Abs.Dual Quadric also exists in projective world
T´Ω´´ * PP Transforming world so thatreduces ambiguity to metric
** ΩΩ´
*
*
projection
constraints
Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics
Tii
Tiii Ωω KKPP
Absolute Dual Quadric and Self-calibration
Projection equation:Projection equation:
Translate constraints on Translate constraints on K K through projection equationthrough projection equation to to constraints on constraints on **
KRd0d]C
~|KR[IPXx
mapping between ∞ to an image is given by the planar homogaphy x=Hd, with H=KR
image of the absolute conic (IAC) = I
1-T-1T KKKKω 1TCHHC
(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT
(iv) K (cholesky factorisation)(v) image of circular points
2T
21T
1
2T
1
ωxxωxx
ωxxcos
Image of the absolute conic
HZ 7.5.1:
Constraints on *
1
ω 22
222
*
yx
yyyyxy
xyxyxx
cc
ccfccsf
cccsfcsf
Zero skewZero skew quadraticquadratic mm
Principal pointPrincipal point linearlinear 22mm
Zero skew (& p.p.)Zero skew (& p.p.) linearlinear mm
Fixed aspect ratio (& Fixed aspect ratio (& p.p.& Skew)p.p.& Skew)
quadraticquadratic m-1m-1
Known aspect ratio Known aspect ratio (& p.p.& Skew)(& p.p.& Skew)
linearlinear mm
Focal length Focal length
(& p.p. & Skew)(& p.p. & Skew)
linearlinear mm
*23
*13
*33
*12 ωωωω
0ωω *23
*13
0ω*12
*11
*22
*22
*11 ω'ωω'ω
*22
*11 ωω
*11
*33 ωω
condition constraint type #constraints
Summary: Self calibration based on the IADC
•Calibrated cameraCalibrated camera– Dual absolute quadric (DAC)Dual absolute quadric (DAC)– Dual image of the absolute conic (DIAC)Dual image of the absolute conic (DIAC)
•Projective cameraProjective camera– DACDAC– DIACDIAC
•AutocalibrationAutocalibration– Determine based on constraints onDetermine based on constraints on– Decompose Decompose
TKK
diagI
*
)0,1,1,1(~
Tii
iTii
T
KKPQP
HIHQ
**
* ~
* i*THIHQ
~*
i
j
Ci
Cj
Illustration of self-calibration
Projective Affine Metric
Degenerate configurations
• Pure translation: Pure translation: affine transformation (5 DOF)affine transformation (5 DOF)
• Pure rotation: Pure rotation: arbitrary pose for (3 DOF) arbitrary pose for (3 DOF)
• Planar motion: Planar motion: scaling axis perpendicular to plane scaling axis perpendicular to plane (1DOF)(1DOF)
• Orbital motion: Orbital motion: projective distortion along rotation axis projective distortion along rotation axis (2DOF) (2DOF)
π
Not unique solution !Not unique solution !
A complete modeling systemprojective
Sequence of frames scene structureSequence of frames scene structure
1.1. Get corresponding points (tracking).Get corresponding points (tracking).
2.2. 2,3 view geometry: 2,3 view geometry: compute F,T between consecutive frames compute F,T between consecutive frames (recompute correspondences).(recompute correspondences).
3.3. Initial reconstruction: Initial reconstruction: get an initial structure from a get an initial structure from a subsequence with big baseline (trilinear tensor, factorization …) subsequence with big baseline (trilinear tensor, factorization …) and bind more frames/points using resection/intersection.and bind more frames/points using resection/intersection.
4.4. Self-calibration.Self-calibration.
5.5. Bundle adjustment.Bundle adjustment.
A complete modeling systemaffine
Sequence of frames scene structureSequence of frames scene structure
1.1. Get corresponding points (tracking).Get corresponding points (tracking).
2.2. Affine factorization. (This already computes ML Affine factorization. (This already computes ML estimate over all frames so no need for bundle estimate over all frames so no need for bundle adjustment for simple scenes.adjustment for simple scenes.
3.3. Self-calibration.Self-calibration.
4.4. If several model segments: Merge, bundle adjust.If several model segments: Merge, bundle adjust.
Examples – modeling with dynamic texture
Cobzas,Yerex,JagersandCobzas,Yerex,Jagersand
Examples: geometric modeling
Debevec and Taylor:Debevec and Taylor: Façade Façade
Examples: geometric modeling
Pollefeys: Pollefeys: Arenberg CastleArenberg Castle
Examples: geometric modeling
INRIA –INRIA –VISIRE projectVISIRE project
Examples: geometric modeling
CIP Prague –CIP Prague –Projective Reconstruction Based on Cake ConfigurationProjective Reconstruction Based on Cake Configuration
Reading: FP Chapter 11.
• The Stereopsis Problem: Fusion and Reconstruction• Human Stereopsis and Random Dot Stereograms• Cooperative Algorithms• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras
Dense stereo
•Go back to original images, do dense matching.Go back to original images, do dense matching.
•Try to get dense depth mapsTry to get dense depth maps
Stereo image rectification
Stereo image rectification
• reproject image planes onto a commonreproject image planes onto a common• plane parallel to the line between optical centersplane parallel to the line between optical centers• pixel motion is horizontal after this transformationpixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each input two homographies (3x3 transform), one for each input
image reprojectionimage reprojectionC. Loop and Z. Zhang. C. Loop and Z. Zhang. Computing RectifyingComputing Rectifying Homographies Homographies
for Stereo Vision for Stereo Vision. IEEE Conf. Computer Vision and Pattern . IEEE Conf. Computer Vision and Pattern Recognition, 1999Recognition, 1999..
All epipolar lines are parallel in the rectified image plane.
Rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
He
0
0
1
Image rectification throughhomography warp
ExampleUnrectified
Rectified
Depth from disparity
f
x x’
baseline
z
C C’
X
f
Stereo matching algorithms•Match Pixels in Conjugate Epipolar LinesMatch Pixels in Conjugate Epipolar Lines
– Assume brightness constancyAssume brightness constancy– This is a tough problemThis is a tough problem– Numerous approachesNumerous approaches
– A good survey and evaluation: A good survey and evaluation: http://www.http://www.middleburymiddlebury..eduedu/stereo//stereo/
Your basic stereo algorithm
For each epipolar line
For each pixel in the left image• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Improvement: match windows• This should look familar...
Stereo as energy minimization
• Find disparities d that minimize an energy function
• Simple pixel / window matching
SSD distance between windows I(x, y) and J(x, y + d(x,y))=
Stereo as energy minimization
I(x, y) J(x, y)
y = 141
C(x, y, d); the disparity space image (DSI)x
d
Stereo as energy minimization
y = 141
x
d
Simple pixel / window matching: choose the minimum of each column in the DSI independently:
Matching windowsSimilarity MeasureSimilarity Measure FormulaFormula
Sum of Absolute Differences Sum of Absolute Differences (SAD)(SAD)
Sum of Squared Differences (SSD)Sum of Squared Differences (SSD)
Zero-mean SADZero-mean SAD
Locally scaled SADLocally scaled SAD
Normalized Cross Correlation Normalized Cross Correlation (NCC)(NCC)
http://siddhantahuja.wordpress.com/category/stereo-vision/ SAD SSD NCC Ground truth
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
Hierarchical stereo matchingD
ow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
Example: reconstruct image from neighboring images
Recommended