Robot Vision SS 2007 Matthias Rüther 1 ROBOT VISION Lesson 6a: Shape from Stereo, short summary Matthias Rüther Slides partial courtesy of Marc Pollefeys

Robot Vision SS 2007 Matthias Rüther 1

ROBOT VISION Lesson 6a: Shape from Stereo, short summary

Matthias Rüther

Slides partial courtesy of Marc Pollefeys Department of Computer Science

University of North Carolina, Chapel Hill


Shape from Stereo

What do we want to do?

Measure 3D point location from two images of the point (from different viewpoints).


Shape from Stereo

How?

• Acquire two images (simultaneously, if the scene is not stationary)

• Determine camera geometry

• Find homologous (corresponding) points, i.e. points in both images which are an image of the same world point.

• Triangulate homologous points, i.e. determine world point from homologous points and camera geometry


(i) We have the camera geometry and an image point, how does this constrain the location of the corresponding image point in the second image? (Correspondence geometry)

(ii) We have corresponding image points, how does this constrain camera geometry? (Motion)

(iii) We have the camera geometry and corresponding image points, what are the 3D point coordinates? (Correspondence geometry)

Questions


What if only C, C’, x are known?

Question i: Epipolar Geometry and F


Family of planes and lines l and l’ Intersection in e and e’

Epipolar Geometry


algebraic representation of epipolar geometry

l'x

we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

Question i: Epipolar Geometry and F


geometric derivation

xHx' π

x'e'l' FxxHe' π

mapping from 2-D to 1-D family (rank 2)

The Fundamental Matrix (F)

0

0

0

12

13

23

aa

aa

aa

a


algebraic derivation

λCxPλX IPP

PP'e'F

xPP'CP'l

(note: doesn’t work for C=C’ F=0)

xP

λX



correspondence condition

0Fxx'T

The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images

0l'x'T



Mapping of corresponding epipolar lines: derivation

l,l’ epipolar lines, k line not through e l’=F[k]xl and symmetrically l=FT[k’]xl’

lke

kl lFk

e'

(pick k=e, since eTe≠0)

leFl' l'e'Fl T

Epipolar Line Geometry


-T -13x3 3x3 3x3 3x3

ˆˆ ˆx H x, x' H 'x' F H ' FH

Derivation based purely on projective concepts

1 -14 4 4 4 4 4 4 4

ˆ ˆx PX PH H X PXx x x xPH H X

F invariant to transformations of projective 3-space

1 14 4 4 4 4 4 4 4

ˆ ˆx' P'X ' ( ' )( ) P'Xx x x xP H H X P H H X

FP'P,

P'P,F

unique

not unique

canonical form

m]|[MP'0]|[IP

MmF

Invariance under projective transformation

without derivation


F matrix corresponds to P,P’ iff P’TFP is skew-symmetric

X0,FPXP'X TT

Possible choice:

]e'|F][[e'P' 0]|[IP

Canonical representation:T

3x1P [I|0] P' [[e'] F e'v |λe']

Canonical cameras given F

Arbitrary 3-vector Arbitrary scalar


C1

C2

l2

P

l1

e1

e2

0m m 1T2 F

Fundamental matrix (3x3 rank 2

matrix)

1. Computable from corresponding points

2. Simplifies matching3. Allows to detect wrong matches4. Related to calibration

Underlying structure in set of matches for rigid scenes

l2

C1m1

L1

m2

L2

M

C2

m1

m2

C1

C2

l2

P

l1

e1

e2

m1

L1

m2

L2

M

l2lT1

Canonical representation:

]λe'|ve'F][[e'P' 0]|[IP T

Epipolar Geometry


given xi↔x‘i , compute P,P‘ and Xi

reconstruction problem:

ii PXx x P'Xi i for all i

without additional information possible up to projective ambiguity

3D reconstruction of cameras and structure


(i) Compute F from correspondences(ii) Compute camera matrices from F(iii) Compute 3D point for each pair of

corresponding points

computation of Fuse x‘iFxi=0 equations, linear in coeff. F8 points (linear), 7 points (non-linear), 8+ (least-squares)

computation of camera matricesuse ]λe'|ve'F][[e'P' 0]|[IP T

triangulationcompute intersection of two backprojected rays

Outline of reconstruction


iii XHPHPXx P-1

P

Reconstruction ambiguity: projective


If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent

i111 X,'P,P i222 X,'P,Pii xx -1

12 HPP -112 HPP

12 HXX 0FxFx :except ii

iiiii 22111-1

112 XPxXPHXHPHXP along same ray of P2, idem for P‘2

two possibilities: X2i=HX1i, or points along baseline

key result: allows reconstruction from pair of uncalibrated images

The projectove reconstruction theorem



(i) Projective reconstruction

(ii) Affine reconstruction

(iii) Metric reconstruction

Stratified reconstruction


ii HXXE

use control points XEi with known coordinatesto go from projective to metric

Eii XPHx -1(2 lin. eq. in H-1

per view, 3 for two views)

Direct metric reconstruction using ground truth


0Fxx'T

separate known from unknown

0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx

0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx

(data) (unknowns)(linear)

0Af

0f1''''''

1'''''' 111111111111

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

Computation of F: basic equation


0Fe'T 0Fe 0detF 2Frank

T333

T222

T111

T

3

2

1

VσUVσUVσUVσ

σσ

UF

SVD from linearly computed F matrix (rank 3)

T222

T111

T2

1

VσUVσUV0

σσ

UF'

FF'-FminCompute closest rank-2 approximation

Imposing the singularity constraint



0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

f

f

f

f

f

f

f

f

f

yxyyyyxxxyxx

yxyyyyxxxyxx

yxyyyyxxxyxx

nnnnnnnnnnnn

~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100

!Orders of magnitude differenceBetween column of data matrix least-squares yields poor results

The NOT normalized 8-point algorithm


Transform image to ~[-1,1]x[-1,1]

(0,0)

(700,500)

(700,0)

(0,500)

(1,-1)

(0,0)

(1,1)(-1,1)

(-1,-1)

1

1500

2

10700

2

Least squares yields good results (Hartley, PAMI´97)

The normalized 8-point algorithm


Maximum Likelihood Estimation

i

iiii dd 22 'x̂,x'x̂,x

(= least-squares for Gaussian noise)

0x̂F'x̂ subject to T

iXt],|[MP'0],|[IP Parameterize:

Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi

iiii XP'x̂,PXx̂

Minimize cost using Levenberg-Marquardt(preferably sparse LM, see book)

(overparametrized)

The Gold Standard Algorithm


Examples


Examples


Examples


1. Do not use unnormalized algorithms

2. Quick and easy to implement: 8-point normalized

3. Better: enforce rank-2 constraint during minimization

4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

Recommendations


Extract feature points to relate images

Required properties:– Well-defined

(i.e. neigboring points should all be different)

– Stable across views

(i.e. same 3D point should be extracted as feature for neighboring viewpoints)

The correspondence problem: feature points


homogeneous

edge

corner

M should have large eigenvalues

(e.g.Harris&Stephens´88; Shi&Tomasi´94)

Find points that differ as much as possible from all neighboring points

Feature = local maxima (subpixel) of F(1, 2)

Feature points


Select strongest features (e.g. 1000/image)

Feature points


Evaluate NCC for all features with

similar coordinates

Keep mutual best matchesKeep mutual best matches

Still many wrong matches!Still many wrong matches!

10101010 ,,´´, e.g. hhww yyxxyx

?

Feature matching


0.96 -0.40 -0.16 -0.39 0.19

-0.05 0.75 -0.47 0.51 0.72

-0.18 -0.39 0.73 0.15 -0.75

-0.27 0.49 0.16 0.79 0.21

0.08 0.50 -0.45 0.28 0.99

1 5

24

3

1 5

24

3

Gives satisfying results for small image motions

Similarity Example


Step 1. Extract features

Step 2. Compute a set of potential matches

Step 3. doStep 3.1 select minimal sample (i.e. 8 matches)

Step 3.2 compute F

Step 3.3 determine inliers

until (#inliers,#samples)<95%

samples#7)1(1

matches#inliers#

#inliers 90% 80% 70% 60% 50%

#samples 5 13 35 106 382

Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches

(generate hypothesis)

(verify hypothesis)

RANSAC


restrict search range to neighborhood of epipolar line (1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches


Degenerate cases– Planar scene

– Pure rotation

No unique solution– Remaining DOF filled by noise

– Use simpler model (e.g. homography)

Model selection (Torr et al., ICCV´98, Kanatani, Akaike)

– Compare H and F according to expected residual error (compensate for model complexity)

Degenerate Cases


Absence of sufficient features (no texture)

Repeated structure ambiguity

(Schaffalitzky and Zisserman, BMVC‘98)

• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition

More problems

Documents

Robot Vision SS 2007 Matthias Rüther 1 ROBOT VISION Lesson 6a: Shape from Stereo, short summary Matthias Rüther Slides partial courtesy of Marc Pollefeys