63
Structure from motion • Multi-view geometry • Affine structure from motion • Projective structure from motion Planches : http://www.di.ens.fr/~ponce/geomvis/l ect4.ppt http://www.di.ens.fr/~ponce/geomvis/l

Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Embed Size (px)

DESCRIPTION

Epipolar Constraint: Calibrated Case Essential Matrix (Longuet-Higgins, 1981)

Citation preview

Page 1: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Structure from motion

• Multi-view geometry• Affine structure from motion• Projective structure from motion

Planches :– http://www.di.ens.fr/~ponce/geomvis/lect4.ppt – http://www.di.ens.fr/~ponce/geomvis/lect4.pdf

Page 2: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Epipolar Constraint

• Potential matches for p have to lie on the corresponding epipolar line l’.

• Potential matches for p’ have to lie on the corresponding epipolar line l.

Page 3: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Epipolar Constraint: Calibrated Case

Essential Matrix(Longuet-Higgins, 1981)

Page 4: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Properties of the Essential Matrix

• E p’ is the epipolar line associated with p’.

• E p is the epipolar line associated with p.

• E e’=0 and E e=0.

• E is singular.

• E has two equal non-zero singular values (Huang and Faugeras, 1989).

T

T

Page 5: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Epipolar Constraint: Small MotionsTo First-Order:

Pure translation:Focus of Expansion

Page 6: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Epipolar Constraint: Uncalibrated Case

Fundamental Matrix(Faugeras and Luong, 1992)

Page 7: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Properties of the Fundamental Matrix

• F p’ is the epipolar line associated with p’.

• F p is the epipolar line associated with p.

• F e’=0 and F e=0.

• F is singular.

T

T

Page 8: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Eight-Point Algorithm (Longuet-Higgins, 1981)

|F | =1.

Minimize:

under the constraint2

Page 9: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Non-Linear Least-Squares Approach (Luong et al., 1993)

Minimize

with respect to the coefficients of F , using an appropriate rank-2 parameterization.

Page 10: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Normalized Eight-Point Algorithm (Hartley, 1995)

• Center the image data at the origin, and scale it so themean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’.

• Use the eight-point algorithm to compute F from thepoints q and q’ .

• Enforce the rank-2 constraint.

• Output T F T’.T

i i i i

i i

Page 11: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Data courtesy of R. Mohr and B. Boufama.

Page 12: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

With

out n

orm

aliz

atio

nW

ith n

orm

aliz

atio

nMean errors:10.0pixel9.1pixel

Mean errors:1.0pixel0.9pixel

Page 13: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Trinocular Epipolar Constraints

These constraints are not independent!

Page 14: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Trinocular Epipolar Constraints: Transfer

Given p and p , p can be computed

as the solution of linear equations.

1 2 3

Page 15: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Trifocal Constraints

Page 16: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Trifocal Constraints

All 3x3 minorsmust be zero!

Calibrated Case

Trifocal Tensor

Page 17: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Trifocal ConstraintsUncalibrated Case

Trifocal Tensor

Page 18: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Trifocal Constraints: 3 Points

Pick any two lines l and l through p and p .Do it again.

2 3 2 3T( p , p , p )=01 2 3

Page 19: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Properties of the Trifocal Tensor

Estimating the Trifocal Tensor

• Ignore the non-linear constraints and use linear least-squaresa posteriori.

• Impose the constraints a posteriori.

• For any matching epipolar lines, l G l = 0.

• The matrices G are singular.

• They satisfy 8 independent constraints in theuncalibrated case (Faugeras and Mourrain, 1995).

2 1 3T i

1i

Page 20: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

For any matching epipolar lines, l G l = 0. 2 1 3T i

The backprojections of the two lines do not define a line!

Page 21: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Multiple Views (Faugeras and Mourrain, 1995)

Page 22: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Two Views

Epipolar Constraint

Page 23: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Three Views

Trifocal Constraint

Page 24: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Four Views

Quadrifocal Constraint(Triggs, 1995)

Page 25: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Geometrically, the four rays must intersect in P..

Page 26: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Quadrifocal Tensorand Lines

Page 27: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Scale-Restraint Condition from Photogrammetry

Page 28: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Euclidean (perspective) Structure-from-Motion Problem

Given m calibrated perspective images of n fixed points Pj we can write

Problem: estimate the m 3x4 matrices Mi = [Ri ti] and

the n positions Pj from the mn correspondences pij .

2mn equations in 11m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

Page 29: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Euclidean Ambiguity of Euclidean SFM

If Ri, ti, and Pj are solutions,

So are Ri’, ti’, and Pj’, where

In fact, the absolute scale cannot be recovered since:

When the intrinsic and extrinsic parameters are known

Euclidean ambiguity up to a similarity transformation.

Page 30: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Affine Structure-from-Motion Problem

Given m images of n fixed points P we can write

Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .

ij ij

2mn equations in 8m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

Page 31: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Affine Ambiguity of Affine SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and

Q is an affinetransformation.

When the intrinsic and extrinsic parameters are unknown

Page 32: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Affine Epipolar Constraint

Note: the epipolar lines are parallel.

Page 33: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Affine Epipolar Geometry

Page 34: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Affine Fundamental Matrix

where

Page 35: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

With

out n

orm

aliz

atio

nW

ith n

orm

aliz

atio

nMean errors:10.0pixel9.1pixel

Mean errors:1.0pixel0.9pixel

Perspective case..

Page 36: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Mean errors: 3.24 and 3.15pixel (without normalization160.92 and 158.54pixel).

Affine case..

Page 37: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

An Affine Trick..

Page 38: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Affine Epipolar Constraint

Note: the epipolar lines are parallel.

Page 39: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

An Affine Trick.. Algebraic Scene Reconstruction Method

Page 40: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Affine reconstruction. Mean relative error: 3.2%

Page 41: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

The Affine Structure of Affine Images

Suppose we observe a static scene with m fixed cameras..

The set m-tuples of allimage points in a sceneis a 3D affine space!

Page 42: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

When do m+1 points define a p-dimensional subspace Y of ann-dimensional affine space X equipped with some coordinateframe basis?

Writing that all minors of size (p+2)x(p+2) of D are equal tozero gives the equations of Y.

Rank ( D ) = p+1, where

has rank 4!

Page 43: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

From Affine to Vectorial Structure

Idea: pick one of the points (or their center of mass)as the origin.

Page 44: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Singular Value Decomposition

Page 45: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Singular Value Decomposition square roots of

Page 46: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Singular Value Decomposition

Page 47: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Singular Value Decomposition

Page 48: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

What if we could factorize D? (Tomasi and Kanade, 1992)

Affine SFM is solved!

Singular Value Decomposition

We can take

Page 49: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Affine reconstruction. Mean relative error: 2.8%

Page 50: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Back to perspective:Euclidean motion from E (Longuet-Higgins, 1981)

• Given F computed from n > 7 point correspondences, and its SVD F= UWVT, compute E=U diag(1,1,0) VT.

• There are two solutions t’ = u3 and t’’ = -t’ to ETt=0.

• Define R’ = UWVT and R” = UWTVT where

(It is easy to check R’ and R” are rotations.)

• Then [tx’]R’ = -E and [tx’]R” = E. Similar reasoning for t”.

• Four solutions. Only two of them place the reconstructedpoints in front of the cameras.

100001010

W

Page 51: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Euclidean reconstruction. Mean relative error: 3.1%

Page 52: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

A different view of the fundamental matrix

• Projective ambiguity ! M’Q=[Id 0] MQ=[A b].

• Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1)T.

• This can be rewritten as: zp = ( A [Id 0] + [0 b] ) P = z’Ap’ + b.

• Or: z (b x p) = z’ (b x Ap’).

• Finally: pTFp’ = 0 with F = [bx] A.

Page 53: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Projective motion from the fundamental matrix

• Given F computed from n > 7 point correspondences, compute b as the solution of FTb=0 with |b|2=1.

• Note that: [ax]2 = aaT - |a|2Id for any a.

• Thus, if A0 = - [bx] F,

[bx] A0 = - [bx]2 F = - bbTF + |b|2 F = F.

• The general solution is M = [A b] with

A = A0 + ( b | b | b).

Page 54: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Two-view projective reconstruction. Mean relative error: 3.0%

Page 55: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Bundle adjustment

Use nonlinear least-squares to minimize:

Page 56: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Bundle adjustment. Mean relative error: 0.2%

Page 57: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Projective SFM from multiple images

z11p11 … z1np1n

… … …zm1pm1 … zmnpmn

M1

…Mm

P_1 … P_n= , D = MP

• If the zij’s are known, can be done via SVD. In principlethe zij’s can be found pairwise from F (Triggs 96).

• Alternative, eliminate zij from the minimization of E=|D-MP|2

• This reduces the problem to the minimization ofE = ij |pij x MiPj|2

under the constraints |Mi|2=|Pj|2=1 with |pij|2=1.

• Bilinear problem.

Page 58: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Bilinear projective reconstruction. Mean relative error: 0.2%

Page 59: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

From uncalibrated to calibrated cameras

Weak-perspective camera:

Calibrated camera:

Problem: what is Q ?

Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.

Page 60: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Reconstruction Results (Tomasi and Kanade, 1992)

Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.

Page 61: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

What is some parameters are known?

Weak-perspective camera:

Zero skew:

Problem: what is Q ?

0

Self calibration!

Page 62: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

П1

Chasles’ absolute conic: x12+x2

2+x32 = 0, x4 = 0.

Kruppa (1913); Maybank & Faugeras (1992)

Triggs (1997);Pollefeys et al. (1998,2002)

, u0, v0

The absolute quadric u0 = v0 = 0The absolute quadratic complex 2 = 2, = 0

u0

v0

kl

f

x’ ≈ P ( H H-1 ) xH = [ X y ]

Page 63: Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches :  ponce/geomvis/lect4.ppt ponce/geomvis/lect4.ppt

Relation between K, , and *