Single View MetrologyClass 3

3D photography course schedule(tentative)

Lecture Exercise

Sept 26 Introduction -

Oct. 3 Geometry & Camera model Camera calibration

Oct. 10 Single View Metrology Measuring in images

Oct. 17 Feature Tracking/matching(Friedrich Fraundorfer)

Correspondence computation

Oct. 24 Epipolar Geometry F-matrix computation

Oct. 31 Shape-from-Silhouettes(Li Guan)

Visual-hull computation

Nov. 7 Stereo matching Project proposals

Nov. 14 Structured light and active range sensing

Papers

Nov. 21 Structure from motion Papers

Nov. 28 Multi-view geometry and self-calibration

Papers

Dec. 5 Shape-from-X Papers

Dec. 12 3D modeling and registration Papers

Dec. 19 Appearance modeling and image-based rendering

Final project presentations

Single View Metrology

Measuring in a plane

Need to compute H as well as uncertainty

Direct Linear Transformation(DLT)

ii Hxx 0Hxx ii

i

i

i

i

xh

xh

xh

Hx3

2

1

T

T

T

iiii

iiii

iiii

ii

yx

xw

wy

xhxh

xhxh

xhxh

Hxx12

31

23

TT

TT

TT

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

Tiiii wyx ,,x

0hA i

Direct Linear Transformation(DLT)

• Equations are linear in h

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

0AAA 321 iiiiii wyx

0hA i• Only 2 out of 3 are linearly independent

(indeed, 2 eq/pt)

0

h

h

h

x0x

xx0

3

2

1

TTT

TTT

iiii

iiii

xw

yw

(only drop third row if wi’≠0)• Holds for any homogeneous

representation, e.g. (xi’,yi’,1)

Direct Linear Transformation(DLT)

• Solving for H

0Ah 0h

A

A

A

A

4

3

2

1

size A is 8x9 or 12x9, but rank 8

Trivial solution is h=09T is not interesting

1-D null-space yields solution of interestpick for example the one with 1h

Direct Linear Transformation(DLT)

• Over-determined solution

No exact solution because of inexact measurementi.e. “noise”

0Ah 0h

A

A

A

n

2

1

Find approximate solution- Additional constraint needed to avoid 0, e.g.

- not possible, so minimize

1h Ah0Ah

DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is last column of V

(iv) Determine H from h

Importance of normalization

0

h

h

h

0001

1000

3

2

1

iiiiiii

iiiiiii

xyxxxyx

yyyxyyx

~102 ~102 ~102 ~102 ~104 ~104 ~10211

orders of magnitude difference!

Monte Carlo simulation for identity computation based on 5 points

(not normalized ↔ normalized)

Normalized DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

,x~x~ ii inormiinormi xTx~,xTx~

norm-1

norm TH~

TH

1

norm

100

2/0

2/0

T

hhw

whw

Geometric distancemeasured coordinatesestimated coordinatestrue coordinates

xxx

2

HxH,xargminH ii

i

d Error in one image

e.g. calibration pattern

221-

HHx,xxH,xargminH iiii

i

dd Symmetric transfer error

d(.,.) Euclidean distance (in image)

ii

iiiii

ii ddii

xHx subject to

x,xx,xargminx,x,H 22

x,xH,

Reprojection error

Reprojection error

221- Hx,xxHx, dd

22 x,xxx, dd

Statistical cost function and Maximum Likelihood

Estimation• Optimal cost function related to noise

model• Assume zero-mean isotropic Gaussian

noise (assume outliers removed) 22 2/xx,2πσ2

1xPr de

22i 2xH,x /

2iπσ2

1H|xPr ide

i

constantxH,xH|xPrlog 2i2i

σ2

1id

Error in one image

Maximum Likelihood Estimate

2i xH,x id

Statistical cost function and Maximum Likelihood

Estimation• Optimal cost function related to noise

model• Assume zero-mean isotropic Gaussian

noise (assume outliers removed) 22 2/xx,2πσ2

1xPr de

22i

2i 2xH,xx,x /

2iπσ2

1H|xPr

ii dd

ei

Error in both images

Maximum Likelihood Estimate

2i

2i x,xx,x ii dd

Gold Standard algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelyhood Estimation of H

(this also implies computing optimal xi’=Hxi)

Algorithm

(i) Initialization: compute an initial estimate using normalized DLT or RANSAC

(ii) Geometric minimization of reprojection error:

● Minimize using Levenberg-Marquardt over 9 entries of h

or Gold Standard error:

● compute initial estimate for optimal {xi}

● minimize cost over {H,x1,x2,…,xn}

● if many points, use sparse method

2i

2i x,xx,x ii dd

Uncertainty: error in one image

(i) Estimate the transformation from the data(ii) Compute Jacobian , evaluated at(iii) The covariance matrix of the estimated is

given by JJ 1

xT

h

h/XJ f

Hh

h

TT

TT

x~x~0

x~0x~1h/xJ

iii

iii

iii

y

x

w

iiii JJJJ 1T1

xT

h

Uncertainty: error in both images

BBAB

BAAAJJ

1X

T1X

T

1X

T1X

T1

XT

B|AJ

separate in homography and point parameters

nnxx...xx|hP 11

Using covariance matrix in point transfer

Thhhx JJ

Error in one image

Txxx

Thhhx JJJJ

Error in two images

(if h and x independent, i.e. new points)

=1 pixel =0.5cm (Criminisi’97)

Example:

=1 pixel =0.5cm

Example:

(Criminisi’97)

Example:

(Criminisi’97)

Monte Carlo estimation of covariance

• To be used when previous assumptions do not hold (e.g. non-flat within variance) or to complicate to compute.

• Simple and general, but expensive

• Generate samples according to assumed noise distribution, carry out computations, observe distribution of result

Single view measurements:3D scene

Background: Projective geometry of 1D

x'x 22H

The cross ratio

Invariant under projective transformations

T21, xx

3DOF (2x2-1)

02 x

4231

4321

4321 x,xx,x

x,xx,xx,x;x,x

22

11detx,x

ji

ji

ji xx

xx

Vanishing points

• Under perspective projection points at infinity can have a finite image

• The projection of 3D parallel lines intersect at vanishing points in the image

Basic geometry

Basic geometry

• Allows to relate height of point to height of camera

Homology mapping between parallel planes

• Allows to transfer point from one plane to another

Single view measurements

Single view measurements

Forensic applications

190.6±2.9 cm

190.6±4.1 cm

A. Criminisi, I. Reid, and A. Zisserman. Computing 3D euclidean distance from a single view. Technical Report OUEL 2158/98, Dept. Eng. Science, University of Oxford, 1998.

Example

courtesy of Antonio Criminisi

La Flagellazione di Cristo (1460) Galleria Nazionale delle Marche

by Piero della Francesca (1416-1492)

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/

More interesting stuff

• Criminisi demo http://www.robots.ox.ac.uk/~vgg/presentations/spie98/criminis/index.html

• work by Derek Hoiem on learning single view 3D structure and apps http://www.cs.cmu.edu/~dhoiem/

• similar work by Ashutosh Saxena on learning single view depth http://ai.stanford.edu/~asaxena/learningdepth/

Next class

• Feature tracking and matching