Camera models and calibration

Read tutorial chapter 2 and 3.1http://www.cs.unc.edu/~marc/tutorial/

Szeliski’s book pp.29-73

Schedule (tentative)

2

# date topic

1 Sep.17 Introduction and geometry

2 Sep.24 Camera models and calibration

3 Oct.1 Invariant features

4 Oct.8 Multiple-view geometry

5 Oct.15 Model fitting (RANSAC, EM, …)

6 Oct.22 Stereo Matching

7 Oct.29 Structure from motion

8 Nov.5 Segmentation

9 Nov.12 Shape from X (silhouettes, …)

10 Nov.19 Optical flow

11 Nov.26 Tracking (Kalman, particle filter)

12 Dec.3 Object category recognition

13 Dec.10 Specific object recognition

14 Dec.17 Research overview

Brief geometry reminder

3

0=xlT

l'×l=x x'×x=l2D line-point coincidence relation:

Point from lines:

2D Ideal points ( )T0,, 21 xx

2D line at infinity ( )T1,0,0=l∞

0=XπT

0π

X

X

X

3

2

1

T

T

T

0X

π

π

π

3

2

1

T

T

T3D plane-point coincidence relation:

Point from planes: Plane from points:

Line from points:

3D line representation:

(as two planes or two points) 2x2

PA B 0

Q

T

T

3D Ideal points 1 2 3, , ,0X X XT

3D plane at infinity 0,0,0,1 T

Conics and quadrics

4

l=Cx

l

xC

1-* CC

0=QXXT 0=πQπ *T

Conics

Quadrics

2D projective transformations

A projectivity is an invertible mapping h from P2 to itself such that three points x1,x2,x3 lie on the same line if and

only if h(x1),h(x2),h(x3) do.

Definition:

A mapping h:P2P2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P2 reprented by a vector x it is true that h(x)=Hx

Theorem:

Definition: Projective transformation

3

2

1

333231

232221

131211

3

2

1

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x=x' Hor

8DOF

projectivity=collineation=projective transformation=homography

Transformation of 2D points, lines and conics

Transformation for lines

l=l' -TH

Transformation for conics-1-TCHHC ='

Transformation for dual conicsTHHCC ** ='

x=x' HFor a point transformation

Fixed points and lines

e=e λH (eigenvectors H =fixed points)

l lH-T (eigenvectors H-T =fixed lines)

(1=2 pointwise fixed line)

Hierarchy of 2D transformations

1002221

1211

y

x

taa

taa

1002221

1211

y

x

tsrsr

tsrsr

333231

232221

131211

hhh

hhh

hhh

1002221

1211

y

x

trr

trr

Projective8dof

Affine6dof

Similarity4dof

Euclidean3dof

Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio

Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l∞

Ratios of lengths, angles.The circular points I,J

lengths, areas.

invariantstransformed squares

The line at infinity

00

l l 0 lt 1

1A

AH

A

TT

T T

The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity

Note: not fixed pointwise

Affine properties from images

projection rectification

APA

lll

HH

321

010

001

0≠,l 3321∞ llll T

Affine rectificationv1 v2

l1

l2 l4

l3

l∞

21∞ vvl

211 llv 432 l×l=v

The circular points

0

1

I i

0

1

J i

I

0

1

0

1

100

cossin

sincos

II

iseitss

tssi

y

x

S

H

The circular points I, J are fixed points under the projective transformation H iff H is a similarity

The circular points

“circular points”

0=++++ 233231

22

21 fxxexxdxxx 0=+ 2

221 xx

l∞

T

T

0,-,1J

0,,1I

i

i

( ) ( )TT 0,1,0+0,0,1=I i

Algebraically, encodes orthogonal directions

03 x

Conic dual to the circular points

TT JIIJ*∞ C

000

010

001*∞C

TSS HCHC *

∞*∞

The dual conic is fixed conic under the projective transformation H iff H is a similarity

*∞C

Note: has 4DOF

l∞ is the nullvector

*∞C

l∞

I

J

Angles

( )( )22

21

22

21

2211

++

+=cos

mmll

mlmlθ

( )T321 ,,=l lll ( )T

321 ,,=m mmmEuclidean:

Projective:

(orthogonal)

Transformation of 3D points, planes and quadrics

Transformation for lines

( )l=l' -TH

Transformation for quadrics

( )-1-TCHHC ='

Transformation for dual quadrics

( )THHCC ** ='

( )x=x' H

For a point transformation

X=X' H

π=π' -TH

-1-TQHH=Q'

THHQ=Q' **

(cfr. 2D equivalent)

Hierarchy of 3D transformations

vTv

tAProjective15dof

Affine12dof

Similarity7dof

Euclidean6dof

Intersection and tangency

Parallellism of planes,Volume ratios, centroids,The plane at infinity π∞

Angles, ratios of length

The absolute conic Ω∞

Volume

10

tAT

10

tRT

s

10

tRT

The plane at infinity

0

00π π π

0- t 1

1

A

AH

A

TT

T T

The plane at infinity π is a fixed plane under a projective transformation H iff H is an affinity

1. canonical position2. contains directions 3. two planes are parallel line of intersection in π∞

4. line // line (or plane) point of intersection in π∞

( )T1,0,0,0=π∞

( )T0,,,=D 321 XXX

The absolute conic

The absolute conic Ω∞ is a fixed conic under the projective transformation H iff H is a similarity

04

23

22

21

X

XXX

The absolute conic Ω∞ is a (point) conic on π.

In a metric frame:

T321321 ,,I,, XXXXXXor conic for directions:(with no real points)

1. Ω∞ is only fixed as a set2. Circle intersect Ω∞ in two circular points3. Spheres intersect π∞ in Ω∞

The absolute dual quadric

00

0I*∞ T

The absolute dual quadric Ω*∞ is a fixed conic under

the projective transformation H iff H is a similarity

1. 8 dof2. plane at infinity π∞ is the nullvector of Ω∞

3. Angles:

( )( )2

*∞21

*∞1

2*∞1

πΩππΩπ

πΩπ=cos

TT

T

θ

Camera model

Relation between pixels and rays in space

?

Pinhole camera

Gemma Frisius, 1544

23

Distant objects appear smaller

24

Parallel lines meet

• vanishing point

25

Vanishing points

VPL VPRH

VP1VP2

VP3

To different directions correspond different vanishing points

26

Geometric properties of projection

• Points go to points• Lines go to lines• Planes go to whole image

or half-plane• Polygons go to polygons

• Degenerate cases:– line through focal point yields point– plane through focal point yields line

Pinhole camera model

TT ZfYZfXZYX )/,/(),,(

101

0

0

1

Z

Y

X

f

f

Z

fY

fX

Z

Y

X

linear projection in homogeneous coordinates!

Pinhole camera model

101

0

0

Z

Y

X

f

f

Z

fY

fX

101

01

01

1Z

Y

X

f

f

Z

fY

fX

PXx

0|I)1,,(diagP ff

Principal point offset

Tyx

T pZfYpZfXZYX )+/,+/(),,(

principal pointT

yx pp ),(

Principal point offset

camX0|IKx

1y

x

pf

pf

K calibration matrix

Camera rotation and translation

( )C~

-X~

R=X~

cam

X10

RC-R

1

10

C~

R-RXcam

Z

Y

X

camX0|IKx XC~

-|IKRx

t|RKP C~

R-t PX=x

~

CCD camera

1yy

xx

p

p

K

11y

x

y

x

pf

pf

m

m

K

General projective camera

1yx

xx

p

ps

K

1yx

xx

p

p

K

C~

|IKRP

non-singular

11 dof (5+3+3)

t|RKP

intrinsic camera parametersextrinsic camera parameters

Radial distortion

• Due to spherical lenses (cheap)• Model:

R

2 2 2 2 21 2( , ) (1 ( ) ( ) ...)

xx y K x y K x y

y

R

http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpgstraight lines are not straight anymore

Camera model

Relation between pixels and rays in space

?

Projector model

Relation between pixels and rays in space

(dual of camera)

(main geometric difference is vertical principal point offset to reduce keystone effect)

?

Meydenbauer camera

vertical lens shiftto allow direct ortho-photographs

Affine cameras

Action of projective camera on points and lines

forward projection of line

μbaμPBPAμB)P(AμX

back-projection of line

lPT

PXlX TT PX x0;xlT

PXx projection of point

Action of projective camera on conics and quadricsback-projection to cone

CPPQ Tco 0CPXPXCxx TTT

PXx

projection of quadric

TPPQC ** 0lPPQlQ T*T*T

lPT

ii xX ? P

Resectioning

Direct Linear Transform (DLT)

ii PXx ii PXx

rank-2 matrix

Direct Linear Transform (DLT)

Minimal solution

Over-determined solution

5½ correspondences needed (say 6)

P has 11 dof, 2 independent eq./points

n 6 points

Apminimize subject to constraint

1p

use SVD

Degenerate configurations

(i) Points lie on plane or single line passing through projection center

(ii) Camera and points on a twisted cubic

Scale data to values of order 1

1. move center of mass to origin2. scale to yield order 1 values

Data normalization

D3

D2

Line correspondences

Extend DLT to lines

ilPT

ii 1TPXl

(back-project line)

ii 2TPXl (2 independent eq.)

Geometric error

Gold Standard algorithm

ObjectiveGiven n≥6 2D to 3D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P

Algorithm

(i) Linear solution:

(a) Normalization:

(b) DLT

(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:

(iii) Denormalization:

ii UXX~ ii Txx~

UP~

TP -1

~ ~~

Calibration example

(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners

typically precision <1/10

(H&Z rule of thumb: 5n constraints for n unknowns)

Errors in the world

Errors in the image and in the world

ii XPx

iX

Errors in the image

iPXx̂

i

(standard case)

Restricted camera estimation

Minimize geometric error impose constraint through parametrization

Find best fit that satisfies• skew s is zero• pixels are square • principal point is known• complete camera matrix K is known

Minimize algebraic error assume map from param q P=K[R|-RC], i.e. p=g(q)minimize ||Ag(q)||

Restricted camera estimation

Initialization • Use general DLT• Clamp values to desired values, e.g. s=0, x= y

Note: can sometimes cause big jump in error

Alternative initialization• Use general DLT• Impose soft constraints

• gradually increase weights

Image of absolute conic

A simple calibration device

(i) compute H for each square (corners (0,0),(1,0),(0,1),(1,1))

(ii) compute the imaged circular points H(1,±i,0)T

(iii) fit a conic to 6 circular points(iv) compute K from through cholesky factorization

(≈ Zhang’s calibration method)

Some typical calibration algorithmsTsai calibration

Zhangs calibration

http://research.microsoft.com/~zhang/calib/

Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.

Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.

http://www.vision.caltech.edu/bouguetj/calib_doc/

81 from Szeliski’s book

82

Next week:Image features