Geometry for Computer Vision - Linköping University · May 20, 2014 Computer Vision lecture 4a Computer Vision Laboratory Camera calibration Zhang’s camera calibration (A flexible

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Geometry for Computer VisionLecture 4a

Calibration and OrientedEpipolar Geometry

1

Per-Erik Forssén



Overview1. Lens effects (distortion, vignetting)2. Extrinsic and intrinsic camera parameters3. Zhang’s camera calibration4. Calibrated epipolar geometry (intro)5. Oriented epipolar geometry

Break

2



The pin-hole camera

A brightly illuminated scene will be projected onto a wall opposite of the pin-hole.

The image is rotated .

3

180�



The pin-hole camera

• From similar triangles we get:

4

x = fX

Zy = f

Y

Z



The pin-hole camera

• From similar triangles we get:

5

�

�

⇤xy1

⇥

⌅ =

�

⇤f 0 00 f 00 0 1

⇥

⌅

�

⇤XYZ

⇥

⌅

• More generally, we write:

• f-focal length, s-skew, a-aspect ratio,(cx,cy)-projection of optical centre



The pin-hole camera

6

�

�

⇤xy1

⇥

⌅ =

�

⇤f 0 00 f 00 0 1

⇥

⌅

�

⇤XYZ

⇥

⌅

�

�

⇤xy1

⇥

⌅ =

�

⇤f s cx

0 fa cy

0 0 1

⇥

⌅

�

⇤XYZ

⇥

⌅



�

�

⇤xy1

⇥

⌅

⌥⌃⇧�x

=

�

⇤f s cx

0 fa cy

0 0 1

⇥

⌅

⌥ ⌃⇧ �K

�

⇤XYZ

⇥

⌅

⌥ ⌃⇧ �X̃

The pin-hole camera

x � KX̃

7



�

�

⇤xy1

⇥

⌅

⌥⌃⇧�x

=

�

⇤f s cx

0 fa cy

0 0 1

⇥

⌅

⌥ ⌃⇧ �K

�

⇤XYZ

⇥

⌅

⌥ ⌃⇧ �X̃

The pin-hole camera

Motivation:

f-focal length, s-skew, a-aspect ratio,(cx,cy)-projection of optical centre

x � KX̃

8



Thin Lens CameraReal cameras use lenses, not pin-holes!

9



Thin Lens CameraA thin lens is a (positive) lens with

Parallel rays converge at the focal pointsRays through the optical centre are not

refracted

d << f

10



Thin Lens Camera

Thin lens relation (from similar triangles):1f

=1Z

+1l

11



Thin Lens Camera

- Focus at one depth only. - Objects at other depths are blurred.

12



Thin Lens Camera

An aperture increases the depth-of-field, the range which is sharp in the image.

A compromise between pinhole and thin lens.13



Lens effects

• Radial distortion• For zoom lenses: Barrel at wide FoV

pin-cushion at narrow FoV

Barrel distortionCorrect Pin-cushion distortion

14

)



Lens effects

• Modelling• Used in optimisation such as BA

DistortedCorrect image

15

x ⇠ Kf(u,⇥0)

)



Lens effects

• Rectification• Used in dense stereo

Distorted image Correct

16

x

0 ⇠ f�1(x,⇥)



Distorsion polynomials• Different models for different classes

of cameras• Radial model for normal and telecentric

lenses with moderate distortion

• Also model centre of distortion17

r =qx

21 + x

22 ' = atan2(x2, x1)

x

01 = r

0cos'

0x

02 = r

0sin'

0r0 = ✓1r + ✓2r3 . . . '0 = '



Distorsion polynomials• For better accuracy:• Tangential distorsion• Rational model [Claus & Fitzgibbon 05]

• Specialised models:• wide-angle cameras [Kannala&Brandt 06]• catadioptric cameras [Micusik&Pajdla 03]• most simple is the FoV model [Devernay &

Faugeras 2001]:

18

r0 = atan(r✓1)/✓1



Lens effects

• Vignetting and cos4-law• Stronger effects in wide FoV

Darkened peripheryCorrect

19



Lens effectsVignetting

20



Lens effectsVignetting

cos4-lawdampening withcos4(w)

http://software.canon-europe.com/files/documents/EF_Lens_Work_Book_10_EN.pdf

21





For a general position of the world coordinate system (WCS) we have:

x � K

�

⇤r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t3

⇥

⌅

�⌥ ⌦[R|t]

�

⇧⇧⇤

XYZ1

⇥

⌃⌃⌅

�⌥ ⌦X

22

Camera parameters



For a general position of the world coordinate system (WCS) we have:

K contains the intrinsic parameters[R | t] contain the extrinsic parameters

x � K

�

⇤r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t3

⇥

⌅

�⌥ ⌦[R|t]

�

⇧⇧⇤

XYZ1

⇥

⌃⌃⌅

�⌥ ⌦X

23

Camera parameters



Camera parametersMetric points transformed to the camera’s

coordinate system are called normalised image coordinates

In contrast to regular image coordinates

K contains the intrinsic parameters[R | t] contain the extrinsic parameters

24

x ⇠ K [R|t]X

x̂ ⇠ [R|t]X

x = Kx̂



Camera calibrationZhang’s camera calibration (A flexible new

technique for camera calibration, PAMI 2000)

In OpenCV, and in Matlab toolboxFinds K from 3 or more photos of a planar

calibration targetModerate lens

distorsion can alsobe estimated.

25



Camera calibrationWe now imagine a world coordinate

system fixed to the planar target

�

�

⇤xy1

⇥

⌅ = K

�

⇤r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t3

⇥

⌅

�

⇧⇧⇤

XYZ1

⇥

⌃⌃⌅

26



Camera calibrationWe now imagine a world coordinate

system fixed to the planar target

�

�

⇤xy1

⇥

⌅ = K

�

⇤r11 r12 t1r21 r22 t2r31 r32 t3

⇥

⌅

�

⇤XY1

⇥

⌅ =

�

⇤h11 h12 h13

h21 h22 h23

h31 h32 h33

⇥

⌅

⌥ ⌃⇧ �H

�

⇤XY1

⇥

⌅

27



Camera calibration

If we estimate a homography between the image and the model plane (lecture 3) we know H

We also know that and

28

rT1 r2 = 0 rT

1 r1 = rT2 r2

H = [h1 h2 h3] = K [r1 r2 t]



Camera calibration

If we estimate a homography between the image and the model plane (lecture 3) we know H

We also know that and

29

rT1 r2 = 0 rT

1 r1 = rT2 r2

H = [h1 h2 h3] = K [r1 r2 t]

hT1 K�T K�1h2 = 0

hT1 K�T K�1h1 = hT

2 K�T K�1h2

)



Camera calibration

For a K of the formIt can be shown that (use e.g. Maple)

30

K�T K�1 = B =

2

64

1↵2 � �

↵2�v0��u0�

↵2�

� �↵2�

�2

↵2�2 + 1�2 ��(v0��u0�)

↵2�2 � v0�2

v0��u0�↵2� ��(v0��u0�)

↵2�2 � v0�2

(v0��u0�)2

↵2�2 + v20

�2 + 1

3

75

K =

2

4↵ � u0

0 � v0

0 0 1

3

5



Camera calibration

For a K of the formIt can be shown that (use e.g. Maple)

Remember our constraints and

31

K�T K�1 = B =

2

64

1↵2 � �

↵2�v0��u0�

↵2�

� �↵2�

�2

↵2�2 + 1�2 ��(v0��u0�)

↵2�2 � v0�2

v0��u0�↵2� ��(v0��u0�)

↵2�2 � v0�2

(v0��u0�)2

↵2�2 + v20

�2 + 1

3

75

K =

2

4↵ � u0

0 � v0

0 0 1

3

5

hT1 Bh2 = 0 hT

1 Bh1 � hT2 Bh2 = 0



Camera calibration

As B is symmetric

32

B =

2

4b1 b2 b4

b2 b3 b5

b4 b5 b6

3

5



Camera calibration

As B is symmetric

If we now defineThe constraints can be written as

33

B =

2

4b1 b2 b4

b2 b3 b5

b4 b5 b6

3

5

b =⇥b1 b2 b3 b4 b5 b6

⇤T

vT

12

(v11 � v22)T

�b = 0

vij = [hi1hj1, hi1hj2 + hi2hj1, hi2hj2, hi3hj1 + hi1hj3, hi3hj2 + hi2hj3, hi3hj3]T



Camera calibration

Each view of the plane gives us two rows in the system:

As b has 6 unknowns, we need 3 views of the plane.

Two views can also work if we require

34

Vb = 0

� = 0



Camera calibrationOnce b has been estimated, we can extract the

parameters in K according to

35

v0 = (b2b4 � b1b5)/(b1b3 � b22)

� = b6 � (b23 + v0(b2b4 � b1b5)/b1

� = �b2↵2�/�

u0 = �v0↵� b4↵2/�

↵ =p

�/b1

� =q

�b1/(b1b3 � b22)



Camera calibrationOnce b has been estimated, we can extract the

parameters in K according to

The H&Z book instead suggests Cholesky factorisation36

v0 = (b2b4 � b1b5)/(b1b3 � b22)

� = b6 � (b23 + v0(b2b4 � b1b5)/b1

� = �b2↵2�/�

u0 = �v0↵� b4↵2/�

↵ =p

�/b1

� =q

�b1/(b1b3 � b22)



Camera calibrationCholesky factorisation of B(b)

Gives us which is invertible.

37

B(b) = K�1TK�1

K�1



Camera calibration

Once K is computed we can also find the extrinsic camera parameters R,t for each image:

( )

38

r1 = �K�1h1 r2 = �K�1h2 r3 = r1 ⇥ r2

R = [r1 r2 r3] t = �K�1h3

� = 1/||K�1h1|| = 1/||K�1h2||



Camera calibration


Finally, K, Ri, ti are refined using ML (minimising the cost function)

39

r1 = �K�1h1 r2 = �K�1h2 r3 = r1 ⇥ r2

R = [r1 r2 r3] t = �K�1h3

arg minnX

i=1

mX

j=1

||xij � x̂(K,Ri, ti,Xj)||2

𝚯



Camera calibration


Optionally, all of K, ,Ri, ti are refined using ML:

40

r1 = �K�1h1 r2 = �K�1h2 r3 = r1 ⇥ r2

R = [r1 r2 r3] t = �K�1h3

argminnX

i=1

mX

j=1

kxij � x̂(K,⇥,Ri, ti,Xj)k2



Camera calibration

So what about the initial homographies?

Assign each point a WCS value

41

H = K [r1 r2 t]X = [x y 0]T



Camera calibration

So what about the initial homographies?

Assign each point a WCS value Do we need to know which point is the upper left one on the checker-board? Why not?

42

H = K [r1 r2 t]X = [x y 0]T



Camera calibrationCan we use any combination images of the calibration plane?

43



Camera calibrationCan we use any combination images of the calibration plane?

The constraints used: andhave to be linearly independent.

Planes must not be parallel!

44

rT1 r2 = 0 rT

1 r1 = rT2 r2

)



Calibrated epipolar geometry

Recall the epipolar constraint

45

x

T1 Fx2 = 0



Calibrated epipolar geometry

Recall the epipolar constraint...and the normalised image coordinates

We can instead express the epipolar constraint in normalised coordinates

The matrix E is called the essential matrix.It has some interesting properties...

46

x

T1 Fx2 = 0

x = Kx̂

ˆ

x

T1 K

T1 FK2ˆ

x2 = 0 or

ˆ

x

T1 E

ˆ

x2 = 0



Calibrated epipolar geometryIn lecture 2 we saw that for cameras P1 and P2:

Now, if

We get and

47

P+2 =

K�1

2

0T

�

F = [e12]⇥P1P+2

P2 = K2 [I|0] and P1 = K1 [R|t]

F = [K1t]⇥K1RK�12

e12 = P1O2



Calibrated epipolar geometryUsing the cross-product-commutator rule:

(A4.3)

on

...we may express F as either of

48

F = K�T1 [t]⇥RK�1

2 F = K�T1 R

⇥RT t

⇤⇥K�1

2

F = K�T1 R [t2]⇥K�1

2

[b]⇥A = det(A)A�T⇥A�1b

⇤⇥

F = [K1t]⇥K1RK�12



Calibrated epipolar geometryThis gives us the essential matrix

expressions:

E has only 5 dof (3 from R, 2 from t)recall that F has 7

A necessary and sufficient condition on E is that it has the singular values [a,a,0](see 9.6.1 in the H&Z book for proof)

49

E = [t]⇥R = R⇥RT t

⇤⇥



Calibrated epipolar geometryThis gives us the essential matrix

expressions:

We can extract R and t (up to scale) from Eif we also make use of one point correspondence (a 3D point known to be in front of both cameras). See 9.6.2 in the H&Z book.

50

E = [t]⇥R = R⇥RT t

⇤⇥



Calibrated epipolar geometry4 cases for R and t, just one has point in front

of both cameras.

51



Oriented epipolar geometryThe regular epipolar constraint

ignores the knowledge that points are in frontof the camera.

52

x

T1 Fx2 = 0



Oriented epipolar geometryIn oriented projective geometry a (visible)

point in front of the camera is defined as having a projection

and a (hidden) point behind the camera has a projection

53

x = �

⇥x1 x2 1

⇤Twith � > 0

x = �

⇥x1 x2 1

⇤Twith � < 0



Oriented epipolar geometryThe oriented epipolar constraint properly

distinguishes points in front of and behind the camera

54

�e1 ⇥ x1 = Fx2 , � 2 R+



Oriented epipolar geometryThe oriented epipolar constraint can be

interpreted as comparing oriented lines and

(NB! image planes drawn in front of cameras)55

�e1 ⇥ x1 Fx2



Oriented epipolar geometryLine normalisation is not unique

The extra information in the sign can be used to encode the line orientation.

56

normD(l) =⇥cos↵ sin↵ �⇢

⇤T

normD(�l) =⇥� cos↵ � sin↵ ⇢

⇤T



Oriented epipolar geometryUsage:The oriented epipolar constraint can be used

to quickly reject a hypothesized F inside a RANSAC loop.

See today’s paper: Chum, Werner and Matas, Epipolar Geometry Estimation via RANSAC benefits from the Oriented Epipolar Constraint, ICPR04

57

Documents

Geometry for Computer Vision - Linköping University · May 20, 2014 Computer Vision lecture 4a Computer Vision Laboratory Camera calibration Zhang’s camera calibration (A flexible