1

Ch. 3: Geometric Camera Calibration Objective: Estimates the intrinsic and extrinsic parameters of a camera.Idea: Formulate camera calibration as an optimization process, in which the discrepancy between the theoretical and observed image features is minimized w.r.t. the camera’s parameters.Steps: (1) Evaluate the perspective projection matrix M of the camera, (2) Estimate the intrinsic and extrinsic parameters of the camera from M.

2

,Mp P

。 Perspective Projection (Imaging Process)

practically,

where

ideally,

3

p PM11 12 13 14

21 22 23 24

31 32 33 34 3 4

m m m m

M m m m m

m m m m

○ Evaluate M

Let

Measure n pairs ( , )p Pi i of corresponding

image and scene points.

4

( , ),i ip P

11 12 13 14

21 22 23 24

31 32 33 3411

ii

ii

i

xu m m m m

yv m m m m

zm m m m

11 12 13 14

21 22 23 24

31 32 33 34

,

1

i i i i

i i i i

i i i

m x m y m z m u

m x m y m z m v

m x m y m z m

1, ,i n

For each pair

we obtain

For all pairs,

5

11 1 12 1 13 1 14 1

21 1 22 1 23 1 24 1

31 1 32 1 33 1 34

11 12 13 14

21 22 23 24

31 32 33

1

n n n n

n n n n

n n

m x m y m z m u

m x m y m z m v

m x m y m z m

m x m y m z m u

m x m y m z m v

m x m y m

34 1nz m

,U x yIn matrix form, where

11 12 13 14 31 32 33 34( )x Tm m m m m m m m

1 1 2 2( 1 1 1)Tn nu v u v u v y

6

1 1 1

1 1 1

1 1 1

2 2 2

2 2 2

2 2 2

1 0 0 0 0 0 0 0 0

0 0 0 0 1 0 0 0 0

0 0 0 0 0 0 0 0 1

1 0 0 0 0 0 0 0 0

0 0 0 0 1 0 0 0 0

0 0 0 0 0 0 0 0 1

x y z

x y z

x y z

x y z

x y zU

x y z

1 0 0 0 0 0 0 0 0

0 0 0 0 1 0 0 0 0

0 0 0 0 0 0 0 0 1

n n n

n n n

n n n

x y z

x y z

x y z

Solve for x using optimization techniques

7

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

q q

q q

p p pq q p

u x u x u x y

u x u x u x y

u x u x u x y

3.1 Least-Squares Parameter Estimation

3.1.1 Linear Least-Squares Methods ○ Consider a system of p linear equations in q unknowns:

U x yIdea: Find the solution x by minimizing the squared deviation ( ) from theoretical (Ux) to observed (y) image features

2U x y

8

11 12 1

21 22 2

1 2

q

q

p p pq

u u u

u u uU

u u u

1

2 x

q

x

x

x

1

2 y

p

y

y

y

,

In vector-matrix form, , wherex yU

9

2x yE U

2x y

x x

dE dU

d d

( )2 ( ) 2 ( )

TTd U

U U Ud

0

x yx y x y

x

( ) , ,x y x y x y T T T T TU U U U U U U U 0 01( )x y yT TU U U U

1( )T TU U U U

Let

○ Consider the over-constrained case (p > q) Find x that minimizes the error

The normal equations

: the pseudoinverse of U.

10

xU 0

U x yx 0

1x

U 0xTU U

○ Homogenous systems:

Two issues: (i) By equation , we obtain trivial

(ii) If x is a solution, x is also a

solution.

。 The least squares error solution of

is the eigenvector of

to the smallest eigenvalue.

solution

To resolve the issues, impose

corresponding

11

1x 2

( ) ( ) ( 1)x x x xT TF U U

( )2( ) 2

xx x

xTdF

U Ud

0

( )TU U x xTU U

。 Find the least squares error solution by the method of Lagrange multipliers Error:

We obtain

The solution x is an eigenvector of with eigenvalue

: Lagrange multiplier

Let

2 2( )x x x xT TE U U U U 0

Constraint

Minimize

where

12

( )T T TE U U x x x x xU 0

TU U

The associated error

The least squares error solution to

is the eigenvector of

to the smallest eigenvalue.

corresponding

Example:

Fit a line to a set of data points in the 2D space

13

2 2 2 2 2 2

a b dx y

a b a b a b

a x b y d

2 2 2 2 2 2, ,

a b da b d

a b a b a b

Line equation:

Let

ax by d 2 2 1.a b

14

ax by d ( , )i ix y

| |i iax by d 2

1

( , , ) ( )n

i ii

E a b d ax by d

The perpendicular distance from point

to line

Error measure:

Minimize E w.r.t. (a, b, d)

1

2 ( ) 0n

i ii

Eax by d

d

1

( ) 0,n

i ii

ax by nd

Let

is

1

( )n

i ii

nd ax by

1 1 1

1 1( ) ( )

n n n

i i i ii i i

d ax by ax by ax byn n

15

2 2

1 1

22

1

( ) ( ( ))

[ ( ) ( )] n

n n

i i i ii i

n

i ii

E ax by d ax by ax by

a x x b y y U

,na

b

1 1

n n

x x y y

U

x x y y

where

xU 02

xE U2

Un

1n TU U

。 Recall , whose squared error

The solution of min w.r.t. n under

is the unit eigenvectorconstraint

with the minimum eigenvalue of

16

1 1 2

2 1 2

1 2

( , , , ) 0

( , , , ) 0( )

( , , , ) 0

f x

q

q

p q

f x x x

f x x x

f x x x

0

1 1( , , , ) ,f Tpf f f 1 2( , , , )T

qx x x x

where

3.1.2 Nonlinear Least-Squares Methods

(0,0, ,0)T 0

17

2( ) 3 4 2, ( ) 6 4f x x x f x x

( )( )

df xf x

dx f(x):

11

11

( ) ( ) ( )( )

( ) ( ) ( )

nn

nn

df f ff

d x x

f fx x

x x xx i i

x

i i x x

f(x) :

e.g.,

2( ) ( , ) 3 4 2 8 7,f f x y x y xy x y xe.g.,

22

( ) ( ) ( ) ( ) ( ) ( )

6 4 2 (6 4 2) (3 4 8)

3 4 8

Tf f f f

f fx y x y

xy yxy y x x

x x

x x x xx x i j

i j

18

1 ( )( )

( ) ( )

( )

T

Tp

fd

df

xf x

f x f xx

x

1 1

1

1

( ) ( )

( )

( ) ( )

q

p q

p p

q

f f

x x

f f

x x

f

x x

x

x x

( ) f x : the Jacobian of f where

( ) :f x

19

1 2( , , , )T

qx x x x

2

2

1 1( ) ( ) ( ) (| | )

0! 1!

( ) ( ) (| | )

f x x f x f x x O x

f x f x x O x

。 Taylor expansion of

( ) ( ) ( ) ff x x f x x x

1 2( )Tpf f f f

( ) ( ) ( ) (| )f f f O 2x x x x x x|

around point

f(x):

f(x) :

( ) :f x

20

x( )f x x 0

A. Newton’s Method (Gradient Descent) (i) Square Systems (p = q) Idea: Given an initial x, find s.t.

.

( ) ( ) ( )ff x x f x x x 0( ) ( ) f x x f x

Since

.

( ) f x -1( ) ( ) fx x f x

1 .n n x x xWhen : nonsingular,

Let

( ), f x

Repeat until f(x) stabilizes at some x

• Drawbacks: i) Square system, ii) Nonsingulariii) Locally optimal.

( ) 0 f x

21

( )xE 0Finding x s.t.Finding x s.t. F(x) = 0 (square system)

Since : p by q matrix, f(x): p by q, ( ) f x

( ) ( ) ( )T fF x x f x : q by q

22

where ( )F x : q by q matrix

23

24

25

26

1 2 0 3 0

2 0 3 0

3

cot cot

sin sin

T T Tx y z

T Ty z

Tz

KR K

u t t u t

v t v t

t

t

r r r

r r

r

27

1 2 0 31

2 2 0 3

33

cot

sin

T T TT

T T T

TT

u

v

r r ra

a r r

ar

28

29

30

31

(g) 1 3 2 3

1 3 2 3

( ) ( )cos

a a a a

a a a a

1 2 0 31

2 2 0 3

33

cot

sin

T T TT

T T T

TT

u

v

r r ra

a r r

ar

Proof: From

1 2 0 3

12 0 3

2

3

3

cot

sin

T T T

TT T

T

T

T

u

v

r r r

a r ra

ar

32

1 2 0 3 31 3

1 3 2 3 0 3 32

2 12

cot

( ) cot ( ) ( )

cot

T T T T

T T T T T T

T T

u

u

r r r ra a

r r r r r r

r r

2 0 33

2 3

2 3 0 3 31

2 2

sin

( ) ( )sin

sin

T TT

T T T TT

v r

v

r ra a

r r r r r

1 2 3 r r r

1 3 2r r r

33

2 1 11 3 2 3 2 2

2 1 1 14 4 2

cot( ) ( )

sin

( ) cot ( ) cos

sin sin

T T T

T T T T

r r ra a a a

r r r r

21 3 sin a a 2

2 3 sin a aFrom and

2 21 3 2 3

1 3 2 3 4 2

1 3 2 3

( sin )( sin )cos( ) ( )

sin

cos

a a a aa a a a

a a a a

1 3 2 3

1 3 2 3

( ) ( ) cos

a a a a

a a a a

34

35

○ Degenerated Point Configurations e.g., points lie on a line or a plane, may cause failure of camera calibration.3.3. Shape Distortions Types of distortions: (a) Tangential distortion (b) Radial distortion Barrel distortion, Pincushion distortion

36

Radial distortion: (a) Changes the distance between the image center and the image point (b) Does not affect the direction joining the image center and the image point

ˆ ( )d dd̂d: actual distance : distorted distance

( )d : distortion function

37

3 2

1

ˆ ( ) 1 ,pp

pd d k d

: coefficients

where

pk

• Polynomial model:

• FOV model:

• Logarithmic model, Fisheye model, Radial model, Rational function model

11ˆ tan (2 tan ),2

d d

ˆtan( ),

2 tan2

dd

2 2ˆ ˆ ˆ ,d x y 2 2d x y where

: distortion coefficient

38

Given an image point (u,v), determine its actual d

。 Consider Polynomial model

39

40

Determine the distortion function

3 2

1

ˆ ( ) 1 ,pp

pd d k d

pki.e., determine its coefficients

41

42

43

44

45

46

47

48

49

50

51

3.5 Application: Mobile Robot Localization -- Calibrate a static camera for monitoring a robot

52

20 images of the planar rectangular grid

Image resolution: 576 by 768Camera: height = 4m, focal length = 4.5mm, Skew = 0, precision = 0.1

pixel 3 radial distortion

coefficients.

Experimental results:Localization error: 2 cm in position

and 1 degree in

orientationMaximum error: 5 cm in position and 5 degrees in

orientation

53

3.4. A Nonlinear Approach

54

55

and2

( )g fdg gdf

df f

56

57

58