Calibration

Dorit Moshe

2

In today’s show

How positions in the image relate to 3D positions in the world? We will use analytical geometry to quantify more precisely the

relationship between a camera, the objects it observes, and the pictures of these objects

We start by briefly recalling elementary notions of analytical Euclidean geometry.

We then introduce the various physical parameters that relate the world and camera coordinate frames, and present as an application various methods for estimating these parameters, a process known as geometric camera calibration.

We also present along the way a linear least-squares technique for parameter estimation

3

Motivation

How positions in the image relate to 3D positions in the world?

The reconstruction of 3D image is not trivial. We have to reconstruct the third coordinate!

32

z

y

x

y

x

4

Example

Rabbit or Man?

The information lays within the 3rd coordinate

Markus Raetz, Metamorphose II, 1991-92

5

2D projections are not the “same” as the real object as we usually see everyday!

6

But…

7

Introduction

Camera calibration: estimation of the unknown values in a camera model.Intrinsic parametersIntrinsic parameters - Link the frame coordinates of

an image point with its corresponding camera coordinates

Extrinsic parametersExtrinsic parameters - define the location and orientation of the camera coordinate system with respect to the world coordinate system

8

Euclidean Geometry - reminder

Orthonormal coordinate Orthonormal coordinate frame (frame (FF)) is defined by

a point O in E3 and three unit vectors i, j and k orthogonal to each other

kOPz

zkyjxiOPjOPy

iOPx

3

z

y

x

p

9

Transformations

FP - the coordinate vector of the point P in the frame F Let’s consider two frames : A and B (A) = (OA, iA, jA, kA)

(B) =(OB, iB, jB, kB)

How can we express BP as a function of AP? Let us suppose that the basis vectors of both coordinate systems are

parallel to each other, i.e., iA = iB, jA = jB and kA = kB, but the origins OA and OB are distinct

We say that two coordinate systems are separated by a pure translation,

POOOPO AABB

10

Pure translation

BP = AP + BOA

11

Pure rotation

When the origins of the two frames coincide, i.e.,OA = OB = O, we say that the frames are separated by a pure rotation.

12

We define the rotation matrixrotation matrix

) ( kjiR AB

AB

ABA

B

BABABA

BABABA

BABABABA R

kkkjki

jkjjji

ikijii

...

...

...

TB

A

TB

A

TB

A

k

j

i

cos|||| baba

13

It means that

for pure rotation

PRP ABA

B

BP BP

14

The inverse of a rotation matrix is equal to its transpose Its determinant is equal to 1 – the transform preserves the

volume. Not every transformation that preserves

the volume keeps the sign. For example - reflection

This ortho normal transform preserves length and angles.

0 1

1 0

15

Example (Pure rotation)

kA=kB=k

The vector iB is obtained by applying to the vector iA a counterclockwise rotation of angle θ about k.

100

0cossin

0sincos

RBA

16

Translation and rotation- rigid transformation

ABAB

AB OPRP

17

As a single matrix equation:

BP APTransformationRotation

18

Homogenous coordinates

Add an extra coordinate and use an equivalence relation

For 3D, equivalence relation k*(X,Y,Z,T) is the same as (X,Y,Z,T)

Motivation – it will be possible to write the action of a perspective camera as a matrix

19

Homogenous/Non-Homogenous transformation for 3D point

Non-homogenous to homogenous – add “1” as the 4th coordinate:

1

z

y

x

z

y

x

Homogenous to non- homogenous – devide 1st 3 coordinates by the 4th

z

y T1

x

T

z

y

x

20

Homogenous/Non-Homogenous transformation for 2D point

Non-homogenous to homogenous – add “1” as the 3rd coordinate:

1

y

x

y

x

Homogenous to non- homogenous – devide 1st 2 coordinates by the 3rd

y 1 x

zz

y

x

21

Camera calibration

Use the camera to tell you things about the world

Relationship between coordinates in the world and coordinates in the image : geometric calibrationgeometric calibration

(We will not discuss here the relationship between intensities in the world and intensities in the image : photometric camera calibration.)

22

Three coordinate systems involved

Camera: perspective projection.Image: intrinsic/internal camera parametersWorld: extrinsic/external camera parameters

23

The camera perspective equation

The coordinates (x, y, z) of a scene point P observed by a pinhole camera are related to its image coordinates (x’, y’) by the perspective equation

We have by similar triangles (x,y,z)-> (f x/z, f y/z, -f ). Ignoring the third coordinate, we get (x,y,z)-> (f x/z, f y/z)

P’P

24

Intrinsic parameters

Relate the camera’s coordinate system to the idealized coordinate system

We can associate with a camera two different image planes: the first one is a normalized planenormalized plane located at a unit distance from the pinhole. We attach to this plane its own coordinate system with an origin located at the point where the optical axis pierces it. According to the perspective eq :

Perspective projectionPerspective projection::C

zy

zx

v

u

ˆ

ˆ

25

f

Intrinsic parameters(cont)

1

26

The second is the physical retinaphysical retina. It is located at a distance f ≠1 from the pinhole, and the image coordinates (u,v) are usually expressed in pixel units.

Pixels are usually rectangular, so the camera has two additional scale parameters k and l, and:

zy

zx

lfv

kfu f is a distance in meters

lk11 dimentions sPixel’

Define : units pixelin expressed , lfkf

Intrinsic parameters(cont)

Meterk,l 1 *pixel units

27

The actual origin of the camera coordinate system is at a corner C of the retina, and not at its center. It adds two parameters u0 and v0 that define the position (in pixel units) of C0 in the retinal coordinate system.

0

0

vv

uu

zy

zx

Intrinsic parameters(cont)

28

The camera coordinate system may also be skewed, due to some manufacturing error, so the angle θ between the two image axes is not equal to 90°.

0

0

sin

cot

vz

y

θ

βv

uz

yθα

z

xαu

Intrinsic parameters(cont)

29

Using homogenous coordinates:

Intrinsic parameters(cont)

3x4 matrix

30

The physical size of the pixels and the skew are always fixed for a given camera, and they can in principle be measured during manufacturing

Intrinsic parameters(cont)

31

Extrinsic parameters

Relate the camera’s coordinate system to a fixed world coordinate system and specify its position and orientation in space.

We consider the case where the camera frame (C) is distinct from the world frame (W).

Non-homogenous coordinates

Homogenous coordinates

32Twww

WC

CW

zyxP

dinatespixel coorp

Ot

RR

t)R KM

whereΜPz

1p

)1,,,(

(

Extrinsic parameters(cont)

33

M can be defined with 11 free coefficients5 are intrinsic parameters – α,β,u0,v0,θ

6 are extrinsic – the 3 angles defining R, 3 coordinates of t

Combining extrinsic and intrinsic calibration parameters

3 raws of R

3 coords of t

M is only defined up to scale in this setting!!

34

Rewriting the equation

PΜz

1p

1. . .m . . .

. . .m . . .

. . .m . . .1

13

2

1

z

y

x

T

T

T

w

w

w

zv

u

Pixel coordinates

World coordinates

35

Z is in the camera coordinate system, but we can solve it, cause

z

Pm 31

And we get

Relation between image positions, u,v to points at 3D positions in P (homogenous coordinates)

36

Calibration methods

Techniques for estimating the intrinsic and extrinsic parameters of a camera

Suppose that a camera observes nn geometric features such as points or lines with known positions in some fixed world coordinate system.

We will:Compute the perspective projection

matrix M associated with the camera in this coordinate system

Compute the intrinsic and extrinsic parameters of the camera from this matrix

Which features should we choose? Which features should we choose?

37

A linear approach to camera calibration

For each feature point, i, we have :

0)(

0)(

32

31

ii

ii

Pmvm

Pmum

For n features, we will get 2n equations

38P =0m

A linear approach to camera calibration(cont)

39

A linear approach to camera calibration(cont)

When n>=6, the system is over-constrained, i.e. there is no non-zero vector m in R12 that satisfies exactly these equations.

On the other hand, a zero vector is always a solution. According to the linear least-squares methods, we want to

compute the value of the unit vector m that minimizes |Pm|2.

In particular, estimating the vector m reduces to computing the eigenvectors and eigenvalues of the 12x12 matrix PTP

40

Linear least squares methods

Let us consider a system of n equations, p unknowns:

A is n x p matrix with coefficients aij, x = (x1,..,xp)T

There is no single solution if n≥p. The non trivial solution exists only if A is non singular.

We will try to find vector x that minimize E (the error measure):

41

Linear least squares methods(cont)

Need to impose a contraint on x, since x=0 yields the minimum. Since E(λx) = λ2E(x), we will use the contraint |x2|=1.

E=xT(ATA)x , where ATA (pxp) matrix is positive smmetric matrix It can be diagonalized in an orthonormal basis of eigenvectors e i (i=1,..,p) associated

with eigenvalues 0≤ λ1 ≤ λ1 ≤ … ≤ λp

we can write x as x=μ1e1+… μnen, that (μ12 +…+ μp

2 )=1

: e1 minimizes the error E. It is the eigenvector

associated with the minimum eigenvalue of ATA (λ1 )

42

Recovering the intrinsic and extrinsic parameters

Once we have the M matrix, we can recover the intrinsic and extrinsic parameters in a simple mathematical process, described in Forsyth&Ponce, section 6.3.1

43

Camera Calibration with a Single Image

Sometimes more than one view of the same picture is used to estimate calibration parameters (For example, Stereo)

Most Camera parameters can be estimated from the measurements of a single image when sufficient geometric object knowledge is available.

The object knowledge used in the approach described here consists of parallelism and perpendicularity assumptions of straight object edges. In buildings parallel and perpendicular edges

are usually abundant. Therefore, this method is often applicable for historic imagery of possibly demolished buildings taken with an unknown camera.

44

Targets for camera calibration

45

Projection of each point gives us two equations and there are 11 unknowns. 6 points in general position are sufficient for calibration.

46

The 6 anchor points clicked by the user are represented in green. If the user had clicked more accurately, they should lie exactly at the corners of the small white squares. Using these 6 points and the corresponding 3D anchor points, the program computes an initial estimate of the projection matrix.

http://www-sop.inria.fr/robotvis/personnel/lucr/detecproj.html

47

We take as input a set of at least 6 non-coplanar 3D anchor points, and their 2D images. The 2D coordinates do not need to be very accurate, they are typically obtained manually by a user who clicks their approximate position.

48

Summary

We saw the goal of calibration We mentioned Euclidean Geometry We learned about internal/external camera parameters We learned how to compute them from a given set of

points We saw an example of calibration by one picture only We will see (stereo lecture) computing 3D coordinates

from more than one picture (More than one view).