ROBOT VISION Lesson 4: Camera Models and Calibration Matthias Rüther Slides partial courtesy of...

Robot Vision SS 2010 Matthias Rüther 1

ROBOT VISION Lesson 4: Camera Models and Calibration

Matthias Rüther

Slides partial courtesy of Marc Pollefeys Department of Computer Science

University of North Carolina, Chapel Hill

Robot Vision SS 2010 Matthias Rüther 2

Content

Camera Models– Pinhole Camera– CCD Camera– Finite Projective Camera– Affine Camera– Pushbroom Camera

Calibration– Inner Orientation– Nonlinear Distortion– Calibration using Planar Targets– Calibration using a 3D Target

The Cyclops, 1914 by Odilon Redon

Robot Vision SS 2010 Matthias Rüther 3

Basic Pinhole Camera Model

Robot Vision SS 2010 Matthias Rüther 4

TT ZfYZfXZYX )/,/(),,(

10100

1ZYX

ff

ZfYfX

ZYX

Basic Pinhole Camera Model

Robot Vision SS 2010 Matthias Rüther 5

10100

ZYX

ff

ZfYfX

1010101

1ZYX

ff

ZfYfX

PXx

0|I)1,,(diagP ff

Basic Pinhole Camera Model

Robot Vision SS 2010 Matthias Rüther 6

Tyx

T pZfYpZfXZYX )/,/(),,( principal point

Tyx pp ),(

00

1 01 1

x x

y y

X XfX Zp f p

Y YfY Zp f p

Z ZZ

Principal Point Offset

Robot Vision SS 2010 Matthias Rüther 7

10100

ZYX

pfpf

ZZpfYZpfX

y

x

x

x

camX0|IKx

1y

x

pfpf

K calibration matrix

Principal Point Offset

Robot Vision SS 2010 Matthias Rüther 8

C~-X~RX~ cam

X10RCR

110

C~RRXcam

ZYX

camX0|IKx XC~|IKRx

t|RKP C~Rt PXx

Camera Rotation and Translation

Robot Vision SS 2010 Matthias Rüther 9

1yy

xx

pp

K

11y

x

y

x

pfpf

mm

K

CCD Camera

Robot Vision SS 2010 Matthias Rüther 10

1yy

xx

pps

K

1yy

xx

pp

K

C~|IKRP

non-singular

11 dof (5+3+3)

decompose P in K,R,C? 4p|MP 4

1pMC~ MRK, RQ

{finite cameras}={P3x4 | det M≠0}

If rank P=3, but rank M<3, then cam at infinity

Finite Projective Camera

Robot Vision SS 2010 Matthias Rüther 11

PXx

MdDp|MPDx 4

Forward projection (3D -> 2D)

Back-projection (2D -> 3D)

xPX 1PPPP

TT IPP

(pseudo-inverse)

0PC

λCxPλX

1p-μxM

1pM-

0xM

μλX 4-1

4-1-1

xMd -1

CD

Action of Projective Cameras on Points

D…direction

Robot Vision SS 2010 Matthias Rüther 13

1yx

xx

pps

K

1

arctan(1/s)

for CCD/CMOS, always s=0

Image from image, s≠0 possible(non coinciding principal axis)

HPresulting camera:

When is skew non-zero?

Robot Vision SS 2010 Matthias Rüther 14

00d

P

Camera center at infinity

0Mdet

Affine and non-affine cameras

Definition: affine camera has P3T=(0,0,0,1)

Moving the Camera Center to Infinity

Robot Vision SS 2010 Matthias Rüther 15

Affine Cameras

Robot Vision SS 2010 Matthias Rüther 16

100000100001

P canonical representation

100K

K 22affine calibration matrix

principal point is not defined

Parallel Projection: Summary

Robot Vision SS 2010 Matthias Rüther 17

Orthographic projection

Scaled orthographic projection

100000100001

P

10tR

H

1T1

2T2

rP r

0 1

tt

1T1

2T2

rP r

0 1/

tt

k

(5dof)

(6dof)

A Hierarchy of Affine Cameras

Robot Vision SS 2010 Matthias Rüther 18

Weak perspective projection

ktt

y

x

/10rr

1α

αP 2

1T1

1T

(7dof)

A Hierarchy of Affine Cameras

Robot Vision SS 2010 Matthias Rüther 19

1. Affine camera=camera with principal plane coinciding with ∞

2. Affine camera maps parallel lines to parallel lines

3. No center of projection, but direction of projection PAD=0(point on ∞)

Affine camera

ktts

y

x

A/10

rr

1α

αP 2

1T1

1T(8dof)

1000P 2232221

1131211tmmmtmmm

A

affine 44100000100001

affine 33P

A

A Hierarchy of Affine Cameras

Robot Vision SS 2010 Matthias Rüther 20

T)(X X,Y,X,T T),,(PX wyx T)/,( wyx

Straight lines are not mapped to straight lines!(otherwise it would be a projective camera)

(11dof)

Pushbroom Cameras

Robot Vision SS 2010 Matthias Rüther 21

ZYX

pppppp

yx

232221

131211 (5dof)

Null-space PC=0 yields camera center

Also decomposition c~|IRKP 22222232

Line Cameras

Robot Vision SS 2010 Matthias Rüther 22

Camera calibration

Robot Vision SS 2010 Matthias Rüther 23

ii xX ? P

Problem Statement

Robot Vision SS 2010 Matthias Rüther 24

ii PXx

ii PXx

0Ap

Basic Equations

Robot Vision SS 2010 Matthias Rüther 25

0Ap

minimal solution

Over-determined solution

5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points

n 6 points

Apminimize subject to constraint

1p

1p̂3 3p̂

P

Basic Equations

Robot Vision SS 2010 Matthias Rüther 26

Geometric Error

Robot Vision SS 2010 Matthias Rüther 27

Gold Standard algorithm

ObjectiveGiven n≥6 2D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelihood Estimation of P

Algorithm(i) Linear solution:

(a) Normalization: (b) DLT:

(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:

(iii) Denormalization:

ii UXX~ ii Txx~

UP~TP -1

~ ~~

Robot Vision SS 2010 Matthias Rüther 28

Calibrated camera, position and orientation unkown

Pose estimation

6 dof 3 points minimal (4 solutions in general)

Exterior Orientation

Robot Vision SS 2010 Matthias Rüther 29

short and long focal length

Nonlinear Distortion

Radial Component

Robot Vision SS 2010 Matthias Rüther 30

Nonlinear Distortion Radial Component

Robot Vision SS 2010 Matthias Rüther 31

Correction of radial distortion

Computing the parameters of the distortion function(i) Minimize with additional unknowns(ii) Straighten lines(iii) …

Correction of radial Distortion

Robot Vision SS 2010 Matthias Rüther 32

Nonlinear Distortion Tangential Component

Distortion function:

Robot Vision SS 2010 Matthias Rüther 33

Robot Vision SS 2010 Matthias Rüther 34

Distortion Workflow (KU wp1)

Project world points: x = P*X

Normalize projected points: xn = K-1 * x

Apply distortion to xn : xd = fdistort(xn)

Denormalize: xp = K * xd