CSc 83020 3-D Computer Vision – Ioannis Stamos 3-D Computer Vision CSc 83020 Camera Calibration

CSc 83020 3-D Computer Vision – Ioannis StamosCSc 83020 3-D Computer Vision – Ioannis Stamos

3-D Computer Vision3-D Computer VisionCSc 83020CSc 83020

Camera CalibrationCamera Calibration


Camera CalibrationCamera Calibration Problem: Estimate camera’s extrinsic & Problem: Estimate camera’s extrinsic &

intrinsic parameters.intrinsic parameters. Method: Use image(s) of known scene.Method: Use image(s) of known scene. Tools:Tools:

Geometric camera models.Geometric camera models. SVD and constrained least-squares.SVD and constrained least-squares. Line extraction methods.Line extraction methods.



FeatureExtraction

PerspectiveEquations

From Sebastian Thrun and Jana Kosecka


Perspective Projection, Perspective Projection, Remember?Remember?

fZ Z

Xfx

XO

x



Intrinsic Camera ParametersIntrinsic Camera Parameters

Determine the intrinsic parameters Determine the intrinsic parameters of a camera (with lens)of a camera (with lens)

What are What are Intrinsic ParametersIntrinsic Parameters??

(can you name 7?)(can you name 7?)



Intrinsic ParametersIntrinsic Parameters

fZ

XO

yx oo ,center image

yx ss , size pixel

flength focal

Z

Xfx

21, distortion lens kk


Image center(ox, oy)


Intrinsic Camera ParametersIntrinsic Camera Parameters IntrinsicIntrinsic Parameters: Parameters:

Focal Length Focal Length ff Pixel size Pixel size ssx x ,, ssyy

Image center Image center oox x ,, ooyy

(Nonlinear radial distortion coefficients (Nonlinear radial distortion coefficients kk1 1 ,, kk22…)…) Calibration = Determine the intrinsic Calibration = Determine the intrinsic

parameters of a cameraparameters of a cameraFrom Sebastian Thrun and Jana Kosecka


Coordinate FramesCoordinate Frames

Xw

YwZw

World Coordinate Frame

Xc

Yc

Zc

Camera Coordinate Frame

Image Coordinate Frame

Pixel Coordinates

IntrinsicParameters

ExtrinsicParameters


Why Calibrate?Why Calibrate?

Image

Image point

Calibration: relates points in the image to rays in the scene

Scene ray


Why Calibrate?Why Calibrate?

Calibration: relates points in the image to rays in the scene

Image

Image point x

y

xy

Scene ray

z


Example Calibration PatternExample Calibration Pattern

Calibration Pattern: Object with features of known size/geometryFrom Sebastian Thrun and Jana Kosecka


Harris Corner DetectorHarris Corner Detector


Perspective CameraPerspective Camera

(X,Y,Z)

(x,y,z)

Center ofProjection

r r’

r =(x,y,z)

r’=(X,Y,Z)

r/f=r’/Z

f: effective focal length:distance of image plane from O.

x=f * X/Zy=f * Y/Zz=f

Extrinsic ParametersExtrinsic Parameters

Pc=R(Pw-T)Translation followed by rotation

T

Extrinsic Parameters (2Extrinsic Parameters (2nd nd

formulation)formulation)

Pc=R Pw +TRotation followed by translation

R same as beforeT different


The Rotation MatrixThe Rotation Matrix

R * R = R * R = I =>R = ROrthonormal MatrixDegrees of freedom?

1 0 0I= 0 1 0

0 0 1

T T

T-1


Perspective Camera ModelPerspective Camera Model Step 1: Transform into camera Step 1: Transform into camera

coordinatescoordinates

Step 2: Transform into image Step 2: Transform into image coordinatescoordinates

yc

c

yim

xc

c

xim

oZ

Y

s

fy

oZ

X

s

fx

~

~

~

~

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

~

~

~



Intrinsic ParametersIntrinsic Parameters

YZ

fy

XZ

fx


Image and Camera FramesImage and Camera Frames

Ximage

Yimage

Xcamera

Ycamera

(xim, yim)

(ox,oy)

Zcamera


Geometric ModelGeometric Model

•Transformation fromWorld to Camera Frame.•Perspective projection (f, R, T)

Point in Camera Frame

•Transformation from Imageto Camera Frame.

(ox,oy,sx,sy)•No distortion!

3D Point inCameraCoordinateFrame

x=

y=

XcT

ZcTYcT

ZcT


Camera Calibration: IssuesCamera Calibration: Issues

Which parameters need to be Which parameters need to be estimated.estimated. Focal length, image center, aspect ratioFocal length, image center, aspect ratio Radial distortionsRadial distortions

What kind of accuracy is needed.What kind of accuracy is needed. Application dependentApplication dependent

What kind of calibration object is used.What kind of calibration object is used. One plane, many planesOne plane, many planes Complicated three dimensional objectComplicated three dimensional object



Calibration object Extracted features



Extract centers of circles


Basic EquationsBasic Equations

yc

c

yim

xc

c

xim

oZ

Y

s

fy

oZ

X

s

fx

zwwwc

ywwwc

xwwwc

TZrYrXrZ

TZrYrXrY

TZrYrXrX

333231

232221

131211



yc

c

y

xc

c

x

oZ

Yfy

oZ

Xfx

zwwwc

ywwwc

xwwwc

TZrYrXrZ

TZrYrXrY

TZrYrXrX

333231

232221

131211

In the remaining slidesx means xim and y means yim



yc

cx

xc

c

x

oZ

Yfy

oZ

Xfx

zwwwc

ywwwc

xwwwc

TZrYrXrZ

TZrYrXrY

TZrYrXrX

333231

232221

131211



Extrinsic Parameters 1) Rotation matrix R (3x3) 2) Translation vector T (3x1)

Intrinsic Parameters 1) fx=f/sx, length in effective horizontal pixel size units.2) α=sy/sx, aspect ratio.3) (ox,oy), image center coordinates.4) Radial distortion coefficients.

Total number of parameters (excluding distortion): ?



zwww

ywww

yy

zwww

xwww

xx

TZrYrXr

TZrYrXrfoy

TZrYrXr

TZrYrXrfox

333231

232221

333231

131211

1) Assume that image center is known.2) Solve for the remaining parameters.3) Use N image points and their corresponding N world points

),( ii yx

Twi

wi

wi ZYX ],,[



zwww

ywww

y

zwww

xwww

x

TZrYrXr

TZrYrXrfy

TZrYrXr

TZrYrXrfx

333231

232221

333231

131211


),( ii yx

Twi

wi

wi ZYX ],,[

(1)

Here x means xim - ox and y means yim - oy



)(

)(

131211

232221

xwi

wi

wixi

ywi

wi

wiyi

TZrYrXrfy

TZrYrXrfx


),( ii yx

Twi

wi

wi ZYX ],,[

(2)



xy

iwii

wii

wii

iwii

wii

wii

TvTv

rvrv

rvrv

rvrv

vyvZyvYyvXy

vxvZxvYxvXx

84

137233

126222

115211

8765

4321

,

,

,

,

0 (3)



xy

iwii

wii

wii

iwii

wii

wii

TvTv

rvrv

rvrv

rvrv

vyvZyvYyvXy

vxvZxvYxvXx

84

137233

126222

115211

8765

4321

,

,

,

,

0 (3)

0vA

How would we solve this system?



(3)0vA

NwNN

wNN

wNNN

wNN

wNN

wNN

wwwwww

wwwwww

yZyYyXyxZxYxXx

yZyYyXyxZxYxXx

yZyYyXyxZxYxXx

A

........

........22222222222222

11111111111111


Rank of matrix A?

Solution up to a scale factor.


Singular Value Singular Value DecompositionDecomposition

TUDVA Appendix A.6 (Trucco)

A: m x nU: m x m, columns orthogonal unit vectors.V: n x n , -//-D: m x n , diagonal. The diagonal elements

σi are the singular values σ1>= σ2>= … >= σn >= 0


Singular Value Singular Value DecompositionDecompositionAppendix A.6 (Trucco)

1. Square A non-singular iff σi != 02. For square A C=σ1/σN is the condition number3. For rectangular A # of non-zero σi is the rank4. For square non-singular A:5. For square A, pseudoinverse:6. Singular values of A = square roots of

eigenvalues of and7. Columns of U, V Eigenvectors of 8. Frobenius norm of a matrix

TUVDA 11 TUVDA 1

0

TAA AAT

TAA AAT

TUDVA


Singular Value Singular Value DecompositionDecompositionAppendix A.6 (Trucco)

0vA

UDVA T

If rank(A)=n-1 (7 in our case) then

the solution is the eigenvector which corresponds to the ONLY zero eigenvalue.

Solution up to a scale factor.

Solving for vSolving for v

(3)0vA

NwNN

wNN

wNNN

wNN

wNN

wNN

wwwwww

wwwwww

yZyYyXyxZxYxXx

yZyYyXyxZxYxXx

yZyYyXyxZxYxXx

A

........

........22222222222222

11111111111111

How would we solve this system: SVD. Solution:

Uknown scale factor γ=?

Aspect ratio α=?

),,,,,,,( 131211232221 xy TrrrTrrr v


Solving for Tz and fx?Solving for Tz and fx?



)(

)(

131211

333231

xw

iw

iw

ix

zw

iw

iw

ii

TZrYrXrf

TZrYrXrx

b

x

z

f

TA




)(

)(

131211

333231

xw

iw

iw

ix

zw

iw

iw

ii

TZrYrXrf

TZrYrXrx

b

x

z

f

TA


bTT

x

z AA)Af

T 1^

^

(

Solution in the least squares sense.

Camera Center


Camera Models (linear Camera Models (linear versions)versions)

•Transformation fromWorld to Camera Frame.•Perspective projection (f, R, T)

Point in Camera Frame

•Transformation from Imageto Camera Frame.

(ox,oy,sx,sy)•No distortion!

3D Point inWorld CoordinateFrame

x=

y=

Camera Models (linear Camera Models (linear versions)versions)

World Point(Xw, Yw,Zw)

Measured Pixel(xim, yim)

Elegant decomposition.No distortion!

?

HomogeneousCoordinates


Camera Calibration – Other Camera Calibration – Other methodmethod

Extracted features

1

Z

Y

X

P

w

v

u

w

vy

w

ux

Step 1: Estimate PStep 2: Decompose P into internal and external parameters R,T,C

Camera Calibration: Step 1Camera Calibration: Step 1

Extracted features

1

Z

Y

X

P

w

v

u

w

vy

w

ux

vwy

uwx

2423222134333231

1413121134333231

pZpYpXppZpYpXpy

pZpYpXppZpYpXpx

Each point (x,y) gives us two equations

w

uv


Extracted features

2423222134333231

1413121134333231

pZpYpXppZpYpXpy

pZpYpXppZpYpXpx

0

0

1

0000

0

1

000

34

33

32

31

24

23

22

21

14

13

12

11

p

p

p

p

p

p

p

p

p

p

p

p

y

x

yZ

xZ

yY

xY

yX

xX

ZYX

ZYX

Each corner (x,y) gives us two equations



Extracted features

0

0

0

0

1

0000

0

1

000

1

0000

0

1

000

34

33

32

31

24

23

22

21

14

13

12

11

p

p

p

p

p

p

p

p

p

p

p

p

y

x

yZ

xZ

yY

xY

yX

xX

ZYX

ZYX

y

x

yZ

xZ

yY

xY

yX

xX

ZYX

ZYX

2n

n points gives us 2n equations

A



Extracted features

0

0

0

0

1

0000

0

1

000

1

0000

0

1

000

34

33

32

31

24

23

22

21

14

13

12

11

p

p

p

p

p

p

p

p

p

p

p

p

y

x

yZ

xZ

yY

xY

yX

xX

ZYX

ZYX

y

x

yZ

xZ

yY

xY

yX

xX

ZYX

ZYX

2n

A

pp

Amin0pAWe need to solve

In the presence of noise we need to solve

The solution is given by the eigenvector with the smallest eigenvalue of AAT



Extracted features

i i

ii

i

ii w

vy

w

ux

22

minp

1i

i

i

i

i

i

Z

Y

X

P

w

v

u

The result can be improved throughnon-linear minimization.



Extracted features

The result can be improved throughnon-linear minimization.

i i

ii

i

ii w

vy

w

ux

22

minp

1i

i

i

i

i

i

Z

Y

X

P

w

v

u

Minimize the distance between thepredicted and detected features.

Documents

CSc 83020 3-D Computer Vision – Ioannis Stamos 3-D Computer Vision CSc 83020 Camera Calibration