Camera Projection Matrix - University of Minnesotahspark/CSci5980/Lec2... · Camera 3D world z Origin at world coordinate Camera Projection (Pure Rotation) X C 1 R W Coordinate transformation

Camera Projection Matrix

Camera Model

Ground plane

Camera Model (1st Person Coordinate) Camera

Ground plane

Camera

1 1

x

y

u f p X

v f p Y

Z

Recall camera projection matrix:

+

Camera intrinsic parameter : metric space to pixel space

Camera Model (1st Person Coordinate)

CX

Ground plane

Camera

1 1

x

y

u f p X

v f p Y

Z

K


+

Camera intrinsic parameter : metric space to pixel space


CX

Ground plane

Camera

1 1

x

y

u f p X

v f p Y

Z

K


3D world (metric) 2D image (pix)


CX

Ground plane

Camera

1 1

x

y

u f p X

v f p Y

Z

K



Origin at camera

-1

1

K XC

u X

v Y

Z

CX


Ground plane

Camera

1 1

x

y

u f p X

v f p Y

Z

K



Origin at camera

1

1 1

-11 1 C

11

K X

u X

v Y

Z

Where is origin? 1CX

2CX

2

2 2

-12 2 C

21

K X

u X

v Y

Z


1 1

x

y

u f p X

v f p Y

Z

K



X

Ground plane

Camera

X3D world

Origin at world coordinate

Camera Model (3rd Person Coord. = World Coord.)

Point Rotation

2D rotation

World

x

y

= ( , )X x y

Point Rotation

2D rotation

World

= ( , )X x y

1 1 1= ( , )X x y

1

1

cos sin=

sin cos

x x

y y

Coordinate Transform (Rotation)

2D coordinate transform:

World

x

y

= ( , )X x y


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

?


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

Coordinate transformation: Inverse of point rotation


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

2 2cos sindet = cos sin 1

sin cos


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

rX

rX

: x axis of camera seen from the world rX


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

rX

rX

: x axis of camera seen from the world rX

: y axis of camera seen from the world Yr

YrrY


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

: x axis of world seen from the camera 1r

: y axis of world seen from the camera 2r

r1

r1

2r 2r

Ground plane

Camera

3D world



X

1C

WR

Coordinate transformation from world to camera:

= ( , , )X x y z

World

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

?

Ground plane

Camera

3D world



X

1C

WRx1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r


= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

?

Ground plane

Camera

3D world



X

1C

WRx1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r


= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

Ground plane

Camera

3D world



X

1C

WRx1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r


= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

Degree of freedom?

Ground plane

Camera

3D world



X

1C

WR


= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

Degree of freedom?

Ground plane

Camera

3D world



X

1C

WR


= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T


• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

Ground plane

Camera

3D world



X

1C

WR


Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T


• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

rX : camera x axis seen from the world coord.

Ground plane

Camera

3D world



X

1C

WR


Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T


• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

rX : camera x axis seen from the world coord.

Yr : camera y axis seen from the world coord. Zr

Ground plane

Camera

3D world



X

1C

WR


Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T


• Right hand rule

1r

2r

3 1 2= r r r1r : world x axis seen from

the camera coord.

r1 2r 3r

Ground plane

Camera

3D world



X

1C

WR


Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T


• Right hand rule

1r

2r

3 1 2= r r r1r : world x axis seen from

the camera coord.

r1 2r 3r

2r : world y axis seen from the camera coord.

3r

x1 x2 x3

y1 y2 y3

z1 z2

C

z3

C

C1 1

1

x

y

x

y

u f p

v f p

f p r r r

f p r r r

r

X

Y

Zr r

X

Y

Z

Ground plane

Camera

3D world


Camera Projection (Pure Rotation)

X

1C

WR


Camera

World

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

1r : world x axis seen from the camera coord.

r1 2r 3r


K

Camera projection of world point:

3r

x1 x2 x3

y1 y2 y3

z1 z2

C

z3

C

C1 1

1

x

y

x

y

u f p

v f p

f p r r r

f p r r r

r

X

Y

Zr r

X

Y

Z

Ground plane

Camera

3D world


Camera Projection (Pure Rotation)

X

1C

WR


Camera

World

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

1r : world x axis seen from the camera coord.

r1 2r 3r


K


K

3r

Ground plane

Camera

3D world


Euclidean Transform=Rotation+Translation

X

C CW ,R t

World

= ( , , )X x y z

Ground plane

Camera

3D world


X

Camera

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t



= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x

y

Coordinate transformation: Inverse of point rotation


= ( , )X x y


World

Camera

C

C

cos sin=

sin cos

x

y

x

y

Translation

C C C= ( , )X x y


= ( , )X x y


World

Camera

C C C= ( , )X x yC

C

cos sin=

sin cos

x

y

x t

ty

x

y

: the location of world coordinate seen from camera coord.

Translation

x

y

t

t

x

y

t

t

Ground plane

Camera

3D world


X

Camera

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t


Ground plane

Camera

3D world


X


Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t


C t

C twhere is the world orgin seen from camera.

Ground plane

Camera

3D world


X


Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t


C t


Ground plane

Camera

3D world


Geometric Interpretation

X


Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

C t

Rotate and then, translate.


Ground plane

Camera

3D world



X


Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

Rotate and then, translate.

x1 x2 x3 x

CW y1 y2 y3 y

z1 z2 z3 z

C

1

= = 11

1

XXR CX

r r r -C

r r r -C

r r r -C

cf) Translate and then, rotate.

where is the camera location seen from world. C

C


Ground plane

Camera

3D world


Camera Projection Matrix

X


Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

C

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z

C

C

C1

1

1

1

x

y

x

y

u f p

v f p

f p r r r t

f p r r r t

r r r t

X

Y

Z

X

Y

Z

K


K CWR C t

Image Projection

r

= 0

0

C

-r 0 -1 0

= 0 0 -1

1 0 0

R

Image Projection

r

r

= 0

0

C

-r 0 -1 0

= 0 0 -1

1 0 0

R

Image Projection

r

r

= 0

0

C

-r 0 -1 0

= 0 0 -1

1 0 0

R

cos

= sin

0

C

r

r

-sin cos 0

= 0 0 -1

-cos -sin 0

R

Image Projection K = [200 0 100; 0 200 100; 0 0 1]; radius = 5; theta = 0:0.02:2*pi; for i = 1 : length(theta) camera_offset = [radius*cos(theta(i)); radius*sin(theta(i)); 0]; camera_center = camera_offset + center_of_mass'; rz = [-cos(theta(i)); -sin(theta(i)); 0]; ry = [0 0 -1]'; rx = [-sin(theta(i)); cos(theta(i)); 0]; R = [rx'; ry'; rz']; C = camera_center; P = K * R * [ eye(3) -C]; proj = []; for j = 1 : size(sqaure_point,1) u = P * [sqaure_point(j,:)';1]; proj(j,:) = u'/u(3); end end

RotateCamera.m

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

11 12 13 14

21 22 23 24

31 32 33 3411

Xu p p p p

Yv p p p p

Zp p p p

What does each number mean?

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

What is point at infinity in world x direction?

X

0

0

1

X

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

What is point at infinity in world x direction?

X

0

0

1

X

This point is at infinite but finite in image.

Xu 11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

X

0

0

1

X

Xu

11 14 11

31 34 31

21 24 21

31 34 31

+lim

+

+lim

+

X

X

p X p pu

p X p p

p X p pv

p X p p

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

X

0

0

1

X

Xu

11 14 11

31 34 31

21 24 21

31 34 31

+lim

+

+lim

+

X

X

p X p pu

p X p p

p X p pv

p X p p

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

11

X 21

311

u

u p

v p

p

11

X 21

31

u

p

p

p

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

Xu

12 14 12

32 34 32

22 24 22

32 34 32

+lim

+

+lim

+

X

X

p Y p pu

p Y p p

p Y p pv

p Y p p

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

12

Y 22

321

u

u p

v p

p

12

Y 22

32

u

p

p

p

Yu

Ground plane

Camera

3D world


C CW ,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t


K CWR C t

1=

XP

13 14 13

33 34 33

23 24 23

33 34 33

+lim

+

+lim

+

X

X

p Z p pu

p Z p p

p Z p pv

p Z p p

11 12 13 14

21 22 23 24

31 32 33 34

0

0

11

u p p p p

v p p p p

p p p p

13

Z 23

331

u

u p

v p

p

13

Z 23

33

u

p

p

p

Practice img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

PredictVanishingPoint.m

Practice img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P = KR I C


Practice img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P = KR I C

R = [-0.8496 0.0498 0.5731 -0.3216 -0.8203 -0.4067 0.4180 -0.5299 0.6835]; C = [0.0070 0.7520 -0.2738]; P = K * R * [eye(3) -C] u_x = P(1:2,1)/P(3,1) u_y = P(1:2,2)/P(3,2) u_z = P(1:2,3)/P(3,3)

u_x = -567.8239 138.2813 u_y = 1.0e+03 * 1.8050 2.9748 u_z = 1.0e+03 * 2.9463 0.3517


Practice img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P = KR I C


u_x = -567.8239 138.2813 u_y = 1.0e+03 * 1.8050 2.9748 u_z = 1.0e+03 * 2.9463 0.3517

Practice img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P = KR I C


u_x = -567.8239 138.2813 u_y = 1.0e+03 * 1.8050 2.9748 u_z = 1.0e+03 * 2.9463 0.3517


Ground plane

Camera

3D world

Inverse of Camera Projection Matrix

CX

u-1K u

C

1

XK

u

v

Ground plane

Camera

3D world


C

C C C

W W+ = ( ) K RXR Xt K C

u C

1

XK

u

v

-1

CW K R u C

X

Ground plane

Camera

3D world


Cu C

1

XK

u

v

-1

CW +

1

KX R C

u

v

-1

CW K R u C

3D ray direction 3D ray origin

X C C C


Ground plane

Camera

3D world

Cheirality

Cu C

1

XK

u

v

-1

CW +

1

KX R C

u

v

-1

CW K R u C

3D ray direction 3D ray origin

X

0 where

C C C


Perspective Camera

World Camera Perspective camera model:

4

11 12 13 14

21 22 23 2

31 32 33 34

=1 1

1

X XP

u p p p p

v p p p p

p p p p

Strong perspectiveness

Affine Camera

World Camera Perspective camera model:

11 12 13 14

A 21 22 23 23=1 1

1 0 0 0 1

X XP

u p p p p

v p p p p

4

11 12 13 14

21 22 23 2

31 32 33 34

=1 1

1

X XP

u p p p p

v p p p p

p p p p

Affine camera model:

11 12 13 14

21 22 23 23 1

Xp p p pu

p p p pv

Strong perspectiveness

Affine Camera

World Camera Camera

Strong perspectiveness Weak perspectiveness

Perspective camera model:

11 12 13 14

A 21 22 23 23=1 1

1 0 0 0 1

X XP

u p p p p

v p p p p

4

11 12 13 14

21 22 23 2

31 32 33 34

=1 1

1

X XP

u p p p p

v p p p p

p p p p

Affine camera model:

11 12 13 14

21 22 23 23 1

Xp p p pu

p p p pv

Orthographic Camera Affine camera:

1f

Orthographic camera:

X Y 0 p p

11 12 13 14

21 22 23 23 1

Xp p p pu

p p p pv

11 12 13

21 22 23 1

Xx

y

r r r tu

r r r tv

Camera Anatomy

Lens Radial Distortion

Radial Distortion Model Assumption: Lens distortion is a function of distance from the principal point.

udistorted

p px y,


udistorted

p px y,

Lu udistorted undistorted( )ρ

distorted uρwhere

2 4 L k k1 2( ) 1ρ ρ ρ

uundistorted

Radial Distortion Model


distorted uρwhere

2 4 L k k1 2( ) 1ρ ρ ρ

2 4 L k k1 2( ) 1ρ ρ ρ


01 k01 k


[X, Y] = meshgrid(1:(size(im,2)), 1:(size(im,1))); h = size(X, 1); w = size(X,2); X = X(:); Y = Y(:); pt = [X'; Y']; pt = bsxfun(@minus, pt, [px;py]); pt = bsxfun(@rdivide, pt, [f;f]); r_u = sqrt(sum(pt.^2, 1)); pt = bsxfun(@times, pt, 1 + k * r_u.^2); pt = bsxfun(@times, pt, [f;f]); pt = bsxfun(@plus, pt, [px;py]); imUndistortion(:,:,1) = reshape(interp2(im(:,:,1), pt(1,:), pt(2,:)), [h, w]); imUndistortion(:,:,2) = reshape(interp2(im(:,:,2), pt(1,:), pt(2,:)), [h, w]); imUndistortion(:,:,3) = reshape(interp2(im(:,:,3), pt(1,:), pt(2,:)), [h, w]);

Documents

Camera Projection Matrix - University of Minnesotahspark/CSci5980/Lec2... · Camera 3D world z Origin at world coordinate Camera Projection (Pure Rotation) X C 1 R W Coordinate transformation