kinect duas camaras

Lab Course Kinect Programming for Computer Vision

Transformations and Camera Calibration!

Loren Schwarz ([email protected])!May 25, 2011!

Lecture Outline"

!!

Part I: Basics of Transformations!!

Part II: Basics of Cameras!

Kinect Lab Course - Transformations and Cameras! 2!

Outline Part I"

•  2D Transformations!

•  Homogeneous Coordinates in 2D!

•  Hierarchy of Transformations in 2D!



2D Transformations"

The basic building blocks of transformations are:!

•  Scaling (isotropic, non-isotropic)!•  Rotation (around one rotation axis)!•  Translation (motion along the two axes)!

Transformations are defined by their action on point coordinates.!!Notation:!

4!Kinect Lab Course - Transformations and Cameras!

p = (x, y)� =�

xy

�p� = (x�, y�)� =

�x�

y�

�

original point! transformed point!

2D Transformations – Scaling"


Isotropic scaling: same scaling factor in both spatial dimensions!

�x�

y�

�=

�s 00 s

� �xy

�

2D Transformations – Scaling"


Non-isotropic scaling: different scaling factors in the two spatial dimensions!

�x�

y�

�=

�sx 00 sy

� �xy

�

2D Transformations – Rotation"


�x�

y�

�=

�cos θ −sin θsin θ cos θ

� �xy

�

θ"

2D Transformations – Rotation"


p�

p

x!

y!

p�

p

r!

p =�

xy

�=

�r cos(α)r sin(α)

�

2D Transformations – Translation"


�x�

y�

�=

�xy

�+

�txty

�

2D Transformations"

Have you noticed we have two types of representations for the basic transformations seen so far?!!Scaling, Rotation: !Matrix-vector product"!!!Translation: !Sum of vectors"""""


�x�

y�

�=

�xy

�+

�txty

�

�x�

y�

�=

�cos θ −sin θsin θ cos θ

� �xy

��x�

y�

�=

�sx 00 sy

� �xy

�

How to achieve a matrix-vector product form for all types of transformations?!

Outline Part I"






Homogeneous Coordinates in 2D"

•  Write a 2D point in homogeneous coordinates:!

!

•  Then, translation can easily be expressed as a matrix-vector product:!


p =

xy1

∈ R3

We will see more properties of homogeneous representation later.!

x�

y�

1

=

1 0 tx0 1 ty0 0 1

xy1

=

x + txy + ty

1



•  Rotation in homogeneous representation:!

!•  Combined transformations (here: rotation and translation):!


x�

y�

1

=

cos θ −sin θ 0sin θ cos θ 0

0 0 1

xy1

x�

y�

1

=

cos θ −sin θ txsin θ cos θ ty

0 0 1

xy1

=

xcos θ − ysin θ + txxsin θ + ycos θ + ty

1

2D Projective Space "

!

•  Points in 2D Euclidean space are represented by Cartesian coordinates!

•  Points in 2D Projective space are represented by Homogeneous coordinates:!


p =�

xy

�∈ R2

p =

xy1

∈ P2

When dealing with points represented in homogeneous coordinates, we are working in a Projective space.!

2D Projective Space "


Point in Euclidean 2D space! Point in Projective 2D space.All points on the orange line represent the same point. A homogeneous vector in 2D projective space represents an equivalence class of points.!

z!

y!

p = (x,y,1)!x!

k(x,y,1)!

x!

y!

p = (x,y)!

z = 1!

Projective Plane!

Points and Lines in 2D Projective Space"

Properties of points"

•  A 2D point has the homogeneous representation . !

•  The homogeneous vector with w≠0 represents the point .!

•  For any non-zero k, and represent the same point.!


(x, y)� (x, y, 1)�

(x, y, w)� (x

w,

y

w, 1)�

k(x, y, w)�(x, y, w)�


Properties of lines"

•  A line has the homogeneous representation .!

•  A point lies on a line if and only if .!

•  Two lines and intersect in the point ! .!

•  The line joining the two points and is .!


ax + by + c = 0 l = (a, b, c)�

x l x�l = 0

l l� x = l× l�

l = x× x�x x�


Special points and lines"

•  Points with 3rd coordinate zero are called points at infinity.!

•  All points at infinity lie on the line at infinity .!

•  Parallel lines intersect at infinity.!


(x, y, 0)�

l∞ = (0, 0, 1)�

In projective space, all lines intersect in a point, even parallel lines.!

l�∞(x, y, 0) = (0, 0, 1)�(x, y, 0) = 0

Outline Part I"






Hierarchy of Transformations in 2D"

•  Classification of transformations in terms of quantities or properties that are invariant (left unchanged) by the transformation!

•  Hierarchy: Starting from most specialized type of transformation, successively remove invariants to get more general types of transformations !!

!!

•  Containment relation from general to specialized, e.g.: affine transformations have all properties of projective transformations, plus additional ones.!


Most general:!! !! !

Most specialized:!

Projective transformations!!Affine transformations!!Similarity transformations!!Euclidean transformations!

Hierarchy of Transformations in 2D: Euclidean Transformations"

•  Transformation:!

•  Concise notation for transformation matrix:!

•  Degrees of freedom: !–  3 (1 rotation, 2 translation)

!•  Invariants: length, area, angles!


Teuclidean =�

R t0� 1

�

x�

y�

1

=

cos θ −sin θ txsin θ cos θ ty

0 0 1

xy1

R�R = Idet(R) = 1

Hierarchy of Transformations in 2D: Similarity Transformations"



•  Degrees of freedom: !–  4 (1 rotation, 2 translation, 1 isotropic scaling)

!•  Invariants: ratio of lengths, angles, parallel lines!


x�

y�

1

=

scos θ −ssin θ txssin θ scos θ ty

0 0 1

xy1

Tsimilarity =�

sR t0� 1

�R�R = I

det(R) = 1

Hierarchy of Transformations in 2D: Affine Transformations"



•  Degrees of freedom: !–  6 (2 translation, 4 entries of A)

!•  Invariants: parallel lines, ratios of areas, line at infinity !


x�

y�

1

=

a11 a12 txa21 a22 ty0 0 1

xy1

Taffinity =�

A t0� 1

�A non-singular!

Hierarchy of Transformations in 2D: Affine Transformations"

Affine transformations geometrically consist of two components:!


A = R(θ)R(−φ)�

λ1 00 λ2

�R(φ)

Rotation! Non-isotropic Scaling!

Hierarchy of Transformations in 2D:Projective Transformations"



•  Degrees of freedom: !–  8 (9 arbitrary matrix entries, up to scale)

!•  Invariants: Intersection points, tangency, cross-ratios!


x�

y�

1

=

a11 a12 txa21 a22 tyv1 v2 v

xy1

Tprojectivity =�

A tv� v

�A non-singular!

Hierarchy of Transformations in 2D – Summary"


Projective "8 DOF!!!!Affine " "6 DOF!!!!Similarity "4 DOF!!!!Euclidean "3 DOF!

Tprojectivity =�

A tv� v

�

Taffinity =�

A t0� 1

�

Tsimilarity =�

sR t0� 1

�

Teuclidean =�

R t0� 1

�

•  Affine Transformations:!

Action of Transformations in 2D on Points at Infinity"


a11 a12 txa21 a22 ty0 0 1

xy0

=

a11x + a12ya21x + a22y

0

Point at infinity,e.g. intersection !

of blue lines!

Mapped by affine transformation to another

point at infinity!

•  Projective Transformations:!

Mapped by projective transformation to a finite point (vanishing point)!

Action of Transformations in 2D on Points at Infinity"


a11 a12 txa21 a22 tyv1 v2 v

xy0

=

a11x + a12ya21x + a22yv1x + v2y

Point at infinity,e.g. intersection !

of blue lines!

Only projective transformations can make points at infinity finite points.!

Outline Part I"








•  Write a 3D point in homogeneous coordinates:!

!

•  3D projective space: properties analogous to 2D homogeneous representation!•  All types of transformations (scaling, translation, rotation, etc.) can be written

as a matrix-vector product!•  Several transformations can be composed into one transformation matrix!


p =

xyz1

∈ R4

Hierarchy of Transformations in 3D (1): Euclidean Transformations"



•  Degrees of freedom: !–  6 (3 rotation, 3 translation)

!•  Invariants: volume, angles!


Teuclidean =�

R t0� 1

�R�R = I

det(R) = 1

x�

y�

z�

1

=

r11 r12 r13 txr21 r22 r23 tyr31 r32 r33 tz0 0 0 1

xyz1

Hierarchy of Transformations in 3D (2): Similarity Transformations"



•  Degrees of freedom: !–  7 (3 rotation, 3 translation, 1 scaling)

!•  Invariants: angles!


Tsimilarity =�

sR t0� 1

�R�R = I

det(R) = 1

x�

y�

z�

1

=

sr11 sr12 sr13 txsr21 sr22 sr23 tysr31 sr32 sr33 tz0 0 0 1

xyz1

Hierarchy of Transformations in 3D (3): Affine Transformations"



•  Degrees of freedom: !–  12 (3 translation, 9 entries of A)

!•  Invariants: parallel planes, ratios of volumes, plane at infinity !


Taffinity =�

A t0� 1

�A non-singular!

x�

y�

z�

1

=

a11 a12 a13 txa21 a22 a23 tya31 a32 a33 tz0 0 0 1

xyz1

Hierarchy of Transformations in 3D (4)Projective Transformations"



•  Degrees of freedom: !–  15 (16 arbitrary matrix entries, up to scale)

!•  Invariants: Intersection points, tangency!


Tprojectivity =�

A tv� v

�A non-singular!

x�

y�

z�

1

=

a11 a12 a13 txa21 a22 a23 tya31 a32 a33 tzv1 v2 v3 v

xyz1

Hierarchy of Transformations in 3D – Summary"


Projective "15 DOF!!!!Affine " "12 DOF!!!!Similarity "7 DOF!!!!Euclidean "6 DOF!

Tprojectivity =�

A tv� v

�

Taffinity =�

A t0� 1

�

Tsimilarity =�

sR t0� 1

�

Teuclidean =�

R t0� 1

�

Lecture Outline"

!!

Part I: Basics of Transformations!!

Part II: Basics of Cameras!


Pinhole Camera Model"Mapping between 3D world and 2D image by central projection!


Pinhole Camera Model"In homogeneous coordinates…!


fXfYZ

!

"

####

$

%

&&&&=

f 0f 01 0

'

(

)))

*

+

,,,

XYZ1

!

"

####

$

%

&&&&

Pinhole Camera Model"Mapping between 3D world and 2D image by central projection!


fXfYZ

!

"

####

$

%

&&&&=

ff1

'

(

)))

*

+

,,,

1 01 01 0

'

(

)))

*

+

,,,

XYZ1

!

"

####

$

%

&&&&

Pinhole Camera Model"Origin of image plane is not necessarily at the camera center!


principal point (perpendicular intersection point of

principal axis and image plane)

Pinhole Camera Model"Origin of image plane is not necessarily at the camera center!


fX + ZpxfY + Zpx

Z

!

"

####

$

%

&&&&=

f px 0f py 0

1 0

'

(

))))

*

+

,,,,

XYZ1

!

"

####

$

%

&&&&

K =

f pxf py

1

!

"

####

$

%

&&&&

intrinsic camera parameters matrix

Pinhole Camera Model"Camera center is not necessarily at the world coordinate system origin!

Xcam =R !R !C0 1

"

#$$

%

&''X

3D point in camera coordinates

x =K I | 0[ ]Xcam

x =KR I |! !C"# $%X

camera center in world coordinates

projection to image plane from camera coordinates

projection to image plane from world coordinates

!Xcam = R !X- !C( )Points with a tilde are in non-homogeneous notation!

Pinhole Camera Model"Camera center is not necessarily at the world coordinate system origin!

x =KR I |! !C"# $%X

K =

f pxf py

1

!

"

####

$

%

&&&&

x = PX

P =KR I |! !C"# $%

projection matrix, 9 DOF

intrinsic parameters matrix, 3 DOF

R, !C

extrinsic parameters, each 3 DOF

K =

mx

my

1

!

"

####

$

%

&&&&

f pxf py

1

!

"

####

$

%

&&&&

K =

!x x0!y y0

1

!

"

####

$

%

&&&&

CCD Camera Model"

mx,my number of pixels per unit distance in camera coordinates

4 DOF

P =KR I |! !C"# $% 10 DOF


Camera coordinates are mapped to pixels of the CCD chip!

K =

!x s x0!y y0

1

!

"

####

$

%

&&&&

P =KR I |! !C"# $%

Finite Projective Camera Model"

s skew parameter

5 DOF

11 DOF


Both axes in camera coordinates need not have identical scaling!

K =

!x s x0!y y0

1

!

"

####

$

%

&&&&

P =KR I |! !C"# $%

Camera Calibration"


•  Relating the ideal camera model to the properties of an actual physical device and determining the position and orientation of the camera w.r.t. the world coordinate frame1.!

•  Estimating the entries of!

•  Camera models with increasing generality:!–  pinhole camera ! ! 9 DOF!–  CCD camera ! !10 DOF!–  finite projective camera !11 DOF!

•  In addition: lens distortion parameters!

1 http://www.dis.uniroma1.it/~iocchi/stereo/calib.html!

K,R, !C

Camera Calibration"


Checkerboard pattern captured from multiple views (the checker sizes

in the real world are known)

Semi-automatic measurements to find sizes in each of the images

•  Linear and non-linear optimization for estimating camera parameters •  MATLAB calibration toolbox: http://www.vision.caltech.edu/bouguetj/calib_doc!

Images from http://www.vision.caltech.edu/bouguetj/calib_doc!

Camera Calibration"

•  sdfsdf!

Kinect Lab Course - Transformations and Cameras! 48!Images from http://www.vision.caltech.edu/bouguetj/calib_doc!

Stereo Calibration"

•  Find the spatial relation between two cameras observing the same scene (stereo setup)!

•  For each orientation of the calibration pattern, capture corresponding left and right camera images!

•  Calibrate each camera separately, as before!

•  Perform stereo calibration using the result of separate calibration as an input!


Calibrating the Kinect"

•  Depth and RGB images do not match exactly, there is an offset!•  Stereo calibration for getting both images from the same camera perspective!


RGB Image! Depth Image!

Calibrating the Kinect"

•  Kinect has two cameras with standard optics!–  RGB camera!–  Infrared (IR) camera!

•  Infrared image is used internally for computing depth image, but is accessible through the API!

•  Checkerboard pattern is not visible in depth images, but visible in IR images!


Infrared!Projector!

RGB!Camera!

Infrared!Camera!

RGB Image!

Infrared Image!

Relating RGB and Depth Images"

Stereo calibration gives and .!

Step 1: Back-project every 2D point from depth image into 3D space!Step 2: Apply rigid transformation between two cameras to 3D points!Step 3: Project every transformed 3D point into the RGB image!


RGB"

IR"

R, t 3D Points!

Back-projection using!

Projection using!

R, t

Relative transformation!between rigidly

connected cameras!


Step 1: Back-project every 2D point from depth image into 3D space!

•  ToF data: depth d in meters for every pixel location !•  Desired data: 3D coordinates for every pixel !!


Depth Image! 3D Point Cloud!

X = (X,Y, Z)�x = (x, y)�


Step 1: Back-project every 2D point from depth image into 3D space!!Apply inverse of IR camera projection to obtain 3D points:!


xyw

=

fx 0 cx 00 fy cy 00 0 1 0

XYZW

=

fxX + cxZfyY + cyZ

Z

X =(x− cx)Z

fxY =

(y − cy)Zfy

Inverse relation for X and Y!



!For every depth pixel at location with depth d, !compute a 3D point with coordinates: !

x = (x, y)�

X = (X,Y, Z)�

Intrinsics of the IR (depth) camera!

Step 1: Back-project every 2D point from depth image into 3D space!


Step 2: Apply rigid transformation between two cameras to 3D points!Step 3: Project every transformed 3D point into the RGB image!!!!•  are the coordinates of the

projection of X into the RGB image!•  look up color corrsponding to

a given 3D point!


Assignment 3"

•  Extend your application such that it can read out Kinect IR images!•  Extend your application for saving IR and RGB images to bitmap files!•  Familiarize yourself with the Matlab Camera Calibration Toolbox

http://www.vision.caltech.edu/bouguetj/calib_doc/!–  Read the examples (especially examples 1 and 5)!–  Download the toolbox!–  Try out the examples!

•  Perform stereo-calibration for your Kinect device!–  Note that you cannot extract IR and RGB images simultaneously…!

"Outlook Assignment 4:"•  Implement a 3D point cloud visualization for kinect depth data!•  Add a feature to color the 3D points with the colors from the RGB camera!


Documents

kinect duas camaras