3D-2D registration

Kazunori UmedaChuo Univ., Japan

umeda@mech.chuo-u.ac.jp

http://www.mech.chuo-u.ac.jp/umedalab/

CRV2010 TutorialMay 30, 2010

Registrationof range image and color image

Necessary for texture mapping

Range image(3D model)

Color image

When 3D-2D registration is given,

Texture mapping

Parameters to obtain for 3D-2D registration

Color camera

Image plane

Object(range image)

Range imagesensor

Sensorcoordinate system

Projection of range intensity image

Intensity image

Intrinsic parameters

Extrinsic parameters (Distortion parameters)

X

Y

Z

Extrinsic parametersObject’ rotation R and translation t or camera’s orientation Rc and position tc

Color camera

Range imagesensor

Sensorcoordinate system

X

Y

Z

R, t (Rc, tc)

z

y

x

t

t

t

rrr

rrr

rrr

R t,

333231

232221

131211

Parameters to obtain for 3D-2D registration

tt Tc

Tc RRR ,

Intrinsic parameters

(X,Y,Z) (u,v)3D space Image plane

cameracoordinate system

0

0

vZ

Yv

uZ

sYXu

v

u

00 ,,,, vusvu

u, v: focal length/pixel sizes: skew, u0, v0: principal point coordinates

Color camera

Image plane

u

v

),,( ZYX

),( vu

uxX

Parameters to obtain for 3D-2D registration

Homogenous coordinates

100

0

111

0

0

v

us

A

Z

Y

X

PZ

Y

X

RAv

u

s

v

u

w

w

w

w

w

w

t

Parameters to obtain for 3D-2D registration

P: 34 matrix

11 unknown parameters (6 extrinsic + 5 intrinsic)2 constraints

When correspondences between range image and color image are given,

…It is hard to obtain correspondences even manually.

Parameters can be calculated.Equivalent to camera calibration problem.For extrinsic parameter estimation,

Equivalent to PnP (Perspective n-Point) problem

6n

3n

3D 2D

Range image

Range intensity image(reflectance image)

By using range intensity image, obtaining correspondences becomes easier!

e.g., corners, edges,SIFT [Böhm 2007]

Our approach: gradient-based method(not explicitly using correspondences)

Two 2Dimages are matched

Gradient-based method

Update camera parameters

Produce a 2D imagefrom a range image

Initial camera parameters

End

Yes

No

uv

Projection of range intensity image

Intensity image

),( vu

),( vu

tvu IvIuI

t

II

v

II

u

II tvu

,,

Optical flow constraint

It: difference between intensity image and projected range intensity image

),,(),,( tvuIttvvuuI Tailor expansion

0

0

vZ

Yv

uZ

sYXu

v

u

(1) Constraints for extrinsic parameters

When intrinsic parameters are constant,

ZZ

YY

Zv

ZZ

sYXY

Z

sX

Zu

vv

uu

2

2

Substituting for the optical flow constraint

tvu IvIuI

tv

vu

uv

vuu

u IZZ

YI

Z

sYXIY

ZI

Z

sIX

ZI

22

XωvX 0 Camera motion: v0,

TZYXX

TzyxT

zyx vvv ωv0 ,000Digital camera

uv

),,( ZYX

),( vu

22

000

,,

,)()()(

Z

YI

Z

sYXIc

ZI

Z

sIb

ZIa

IbXaYaZcXcYbZcvbvav

vv

uu

vvu

uu

tzyxzyx

Linear equation for 6 motion parameters v0,

v0, can be solved with 6 or more points by linear least square method.

Motion parameters are supposed to be smallIteration is necessary

v0, R(33 rotation matrix) and t (3D translation vector)

[Yamamoto 1985][Horn IJCV1988]

cf.

(2) Constraints for intrinsic parameters

0

0

vZ

Yv

uZ

sYXu

v

u

When intrinsic parameters are also variables,

02

02

vZ

YZ

Z

YY

Zv

usZ

Y

Z

XZ

Z

sYXY

Z

sX

Zu

vvv

uuu

Substituting for the optical flow constraint

tvu IvIuI

tvuuvvuu

zyxzyx

IvIuIsZ

YI

Z

YI

Z

XI

bXaYaZcXcYbZcvbvav

00

000 )()()(

a,b,c: same as previous equation

Linear equation for 6 motion parameters v0, and5 intrinsic parameters

v0, and intrinsic parameters can be solved with 11 or more points by linear least square method.

(3) Constraints for distortion

0

0

vZ

Yv

uZ

sYXu

v

u

02

22

1

02

22

1

1

1

vZ

YXk

Z

Yv

uZ

YXk

Z

sYXu

v

u

Distortion model (the simplest)

)'1(

)'1(2

1

21

rkyy

rkxx

d

d22 yxr

uxxX d

Distortion

4

22

124

22

12

3

22

13

22

1

313

22

1

)(3))((3

,)3(2)3(

,22)3(

Z

YXYk

Z

YI

Z

YXsYXk

Z

sYXIc

Z

YXk

ZI

Z

XYYXsk

Z

sIb

Z

XYkI

Z

sXYYXk

ZIa

vvv

uuu

vvv

uu

vv

uuu

tv

vu

uvu

uvvuu

zyxzyx

IkZ

YXYI

Z

YXsYXIvIuI

sZ

YXYk

Z

YI

Z

YXYk

Z

YI

Z

YXXk

Z

XI

bXaYaZcXcYbZcvbvav

13

22

3

22

00

3

221

3

221

3

221

000

)())((

)()()(

)()()(

Linear equation for 6 motion parameters v0, , 5 intrinsic parameters and a distortion parameter

The parameters can be solved with 12 or more points by linear least square method.

Implementation

Differential imagesSo as to absorb the differences between a range intensity image and an intensity image2 images: horizontal, vertical. Prewitt operator.

-1-1-1

111

000

-101

-101

-101

Coarse to fineControl of resolution and of GaussianExtrinsic onlyv0+Intrinsicall

[Irani ICCV1998]

Experimental results

Range image sensor: ShapeGrabber PLM300　（ Slit laser, triangulation ， wavelength 670nm)

R-channel ， RAW format256019201280960 at registration

Digital camera: Nikon COOLPIX 5000(5M, 25601920 pixels,2/3” CCD, pixel dimension 3.4m?, f=7.1-21.4mm)

Measurement of a range image

312730 points

Summary

３ D-2D registration (for texture mapping, etc.)

•Projective geometry• Obtaining camera’s extrinsic and intrinsic parameters

•Range intensity (reflectance) image is useful•With correspondences

• Equivalent to {camera calibration / PnP} problems•Using optical flow constraint

• Explicit correspondences are not necessary• Linear equation for motion parameters