FUNDAMENTAL TECHNIQUES

1.0 SUNU WIBIRAMA 2011

Apart from any fair dealing for the purposes of research or private study, or criticism or review, this publication may only be reproduced, stored or

transmitted, in any form or by any means, with the prior permission in writing of the author. Citing is expected when part or whole content of

this material is used in academic publication

version AUTHOR YEAR

FOR 3D COMPUTER VISIONA Tutorial for Entry Level Researcher

FUNDAMENTAL TECHNIQUES FOR 3D COMPUTER VISION A Tutorial for Entry Level Researcher

Written by:

Sunu Wibirama, M.Eng Department of Electrical Engineering and Information Technology,

Faculty of Engineering, Gadjah Mada University

INDONESIA

Copyright Statement

This tutorial is taken from a research report entitled Real-time 3D Eye Movements Tracking and

Visualization using Dual Cameras Acquisition funded by AUN/Seed-Net-JICA Programme

in 2008-2010. Apart from any fair dealing for the purposes of research or private study, or criticism or

review, this publication may only be reproduced, stored or transmitted, in any form or by any means, with

the prior permission in writing of the author. Please cite the following paper when part or whole

content of this tutorial is used in academic publication:

S. Wibirama, S. Tungjitkusolmun, C. Pintavirooj, and K. Hamamoto, “Real Time Eye Tracking using Initial Centroid and Gradient Analysis,” in Proceeding of The 2009 6th International Conference of Electrical Engineering/Electronics, Computer, Telecommunications, and Information Technology (ECTI). ISBN: 978-1-4244-3388-9. Pattaya, Thailand. 6-8 May 2009, pp. 1058-1061.

ρ3 ρ2

Si

S

Xx fZ

= Si

S

Yy fZ

=

v

S

S

S

Xv Y

Z

=

v v

1

S

S

S

XY

vZ

=

0 0 0

0 0 00 0 1 0

fP f

=

w Pv=

S

S

S

fXw fY

Z

=

sz

S

S

S

S

XfZ

wYfZ

=

w

0 0( , )p x y

( , )p ix p ixx y

0

0 =

pix x i i

S Sx

S S

x k x sy xX Yk f sf xZ Z

= + +

+ +

0

0 =

pix y i

Sy

S

y k y yYk f yZ

= +

+

xk yk s ( , )p ix p ixx y u

[ ] , where and Tpix pix

u vu u v w x yw w

= = =

0

0

0. 0 0

0 010 1

Sx

Sy

S

Xxsu

Yv G v y

Zw

αα

= =

x xk fα = y yk fα = f

[ ]0 0

0 0 3 3

0 1 0 0 00 0 0 0 1 0 0 |0 0 0 0 0 1 01 10 0

x x

y y

x xs sG y y K I O

α αα α

= = =

.

w P v=

v w P

Gw PT v=

1 0 00 1 00 0 10 0 0 1

G

GG

G

XY

TZ

− − = −

Gw PRT v=

R R Rβ α=

cos 0 sin 00 1 0 0sin 0 cos 00 0 0 1

Rβ

β β

β β

− =

1 0 0 00 cos sin 00 sin cos 00 0 0 1

Rα

α αα α

= −

C Gw PT RT v=

1 0 00 1 00 0 10 0 0 1

O

OC

O

XY

TZ

− − = −

[ ]3 33

| 1

1

S

ST

S

Xu

R T Yv K I O

ZOw

− =

3 1T

R TO

−

WX [ ]1 T

S S Sv X Y Z= WX

1

WW

XX

=

[ ]Tu u v w=

( )Wu K R X t= −

u

[ ] | = 1 1

W WW

X Xu KR KRt M M X = − =

WX α Mα 0α ≠

[ ][ ] TTpix pixr x y p q= =

nX nr

1

nn n

Xr Mα =

ρ3 to ρ2

[ ]TX X Y Z= [ ]Tr p q=

ijm

1311 12 14

21 22 23 24

31 32 3433 1

Xmm m mp

Yq m m m m

Zm m mm

ααα

=

11 12 13 14

21 22 23 24

31 32 33 34

p m X m Y m Z mq m X m Y m Z m

m X m Y m Z m

ααα

+ + + = + + + + + +

α 31 32 33 34( )m X m Y m Z m+ + + [ ]p qα α α

( )( )

31 32 33 34 11 12 13 14

31 32 33 34 21 22 23 24

p m X m Y m Z m m X m Y m Z m

q m X m Y m Z m m X m Y m Z m

+ + + = + + +

+ + + = + + +

11 34,.....,m m

11

12

34

1 0 0 0 00 0 0 0 1 0

mX Y Z pX pY pZ p

mX Y Z qX qY qZ q

m

− − − −

− − − − =

0AM =

Tmn mm mn nnA U D V=

[ ] [ ] | |M K R K R t A b= − =

A KR= b KRt= −

1t A b−= −

( ), ,X Y Z

1k 2k 3k

1p 2p ( , )pix pixx y ( ), ,S S SX Y Z

0

0 =

pix x i i

S Sx

S S

x k x sy xX Yk f sf xZ Z

= + +

+ +

0

0 =

pix y i

Sy

S

y k y yYk f yZ

= +

+

0 0( , )x y f xk yk s ( , )d dx y ( , )pix pixx y r ( , )d dx y

( ) ( )2 20 0d dr x x y y= − + −

0 0 ; d dx x x y y y= − = −

( )2 4 61 2 3xc x k r k r k r= + +

( )2 4 61 2 3yc y k r k r k r= + +

( )221 22 2xd p x y p r x= + +

( ) 221 22 2yd p r y p x y= + +

( , )pix pixx y

( )pix d x xx x c d= + +

( )pix d y yy y c d= + +

( ) ( ) ( ) ( )

22 4 6 21 2 3 1 2

22 4 6 21 2 3 1 2

2 2

2 2

pix d

pix d

x x x k r k r k r p xy p r x

y y y k r k r k r p r y p xy

= + + + + + +

= + + + + + +

( ) ( )

22 4 6 21 2 3 1 2

22 4 6 21 2 3 1 2

2 2

2 2

pix d

pix d

x x xk r xk r xk r p xy p r x

y y yk r yk r yk r p r y p xy

= + + + + + +

= + + + + + +

( ) ( )( ) ( )

22 4 6 21 2 3 1 2

22 4 6 21 2 3 1 2

2 2 0

2 2 0

pix d

pix d

x x xk r xk r xk r p xy p r x

y y yk r yk r yk r p r y p xy

− − − − − − + =

− − − − − + − =

0AX = X

A X

( ) ( )( ) ( )

22 4 6 21

2 22 4 6 2

3

1

2

12 2

2 2 0

pix d

pix d

x x xr xr xr xy r x kk

y y yr yr yr r y xykpp

− − − − − − + − − − − − + − =

0AX =

Tmn mm mn nnA U D V=

( , )pix pixx y

1C 2C W 'R 'T ''R ''T 'u ''u X

1 2 3' , ' , 'T T Tm m m 1 2 3'' , '' , ''T T Tm m m

11 12 13 14 1

21 22 23 24 2

31 32 33 34 3

'' ' ' ' ' '

'' ' ' ' ' ' ' '

'' ' ' ' ' '

1

T

T

T

Xu m m m m m

Yu v m m m m M X m X

Zw m m m m m

= = = =

11 12 13 14 1

21 22 23 24 2

31 32 33 34 3

'''' '' '' '' '' ''

'''' '' '' '' '' '' '' ''

'''' '' '' '' '' ''

1

T

T

T

Xu m m m m m

Yu v m m m m M X m X

Zw m m m m m

= = = =

'u ''u X

1 2 3

1 2 3

' : ' : ' ' : ' : ''' : '' : '' '' : '' : ''

T T T

T T T

u v w m X m X m X

u v w m X m X m X

=

=

2 1 2 1

3 1 3 1

3 2 3 2

' ' ' ' '' '' '' ''' ' ' ' '' '' '' ''' ' ' ' '' '' '' ''

T T T T

T T T T

T T T T

u m X v m X u m X v m Xu m X w m X u m X w m X

v m X w m X v m X w m X

= =

= =

= =

2 1 2 1

3 1 3 1

3 2 3 2

' ' ' ' '' '' '' '' ' ' ' 0 '' '' '' '' 0' ' ' ' '' '' '' ''

T T T T

T T T T

T T T T

u m v m u m v mu m w m X u m w m Xv m w m v m w m

− − − = − = − −

'w ''w 'v− ''v−

( ) ( )( ) ( )

2 1 3 1 2 3

2 1 3 1 2 3

' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 0

'' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' 0

T T T T T T

T T T T T T

u w m v w m u v m v w m X u w m u v m X

u w m v w m u v m v w m X u w m u v m X

− − + = − =

− − + = − =

( ) ( )2 3 2 3' ' ' ' 0 '' '' '' '' 0T T T Tw m v m X w m v m X− + = − + =

( ) ( )( ) ( )

2 3 2 3

2 3 2 3

' ' ' ' ' ' ' ' ' ' ' 0

'' '' '' '' '' '' '' '' '' '' '' 0

T T T T

T T T T

u w m u v m X u w m v m X

u w m u v m X u w m v m X

− = − − + =

− = − − + =

( )( )

( )( )

3 1 3 1

3 2 3 2

' ' ' ' 0 '' '' '' '' 0

' ' ' ' 0 '' '' '' '' 0

T T T T

T T T T

u m w m X u m w m X

v m w m X v m w m X

− = − =

− = − =

3 1

3 2

3 1

3 2

' ' ' '' ' ' '

0'' '' '' '''' '' '' ''

T T

T T

T T

T T

u m w mv m w m

X AXu m w mv m w m

− − = = −

−

( )0X ≠ det( ) 0A =

Tmn mm mn nnA U D V=

3 1

3 2

3 1

3 2

3 1

3 2

' '' '

" "0

" "''' '''''' '''

T T

T T

T T

T T

T T

T T

u wv w

u wv w

u wv w

− − −

= = −

−

−

MATLAB IMPLEMENTATION

%---------------------------------------------% % Matlab implementation of 3D Reconstruction % % Written by: Chuchart Pintavirooj and % % Sorapong Aoothaphao % % Modified by: Sunu Wibirama % % Please use for educational purpose only % %---------------------------------------------%

%read input

clear

for j=1:3

switch (j)

case 1

a=imread('pic1.jpg','jpg');

aa=a(:,:,1);

figure(1);imagesc(aa);colormap('gray');

case 2

a=imread('pic2.jpg','jpg');

aa=a(:,:,1);

figure(2);imagesc(aa);colormap('gray');

case 3

a=imread('pic3.jpg','jpg');

aa=a(:,:,1);

figure(3);imagesc(aa);colormap('gray');

end

hold on;

%define 2D coordinate of each point

for i=1:8

[u(i,1),u(i,2)]=ginput(1);

plot(u(i,1),u(i,2),'r+');

end

% define world coordinate of the box

xx=[0 0 0 1

10.5 0 0 1

10.5 10.5 0 1

0 10.5 0 1

0 0 10.5 1

10.5 0 10.5 1

10.5 10.5 10.5 1

0 10.5 10.5 1];

%compute matrix M

for i=1:8

G(2*i,1)=0;

G(2*i,2)=0;

G(2*i,3)=0;

G(2*i,4)=0;

G(2*i,5)=xx(i,1);

G(2*i,6)=xx(i,2);

G(2*i,7)=xx(i,3);

G(2*i,8)=1;

G(2*i,9)=-u(i,2)*xx(i,1);

G(2*i,10)=-u(i,2)*xx(i,2);

G(2*i,11)=-u(i,2)*xx(i,3);

G(2*i,12)=-u(i,2);

G((2*i)-1,1)=xx(i,1);

G((2*i)-1,2)=xx(i,2);

G((2*i)-1,3)=xx(i,3);

G((2*i)-1,4)=1;

G((2*i)-1,5)=0;

G((2*i)-1,6)=0;

G((2*i)-1,7)=0;

G((2*i)-1,8)=0;

G((2*i)-1,9)=-u(i,1)*xx(i,1);

G((2*i)-1,10)=-u(i,1)*xx(i,2);

G((2*i)-1,11)=-u(i,1)*xx(i,3);

G((2*i)-1,12)=-u(i,1);

end

[U, D, V]=svd(G);

VT=V;

M=VT(:,end); %last columm

%M = transpose(reshape(M_vector, 4, 3));

m(1,1)=M(1);

m(1,2)=M(2);

m(1,3)=M(3);

m(1,4)=M(4);

m(2,1)=M(5);

m(2,2)=M(6);

m(2,3)=M(7);

m(2,4)=M(8);

m(3,1)=M(9);

m(3,2)=M(10);

m(3,3)=M(11);

m(3,4)=M(12);

switch (j)

case 1

u1=u;

m1=m;

case 2

u2=u;

m2=m;

case 3

u3=u;

m3=m;

end

vv=m*xx';

vv=vv';

vv(:,1)=vv(:,1)./vv(:,3);

vv(:,2)=vv(:,2)./vv(:,3);

title('Re project')

for i=1:8

plot(vv(i,1),vv(i,2),'go');

end

end%for j=1:3

size(u);m=ans(1);

%3D Reconstruction using Direct Linear Transformation

for i=1:8

A(1,1:4)=u1(i,1)*m1(3,1:4)-m1(1,1:4);

A(2,1:4)=u1(i,2)*m1(3,1:4)-m1(2,1:4);

A(3,1:4)=u2(i,1)*m2(3,1:4)-m2(1,1:4);

A(4,1:4)=u2(i,2)*m2(3,1:4)-m2(2,1:4);

A(5,1:4)=u3(i,1)*m3(3,1:4)-m3(1,1:4);

A(6,1:4)=u3(i,2)*m3(3,1:4)-m3(2,1:4);

[U, D, V]=svd(A);

zz(:,i)=V(:,4);

end

ake=zz;

zz=zz';

zz(:,1)=zz(:,1)./zz(:,4);

zz(:,2)=zz(:,2)./zz(:,4);

zz(:,3)=zz(:,3)./zz(:,4);

% Plot image

figure(4);

cla reset; hold on

d = [1 2 3 4 1 5 6 7 8 5 6 2 3 7 8 4];

plot3(zz(d,1),zz(d,2),zz(d,3),'b:');

plot3(zz(:,1),zz(:,2),zz(:,3),'b.','markersize',20)

A=m1(:,2:4);

%scaling = sqrt(m1(3,2)^2 + m1(3,3)^2 + m1(3,4)^2);

%m1 = m1 ./ scaling;

A=m1(:,2:4);

[Q R]=qr(A^-1);

R1=R^-1;

R1=R1/R1(3,3)

A=m2(:,2:4);

%scaling = sqrt(m2(3,2)^2 + m2(3,3)^2 + m2(3,4)^2);

%m2 = m2 ./ scaling;

A=m2(:,2:4);

[Q R]=qr(A^-1);

R2=R^-1;

R2=R2/R2(3,3)

A=m3(:,1:3);

%scaling = sqrt(m3(3,2)^2 + m3(3,3)^2 + m3(3,4)^2);

%m3 = m3 ./ scaling;

A=m3(:,2:4);

[Q R]=qr(A^-1);

R3=R^-1;

R3=R3/R3(3,3)