2015.07.10.

BME IIT CG

3D reconstruction

with depth image fusion

Szirmay-Kalos László

WAIT 2015 BME IIT CG

Depth (range) cameras

SSIP

WAIT 2015 BME IIT CG

Application: limitations of compositingChroma keying

Augmented reality

Compositing can be based on color:

• Fixed order

• No shadows

• No reflections, refractions, cross illumination

SSIP WAIT 2015 BME IIT CG

Depth compositing (Zinemath)

SSIP

2015.07.10.

WAIT 2015 BME IIT CG

3D reconstruction of a point

t

=***

***

***

zzz

yyy

xxx

kji

kji

kji

R

tRpp += cw

f

(xc,yc,zc)

xi = fxc/zc

yi = fyc/zc

(cx, cy)(cx,cy)

(X,Y)

xi =(X-cx)sx

yi =(Y-cy)sy

(xi,yi)

zc

tdtRp +=+

−

−

= ccyy

xx

w zYXzscYf

scXf

),(

1

)(

)(

SSIP

Back projection

WAIT 2015 BME IIT CG

Stereo vision and active light

111 tdp += cw z222 tdp += cw z

111 tdp += cw z

222111 tdtdp +=+= ccw zz

SSIP

WAIT 2015 BME IIT CG

Time of flight sensors

Pulsed modulation:

Accurate time measurement

expensive

Continuous modulation

Periodic distance

SSIP WAIT 2015 BME IIT CG

Temporal noise filtering and back

projection for static camera

R, t

SSIP

2015.07.10.

WAIT 2015 BME IIT CG

3D triangle mesh: height field

Topology is a

height field

R, t

SSIP WAIT 2015 BME IIT CG

Linear and Bilateral filtering

∑∑∑ +=

+= )(|)(|

|)(|

)(|)(|)(

~opo

o

opop Dw

w

DwD

∑∑

−+

+−+=

|))()((||)(|

)(|))()((||)(|)(

~

popo

oppopop

DDww

DDDwwD

is

is

∑∑

•+

+•+=

),,(

)(),,()(

~

opo

opopop

r

DrD

Linear filter:

Bilateral filter:

Generalization:

|))()((||))()((||)(|),,( pIopIpopoopo −+−+=•+ cds wDDwwr

SSIP

WAIT 2015 BME IIT CG

Moving objects: optical flow

p

p + v

t

t+1

Classical optical flow constraint:

Gradient constancy:

)1,(),(),( ++−= ttEc vpIpIvp

)1,(),(),( ++−= ttEg vpIp

pIp

vpd

d

d

d

Smoothness:22

,,,,),(tyxtyx

Eyyyxxx

s ∂

∂

∂

∂

∂

∂+

∂∂

∂∂

∂∂

=vvvvvv

vp

Energy functional:

pvpvpvpv d),(),(),()( sgc EEEE βα ++= ∫

SSIP WAIT 2015 BME IIT CG

Spatio-Temporal filtering

p*=p - v

p*

q=p+o

q*

*),(

)**,(

|)*)()((|

|)*)()((|

*)*,,,(

qq

qp

qp

qIpI

qpqp

d

s

d

c

w

w

DDw

w

r

−

−

=

p

t

t+1

q*

p*

q*

q

p

SSIP

p

2015.07.10.

WAIT 2015 BME IIT CG

System

RGB

camera

depth

camera

Lens distortion

compensation

3D

mesh

Lens distortion

compensation

R, t

reprojection

+ upsampling

Optical

flow

Spatio-

temporal

filtering

Filtered

Point cloud

Rendering

with OpenGL

GLSL

shaders

WAIT 2015 BME IIT CG

Results

SSIP

WAIT 2015 BME IIT CG

Dynamic camera, static scene

FusionPoint cloud

Ob

jectProjection

Problems:

• in different images the camera changes

camera tracking based on static objects

• in different frames different points are visible

We need to maintain surface information

between points

SSIP

Solution (Curless/Levoy):

• Scene is represented by an emerging

distance field

WAIT 2015 BME IIT CG

Distance field (signed, truncated)

d

SSIP

d(x,y,z)

voxel

2015.07.10.

WAIT 2015 BME IIT CG

Rendering distance fields: ray marching

normal = grad d

d(x,y,z) < 0

d(x,y,z) > 0

SSIP

d(x,y,z) t

t

WAIT 2015 BME IIT CG

Ray marching on GPUfor(t = 0; t < exit; t += dt) {

float3 q = eye + raydir * t;

if (tex3D(volume, q) < 0) {

float3 normal = float3( tex3d(volume, q + float3(1/RES,0,0)) –

tex3d(volume, q - float3(1/RES,0,0)),

tex3d(volume, q + float3(0,1/RES,0)) –

tex3d(volume, q - float3(0,1/RES,0)),

tex3d(volume, q + float3(0,0,1/RES)) –

tex3d(volume, q - float3(0,0,1/RES)));

normal = normalize( normal );

color = lightint * kd * max(dot(lightdir, normal), 0);

distance = t;

return;

}

}

color = float3(0,0,0);

distance = -1;

return;

SSIP

WAIT 2015 BME IIT CG

Surface generation: marching cubes

d(x,y,z) > 0

d(x,y,z) < 0

0

1

4

2

3

5

6

7

Case: 000011102 =14

2; 0-1; 0-3; 2-6; 0-3; 3-7; 2-6 14

0;

14

0

255 0;

SSIP WAIT 2015 BME IIT CG

Volumetric fusion

1 0.9 0.5 -0.1 -0.8 -1

0.5

SSIP

2015.07.10.

WAIT 2015 BME IIT CG

Camera tracking: Iterative Closest Point

Backprojection

of current

image

Zero crossings of

the 3D distance

field

q1

q2

qnR, t

p1p2

pn

min1

)( →−⋅−= ∑i

iin

E tqRptR,

SSIP

Projective data association

WAIT 2015 BME IIT CG

System

3D distance field3D

mesh

Depth

To Distance

Volumetric

FusionRay

marching

ICP

Camera track

(PCL library)

Ne

w d

ep

th i

ma

ge

R, t

Depth + normal image

Point cloud

and mesh

generation

• Volumetric model: 128 x 128 x 128 +

• Image resolution: 640 x 480

• Speed: 30 FPS

• Hardware: NVIDIA 690 GT, CUDA

SSIP

WAIT 2015 BME IIT CG

Result on synthetic data

SSIP WAIT 2015 BME IIT CG

Additional filtering of distance fields

2015.07.10.

WAIT 2015 BME IIT CG

Additional filtering of distance field

WAIT 2015 BME IIT CG

Implementation and

Conclusions

• With GPU real-time 3D reconstruction is possible

• Height field representation:

–Simplified implementation: http://3d-scene.org/

• Volumetric reconstruction:

–Public domain simplified implementation: PCL library

–Commercial: KinectFusion

SSIP