The 18th Meeting on Image Recognition and Understanding2015/7/29

Depth Image Enhancement

Using Local Tangent Plane Approximations

Kiyoshi Matsuo Yoshimitsu AokiHokuyo Automatic Co., Ltd. Keio University

○

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

• Background– Research purpose– Related work

• Proposed method (overview and highlight)– Local coordinates based depth image enhancement

• Experimental results

• Conclusion

2

Contents

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

Depth image enhancement for consumer RGB-D cameras

3

Research purpose

Ex. Soft Kinetic DS311

> Designed for human interaction applications

* Consumer RGB-D camera

+ low cost+ portable

- low accuracy- low resolution

Enhancement of both the accuracy and resolution could help many applications.

Ex. fine-grained object recognition, precise indoor navigation, and etc…

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

4

Related work

“An application of Markov random fields to range sensing” [ NIPS 2006 Diebel et al. ]“High quality depth map upsampling for 3d-tof cameras” [ ICCV 2011 Park et al. ]“A Joint intensity and depth co-sparse analysis model for depth map surperresolution” [ ICCV 2013 Kiechle et al. ]“Depth map inpainting under a second-order smoothness prior” [ Image Analysis vol. 7944 Herrera et al. ] etc.

1. Measured depth data is used as optimization data prior.2. Various pixel based information is used as optimization smoothness prior. (gradient, segmentation, edge saliency, non-local mean, and co-sparseness etc.)

3. Depth enhancement is achieved by an optimization of over the image plane.

[ Diebel et al. 2006 ]

Global optimization based enhancement Markov Random Field

“Joint bilateral upsampling” [ ACM transactions on Graphics 2007 Kopf et al. ]“Spatial-depth super resolution for range images” [ CVPR 2007 Yang et al. ]“Pixel weighted average strategy for depth sensor data fusion” [ ICIP 2010 Garcia et al. ]“Upsampling range data in dynamic environments” [ CVPR 2010 Dolson et al. ]“Joint geodesic upsampling of depth images” [ CVPR 2013 Liu et al. ] etc.

Local filter based enhancement1. Local measured depth data is summarized by using similarity weights.2. Similarity weights are defined on the global image plane coordinates. (pixel-distance, color-difference, gradient, and color geodesic distance, etc.)

3. Depth enhancement is achieved by local calculations on the image plane.

Joint bilateral upsampling filter

[ Kopf et al. 2007 ]

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

5

Related work

Previous enhancement methods use pixel-coordinates. These methods smooth the surfaces to be parallel to the image plane.

Measuring object

Image plane

No geometric relationship

The geometries of surfaces are not take into account,(and are sometimes corrupted by them)

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

6

Contributions

To improve accuracy, we introduce local tangent planes as local coordinates to handle the geometries.

Local coordinatesGlobal coordinates

* pixel-coordinates of the image * charts defined on each local tangent * depend only on the measuring device (independent of the measuring surfaces)

* depend on the local geometries of measuring surfacesAccurate depth image enhancement is achieved.

(especially in noisy cases)

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

7

Proposed method (overview)

A pair of depth and color images

* Input data

2. Surface reconstruction * Output data

A high-resolution accurate depth image

Only overview of our method(more details will be shown at our poster.)

1. Estimation of local tangent planes

Proposed in ICCVW 2013

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

8

How to use local coordinates ?

Measured surface

Height smoothed surface

To recover smooth surfaces, we smooth normal components of surfaces on each local coordinates

Colored by the height fromthe tangent plane

Smoothed surface

These filters on local coordinates smooth the surfaces while preserving the local geometries of local tangent planes.

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

The interior regions of surfaces were reconstructed more accurately. 9

Experimental results

MRF [Diebel and Thrun]

PWAS [Garcia et al]

Previous ours [Matsuo and Aoki] Our method

Source RGB-D image

Captured by SoftKinetic DS311

The 18th Meeting on Image Recognition and Understanding2015/7/29

Unique Product & Best SensingHOKUYO

10

Conclusion

• Local coordinate based depth image enhancement.– Local tangent planes are used as local coordinates.

• Normal component smoothing reduces granular noises.– These filters preserve the local geometries of local tangents.

• The effectiveness is shown by real sensor data.

The 18th Meeting on Image Recognition and Understanding2015/7/29

Thank you for this opportunity to present to you today.And thank you for your attention !

If you have any questions and comments,visit our poster IS 2-4.