Efficient Surface and Feature Estimation inRGBDZoltan-Csaba Marton, Dejan Pangercic, Michael BeetzIntelligent Autonomous Systems Group

Technische Universität München

RGB-D Workshop on 3D Perception in Roboticsat the European Robotics ForumVästerås, April 2011

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Introduction

Motivation:I High frame-rate and RGB-D allow multi-view classification

or dynamic scenes even if processing methods are fastI Vision has a long history, but there is no standard

geometric featureI Used 3D is often an image feature on the depth image

Goal:I We are interested in geometry, not points, and in a

compact way (PFH versions for example arehigh-dimensional)

I Reuse as much as possible: voxelization, smooth surfacereconstruction, etc.

I Combination with color that is very integrated and allowspart based detection

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Getting Local NeighborhoodsAdvantages of Voxelization

I Pixel window is good, butvoxelization assures spacialcloseness

I Supports e.g. occupancy labels:mapping, collision avoidance,visibility checks

I Speeding up model fittingI PCL: added way to retrieve

points of a cell and its neighborsto downsampling

I Point-wise features can beparallelized but voxel-wise iseven faster

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Radius-based Suface Descriptor (RSD)Theory and Approximation

I Estimate minimum and maximum curvature radius fromangle/distance pairs:

p1p2

n1n2

αr

d

0 10 20 30 40 50 60 70 80 90

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

sqrt(1-cos(a))Taylor De-composition Most Significant Element (linear)

Angle (a) between the normals of two points

Dis

tanc

e (

d)

rela

tive

to th

e m

axi

mum

dis

tanc

e

Near-linear relation between distance and angle: the radius [m]

d(α) =√

2r√

1− cos(α)⇒ d(α) = rα+rα3

24+ O(α5)⇒ d = r · α

[IROS2010:] Marton et al., General 3D Modelling of Novel Objects from aSingle View

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Radius-based Surface Descriptor (RSD)Principle Radii

I Local variation of normal angles by distance (similar toPFH and “spin images with normals”):

Synthetic Data

plane sphere sphere side corner cylinder cylinder top cylinder side edge handle

Real Data

small cylinder medium cylinder big cylinder handle1 handle2 handle3

The tilt angles of the lines starting from bottom left cornercorrespond to the physical radii: smallest tilt that still coversoccupied cells to the min. radius, while the biggest to the max.

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Radius-based Surface DescriptorComparison to (F)PFH

Cheaper and more descriptivewith only 2 values

I No learning needed due tophysical meaning:

0 r_min 0.2 m

r_m

ax0.

2 m

PLANECYLINDEROR RIM

SPHERE

EDGE

CORNER OR NOISE

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Integrating Computational StepsSmoothing, Normal Estimation and Surface Radii at once, per Voxel

I Idea: use points in voxelsto smooth, downsample,estimate normals and radiiat once based onpolynomial approximation.

I Results on data coming from Kinect (minimum andmaximum radii for a box and a mug):

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Voxel-based RGB-D Object FeatureAdvantage of Considering RGB and Depth

I RGB and 3D information are discriminative in differentdimensions (see monochrome cube and dice point clouds)

I The question is how to combine them efficiently into thesame framework

I Combination of GRSD andC3 − HLAC

I Voxelized point cloudclusters

I Adjacency voxels statisticsI (Normalized)

Histogram-based algorithmWork done together with Asako Kanezaki.

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Voxel-based RGB-D Object FeatureSetup and Early Results

Rotating table(for training data)

Kinect RGBD Device =+

RGB Shape

RGB Correlations

Sh

ape

Recognition of Objects

Database (63 Objects)

SVM or LSM Models

I The feature is additive, so objectscan be recognized based on theirparts or using a sliding box

I Feature extraction takes 57 µsper point on a single core

I Initial classification results on 63objects (many of them of thesame type or brand) using SVM:76.4% (0.05 s per object)

I More training and labeled testingdata is needed (thanks Kevin!)

I Better classification: learningviews separately but classifyingobjects (+12% in similar setup)

Work done together with Asako Kanezaki.RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

ConclusionsAdvantage of Considering RGB and Informative Depth

Efficient estimation of the physical radius of local surface:I uses in CAD matching, model fitting and segmentationI more descriptive features (together with RGB)

Mozos, Marton, Beetz: “Using Web Catalogs to Locate and Categorize Unknown

Furniture Pieces in 3D Laser Scans”, Robotics & Automation Magazine (in press)

Integration with RGB to obtain a global feature:I efficient integration of well matched RGB and 3D features

Futur work:I deeper integration and improving the geometric part (e.g.

including “voxelized“ VFH or others from PCL)I quantitative comparisons to find strengths and weeknesses

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz

Introduction Descriptive Local Geometric Feature RGB-D Object Feature Conclusions

Q&A

Thanks!

Intelligent Autonomus Systems Group:http://ias.cs.tum.edu

Point Cloud Library:http://pointclouds.org

Contact:marton@cs.tum.edu

RGB-D Workshop, euRobotics, April 2011

Efficient Surface and Feature Estimation in RGBD Marton, Pangercic, Beetz