Visual Perception for Robots

Sven Behnke

Computer Science Institute VI

Autonomous Intelligent Systems

Our Cognitive Robots

Soccer robot

2

Communication robot Service robot

Flying robot

Exploration robot

Complete systems for example scenarios

Equipped with rich sensors

Our Humanoid Soccer Robots

3

Dynaped Copedo NimbRo-OP

Size: 95-114 cm, Weight: 6,6-8 kg

13-20 articulated joints

PC, wide-angle camera(s), IMU

Visual Perception

YUV color segmentation

Recognition of field, ball, goals,

obstacles, field lines, corners

Egocentric modeling

Probabilistic localization

4

[Schulz & Behnke, Advanced Robotics 2012]

Features for Localization

Goals

Field lines

Corners of lines

Side poles

Egocentric view Localization

Observation Likelihood

Lines Side poles

Line corners All features

RoboCup 2013 Final

7 NimbRo 4:0 CIT Brains => Won fifth time in a row.

Intuitive Multimodal Communication

Not keyboard, mouse, screen, but

Eye contact

Facing with head and trunk

Facial expressions

Gestures

Speech

Body language

Transfer established human communication techniques to the man-machine interface

Application: museum guide

Perception of Communication Partners

Detection and tracking of faces

Head pose estimation

Gesture recognition

Speech recognition (Loquendo)

[Bennewitz – Behnke:

Humanoids’05]

[Axenbeck, Bennewitz,

Behnke, Burgard:

Humanoids’08]

[Vatahska, Bennewitz,

Behnke: Humanoids’07]

Robotinho in Deutsches Museum Bonn

[Nieuwenhuisen & Behnke, Journal of Social Robotics (SORO), 2013] 10

Our Service Robots

11

Dynamaid Cosero

Size: 100-180 cm, weight: 30-35 kg 36 articulated joints PC, laser scanner, Kinect, microphone, …

2D Mapping of the Environment

12

3D-Mapping with Surfels

13

14

3D-Mapping with Surfels

3D-Mapping and Localization

Registration of 3D laser scans

Representation of point distributions in voxels

Drivability assessment trough region growing

Robust localization using 2D laser scans

15 [Kläß, Stückler, Behnke: Robotik 2012]

3D Mapping by RGB-D SLAM

Modelling of shape and

color distributions in Voxels

Local multiresolution

Efficient registration

of views on CPU

Global

optimization

Multi-camera SLAM

16

[Stückler, Behnke:

Journal of Visual Communication

and Image Representation 2013]

5cm

2,5cm

[Stoucken, Diplomarbeit 2013]

Learning and Tracking Object Models

Modeling of objects by RGB-D-SLAM

Real-time registration with current RGB-D image

17

Transfer of Object Knowledge

18

Non-rigid registration of

known models and actual

object

Transfer of grasp and

end-effector

[Stückler, Behnke: submitted to ICRA]

Analysis of Table-top Scenes and Grasp Planning Detection of Clusters above horizontal plane

Two grasps (top, side)

Flexible grasping of many unknown objects

[Stückler, Steffens, Holz, Behnke, Robotics and Autonomous Systems 2012] 19

Tool use: Bottle Opener

Perception of tool

tip

Extension of arm

kinematics

Perception of

crown cap

20

Tool use: Pair of Tongs

Perception of tool

tip

Extension of arm

kinematics

Estimation of

sausage pose

21

Our team NimbRo has won

the last three international

RoboCup@Home

competitions

Perception of Persons

Detection in laser scans

and tracking

Visual verification and

identification (VeriLook)

Systematic exploration

Speech recognition and

synthesis (Loquendo)

Gesture recognition

Natural gaze control

30cm 1m

22

[Stückler & Behnke, RoboCup 2010]

[Droeschel et al, ICRA 2011]

Object detection with laser or Kinect

Recognition based on color and texture features (SURF)

Object tracking

Visual Object Recognition

23

Semantic Mapping

Pixel-wise classification of RGB-D

images by random forests

Inner nodes compare color /

depth of regions

Size normalization

Training and recall on GPU

3D fusion through RGB-D SLAM

Evaluation on own data set and

NYU depth v2

24

Accuracy in % Ø Classes Ø Pixels

Silberman et al. 2012 59,6 58,6

Couprie et al. 2013 63,5 64,5

Random forest 65,9 68,6

3D-Fusion 67,0 70,9

Ground truth

Segmentation

[Stückler,

Biresev,

Behnke:

IROS 2012]

[Stückler et al., Accepted with minor revision for Journal of Real-Time Image Processing]

Learning Depth-Sensitive CRFs

SLIC+depth super pixels

Unary features: random forest

Height feature

Pairwise features

Color contrast

Directed angle

Depth difference

Normal differences

Results:

25 [Müller and Behnke, submitted to ICRA]

similarity

between

superpixel

normals

Random forest

CRF prediction

Ground truth

Object Class Detection in RGB-D

Hough forests make not only object class decision,

but describe object center

RGB-D objects data set

Color and depth features

Training with rendered scenes

Detection of object position

and orientation

Depth helps a lot

26 [Badami, Stückler, Behnke: SPME 2013]

Scene Class prob. Object centers Orientation Detected

objects

Bin Picking

Known objects in

transport box

Matching of graphs of 2D and 3D shape primitives

Grasp and motion planning

27

3D 2D

Offline Online

[Nieuwenhuisen et al.: ICRA 2013]

Articulated Objects: Doors

Door motion is important

Detection of changes between

maps

Instantiation of door models

Estimation of opening angle

from laser scan

Localization more reliable, more

precise

Navigation planning can use door

opening state

28 [Nieuwenhuisen, Stückler, Behnke, ICRA’10]

Adaptive Person Model

Model: geometric primitives, connected by joints

Registration through articulated ICP

Adaptation of primitive parameters to body

proportions

[Droeschel, Behnke: ICIRA 2011] 29

Hierarchical Object Discovery trough Motion Segmentation Motion is strong segmentation cue

Both camera and object motion

Segment-wise registration of a sequence

Inference of a segment hierarchy

30 [Stückler, Behnke: IJCAI 2013]

Autonomous Flight near Obstacles

Octocopter with many sensors

and strong computer

Multimodal obstacle detection

3D laser scanner

Stereo cameras

Ultrasound

Local obstacle avoidance

31 [Nieuwenhuisen et al., ECMR 2013]

Exploration in Rough Terrain

32

Wheeled robot with Intel 4th Core-i7 Quad

Omnidirectional RGB-D sensor

3D laser scanner

3D Mapping and 6D Localization

Efficient registration of

Multiresolution surfel maps

Global optimization

6D localization with 2D laser

scan using particle filter

33 [Schadler, Stückler, Behnke: accepted for SSRR 2013]

Conclusion

Robot operation in complex environments is challenging

Simple skills realized

Autonomous control is limited

Often perception is the problem

3D sensors are helpful

Need for further research

Possibilities with robots Multimodal sensor fusion

Active perception

Interactive perception

34

Thanks for your attention!

Questions?

35