Robot Vision SS 2005 Matthias Rüther 1

710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU

Matthias Rüther

Kawada Industries Inc. DLR

Robot Vision SS 2005 Matthias Rüther 2

Organization

VO: Tuesday 14:15-15:45 Seminarraum ICG

Exam: Written Exam Oral Exam if Requested

KU:implementation of lecture topics in the real world (on the lab-robots)

Groups of three students Possible problems on the last slide Scheduling of topics: 8.3.2005 If you are interested: excursions to industrial vision

companies (Alicona Imaging, M&R)

Robot Vision SS 2005 Matthias Rüther 3

Time Table

1.3. : Introduction and Overview

8.3. : Projective Geometry (1)15.3. : Projective Geometry (2)12.4. : Projective Geometry (3)19.4. : Projective Geometry (4)

26.4. : Camera Technologies

3.5. : Shape From X (1)10.5. : Shape From X (2)24.5. : Shape From X (3)

31.5. : Robot Kinematics (1)7.6. : Robot Kinematics (2)

14.6. : Tracking of Moving Objects

21.6. : Visual Servoing / Hand Eye Coordination

Robot Vision SS 2005 Matthias Rüther 4

Literature

• Sciavicco, L., Siciliano, B., Modelling and Control of Robot Manipulators 2nd Ed., Springer, 2000

• Ballard D.H., Brown C.M., "Computer Vision", Prentice-Hall, 1982

• Sonka M., Hlavac V., Boyle Image Processing, Analysis and Machine Vision, Chapman Hall, 1998

• Nalva V.S., "A Guided Tour of Computer Vision", Addison-Wesley Publishing Company, 1993

• Horn B.K.P., "Robot Vision", MIT Press, Cambridge, 1986

• Shirai Y., "Three- Dimensional Computer Vision", Springer Verlag, 1987

• Faugeras O., Three-Dimensional Computer Vision A Geometric Viewpoint, MIT Press, 1993

• Hartley R., Zissermann A., Multiple View Geometry in Computer Vision, Cambridge, 2001.

Robot Vision SS 2005 Matthias Rüther 5

Robotics

What is a robot?"A reprogrammable, multifunctional manipulator designed to move

material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks"

Robot Institute of America, 1979

… in a three-dimensional environment.

Industrial– Mostly automatic manipulation of rigid parts with well-known shape in a

specially prepared environment.

Medical– Mostly semi-automatic manipulation of deformable objects in a

naturally created, space limited environment.

Field Robotics– Autonomous control and navigation of a mobile vehicle in an arbitrary

environment.

Robot Vision SS 2005 Matthias Rüther 6

Experimental/Industrial/Commercial Robots

Robot Vision SS 2005 Matthias Rüther 7

Industrial Robots

Robot Vision SS 2005 Matthias Rüther 8

Challenging Environments

Robot Vision SS 2005 Matthias Rüther 9

Service and Assistance

Robot Vision SS 2005 Matthias Rüther 10

FRIEND Project

Robot Vision SS 2005 Matthias Rüther 11

Robot vs Human

Robot Advantages:

– Strength

– Accuracy

– Speed

– Does not tire

– Does repetitive tasks

– Can Measure

Human advantages:

– Intelligence

– Flexibility

– Adaptability

– Skill

– Can Learn

– Can Estimate

Robot Vision SS 2005 Matthias Rüther 12

Robotics: Goals and Applications

Robotics does not intend to develop the artificial human![Whitney, D. E., Lozinski, C. A. and Rourke, J. M. (1986) Industrial robot forward calibration method and results.]

Goal: combine robot and human abilities.

Applications: – Automation (Production)

– Inspection (Quality control)

– Remote Sensing (Mapping)

– Man-Machine interaction („Cobot“)

– Robot Companion (Physically challenged people)

– See [Brady, M. et. al. (eds). „Robot Motion: Planning and Control“]

Robot Vision SS 2005 Matthias Rüther 13

What can Computer Vision do for Robotics?

Accurate Robot-Object Positioning

Keeping Relative Position under Movement

Visualization / Teaching / Telerobotics

Performing measurements

Object Recognition (see LV „Bildverarbeitung u. Mustererkennung“, „Bildverstehen“, „AK Computer Vision“)

Registration

Visual Servoing

Robot Vision SS 2005 Matthias Rüther 14

Combining Computer Vision and Robotics

Abstraction level

Motor Modeling: what voltage should I set now ?

Control (PID): what voltage should I set over time ?

Kinematics: if I move this motor somehow, what happens in other coordinate systems ?

Motion Planning: Given a known world and a cooperative mechanism, how do I get there from here ?

Bug Algorithms: Given an unknowable world but a known goal and local

sensing, how can I get there from here?

Mapping: Given sensors, how do I create a useful map?

Localization: Given sensors and a map, where am I ?

low

high

Vision: If my sensors are eyes, what do I do?

Robot Vision SS 2005 Matthias Rüther 15

Computer Vision

What is Computer Vision?

"Computer Vision describes the automatic deduction of the structure and the properties of a (possible dynamic) three-dimensional world from either a single or multiple two-dimensional images of the world" [Nalva VS, "A Guided Tour of Computer Vision"]

Measurement– Measure shape and material properties in a 3D environment. Accuracy

is important.

Recognition– Cognitive systems interpret a 3D environment (object classification,

categorization). Systems are allowed to fail to a certain extent (similar to humans).

Navigation– Navigation Systems orient themselves in a 3D environment.

Robustness and time are important.

Robot Vision SS 2005 Matthias Rüther 16

Measurement

„Shape from X“ techniques measure shape properties of objects from 2D digital images.

– Shape from Stereo: two cameras obeserve an object from different viewpoints (similar to human eye).

– Shape from focus: limited depth of focus allows to measure object-camera-distance.

– Shape from structured light: a light pattern is projected on the object, the pattern deformation gives shape information.

– Shape from Shading: an object is illuminated from a single direction. Light reflection depends on object shape and follows a reflectance function.

Robot Vision SS 2005 Matthias Rüther 17

Shape from Stereo

Robot Vision SS 2005 Matthias Rüther 18

Shape from Stereo

Robot Vision SS 2005 Matthias Rüther 19

Shape from Focus

Robot Vision SS 2005 Matthias Rüther 20

Shape from Structured Light

Structured Light Sensor

                                                                             

                                                             

                                                         

Figures from PRIP, TU Vienna

Robot Vision SS 2005 Matthias Rüther 21

Shape from Shading

Robot Vision SS 2005 Matthias Rüther 22

Navigation

SLAM: Simultaneous Localization and Mapping. – Where am I on my map?

– If the place is unknown, build a new map, try to merge it with the original map.

Visual Odometry: calculate the relative motion of the camera between two frames. Summing up the motion gives the camera path. Error propagation!

Visual Servoing: move to / maintain a relative position between robot end effector and an object.

Tracking: continuously measure the position of an object within the sensor coordinate frame.

Robot Vision SS 2005 Matthias Rüther 23

SLAM

Mapping:

Robot Vision SS 2005 Matthias Rüther 24

SLAM

The final map:

Robot Vision SS 2005 Matthias Rüther 25

SLAM

Navigation:

Robot Vision SS 2005 Matthias Rüther 26

Visual Odometry

Robot Vision SS 2005 Matthias Rüther 27

Visual Servoing

Robot Vision SS 2005 Matthias Rüther 28

Tracking

Robot Vision SS 2005 Matthias Rüther 29

Tracking

Robot Vision SS 2005 Matthias Rüther 30

Registration

Registration of CAD models to scene features:

Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

Robot Vision SS 2005 Matthias Rüther 31

KU: Student Problems

Shape from Stereo 3 students

Shape from Focus 3 students

Shape from Structured Light:Laser 3 students

Shape from Structured Light:Pattern 3 students

Shape from Shading 3 students

Robot Kinematics 3 students

2D Grip Planning 2..3 students

2D Visual Servoing 3 students

2D Tracking 3 students

Registration / Model Fitting 3 students

Visual Odometry + Randomized RANSAC 3 students