1

Computer Vision and Robotics Research Group

Dept. of Computing Science, University of Alberta

http://webdocs.cs.ualberta.ca/~vis/research.htm

Page 1

An Introduction to Visual Servoing in Robotics

R. Tatsambon Fomena and C. Perez

2

Page 2

Visual servoing: the control concept

HRI

Specification

of Goal S*

World ACTION

PERCEPTION

Robot

+

Camera(s)

S*

S

+ -

perception for action

3

Page 3

Visual servoing: why visual sensing?

How to control the position of the end-effector of a robot with

respect to an object of unknown location in the robot base frame?

How to track a moving target?

A visual sensor provides relative position information

4

Page 4

Visual servoing: how can you use visual data in control?

Look then move

Visual feedback control loop

ACTION

PERCEPTION

Robot

+

Camera(s)

S*

S

- +

ACTION

PERCEPTION

Robot

+

Camera(s)

S-S*

5

Page 5

Quiz

Is visual servoing:

a) an open loop control

approach?

b) a closed-loop control

approach?

6

Page 6

Visual servoing: Ingredients for a fully integrated system

HRI

Visual tracking method

Motion control algorithm

HRI

Specification

of Goal S*

ACTION

PERCEPTION

Robot

+

Camera(s)

S*

S

7

Page 7

Visual servoing: HRI

Important for task specification

• point to point alignment for gross motions

• points to line alignment for fine motions

Should be easy and intuitive

8

Page 8

Visual servoing: Visual tracking

Crucial as it provides the necessary visual feedback

• coordinates of image points or lines

Should be reliable and accurate.

Camshift color tracker

provides 2D (x,y) coordinates

of the tracked objects

PERCEPTION

Current image

Tracker searches

for the end-effector

S

S=(x,y)

Selection of the set of

Measurements to use for

control

9

Page 9

Visual servoing: motion control algorithm

3 possible control methods depending on the selection of S: 2D,

3D, and 2 ½ D

(Corke, PhD, 94)

High bandwidth requires precise calibration: camera and robot-camera

3D VS

2D VS

2 ½ D VS

10

Page 10

Visual servoing: motion control algorithm

Key element is the model of the system

3D VS

2D VS

2 ½ D VS

Robustness to image noise, calibration errors

Suitable for unstructured environments

(Corke, PhD, 94)

2D VS 2 ½ D VS 3D VS

Abstraction for control

11

Page 11

Quiz

Which control method is more

sensitive to image noise? 1-a) 2D 2-a) 2 1/2D

1-b) 3D 2-b) 2D

1-c) 2 ½ D

12

Page 12

Visual servoing: motion control algorithm

Key element is the model of the system: how does the

image measurements S change with respect to changes in robot

configuration q?

can be seen as a sensitivity matrix

13

Page 13

Visual servoing: motion control algorithm

How to obtain ?

1) Machine learning technique

• Estimation using numerical methods, for example Broyen

2) Model-based approach

• Analytical expression using the robot and the camera projection model

• Example S=(x,y)

14

Page 14

Visual servoing: motion control algorithm

How to move the robot knowing e = S-S* and ?

Classical approach: the control law imposes an exponential decay

of the error

Classical control

15

Page 15

Visual servoing: motion control algorithm

:=VisualTracker(InitImage)

Init =

Init

While ( > T ) {

CurrentImage := GrabImage(camera)

:= VisualTracker(CurrentImage)

Compute =

Estimate

Compute

Change robot configuration with }

16

Page 16

Now do it yourself!