Vision Based Control Vision Based Control MotionMotionMatt BakerMatt Baker

Kevin VanDykeKevin VanDyke

RobotsRobots

Today’s robots perform complex tasks with Today’s robots perform complex tasks with amazing precision and speedamazing precision and speed

Why then have they not moved from the Why then have they not moved from the structure of the factory floor into the “real” structure of the factory floor into the “real” world? What is the limiting factor?world? What is the limiting factor?

Vision

A “Seeing Robot”A “Seeing Robot”

A robot that can perceive A robot that can perceive and react in complex and and react in complex and unpredictable surroundingsunpredictable surroundings

This is not possible with the This is not possible with the marker-based systems in marker-based systems in use in most laboratory use in most laboratory vision-based control vision-based control systemssystems

Common reasons for failure of Common reasons for failure of vision systemsvision systems

Small changes in the environment can Small changes in the environment can result in significant variations in image result in significant variations in image datadata Changes in contrastChanges in contrast Unexpected occlusion of featuresUnexpected occlusion of features

RobustnessRobustness

Stable measurements of Stable measurements of local feature attributes, local feature attributes, despite significant despite significant changes in the image changes in the image data, that result from data, that result from small changes in the 3D small changes in the 3D environment environment [1].[1].

Enhanced TechniquesEnhanced Techniques

The Hough-TransformThe Hough-Transform Robust color classificationRobust color classification Occlusion predictionOcclusion prediction Multisensory visual servoingMultisensory visual servoing

Hough TransformHough Transform

Used to extract geometrical object features from Used to extract geometrical object features from digital imagesdigital images

Hough Transform (con’t)Hough Transform (con’t)

Features are extracted by detecting Features are extracted by detecting maximums in the imagemaximums in the image

Example geometric features encountered: Example geometric features encountered:

Lines:

Circles:

Ellipses:

Hough Transform (cont’d)Hough Transform (cont’d)

AdvantagesAdvantages Noise and background clutter do not impair Noise and background clutter do not impair

detection of local maximadetection of local maxima Partial occlusion and varying contrast are Partial occlusion and varying contrast are

minimizedminimized

NegativesNegatives Requires time and space storage that Requires time and space storage that

increases exponentially with the dimensions increases exponentially with the dimensions of the parameter spaceof the parameter space

Hough Transform (con’t)Hough Transform (con’t)

a real-time application of HT requires both a fast a real-time application of HT requires both a fast image preprocessing step and an efficient image preprocessing step and an efficient implementationimplementation

Implementation of a circle tracking algorithm based on HT

Robust color classificationRobust color classification

Color has high disambiguity powerColor has high disambiguity power Real-time is requiredReal-time is required

Supervised color segmentationSupervised color segmentation The color distribution of the current scene is The color distribution of the current scene is

analyzed and colors that do not appear in the analyzed and colors that do not appear in the scene are used as marker colorsscene are used as marker colors

These markers are then used as the input to the These markers are then used as the input to the visual servoing systemvisual servoing system

Colors represented by their hue-saturation value Colors represented by their hue-saturation value (H&S relate to color, V relates to brightness)(H&S relate to color, V relates to brightness)

Robust color classification (con’t)Robust color classification (con’t)

Color segmentationColor segmentation Choose four colors as Choose four colors as

marker colorsmarker colors Color markers brought Color markers brought

onto object we wish to onto object we wish to tracktrack

markers outlinedmarkers outlined Color distribution Color distribution

computedcomputed Initial segmentationInitial segmentation

Model-based handling of occlusionModel-based handling of occlusion

The previous two techniques take care of The previous two techniques take care of bad illumination and partial occlusionbad illumination and partial occlusion

What about aspect changes (complete What about aspect changes (complete occlusion)?occlusion)? Build and maintain a 3D model of the Build and maintain a 3D model of the

observed objects so they can be tracked observed objects so they can be tracked despite occlusion despite occlusion

Then use predictionThen use prediction

Tracking system modelTracking system model

Sensor dataFeature extraction

3D pose estimation

Robot control

Pose prediction

Visibility determination

Feature selection

Geometric model

Designed to handle aspect changes online

PredictionPrediction

Extract measurements of object features based on raw Extract measurements of object features based on raw sensor datasensor data

Estimate the spatial position and orientation of the target Estimate the spatial position and orientation of the target objectobject

Based on history of estimated poses and assumptions Based on history of estimated poses and assumptions about the object motion you can predict an object pose about the object motion you can predict an object pose expected in next sampling intervalexpected in next sampling interval

With predicted pose and 3D model we are able to With predicted pose and 3D model we are able to determine feature visibility in advancedetermine feature visibility in advance

Guide the feature extraction process for the next frame Guide the feature extraction process for the next frame without the risk of searching for occluded featureswithout the risk of searching for occluded features

Model-based handling of occlusion Model-based handling of occlusion (con’t)(con’t)

Efficient Hidden Line RemovalEfficient Hidden Line Removal Explicit modeling of curved object structures allows us Explicit modeling of curved object structures allows us

to remove to remove virtual linesvirtual lines – or lines that do not have a – or lines that do not have a physical correspondence in the camera imagephysical correspondence in the camera image

Object tracking with visibility Object tracking with visibility determinationdetermination

Multisensory ServoingMultisensory Servoing

Redundant information is used to increase the Redundant information is used to increase the performance of the servoing system as well as performance of the servoing system as well as the robustness against failing sensorsthe robustness against failing sensors

Vision Controlled Robot ModelVision Controlled Robot Model

ConclusionsConclusions We explored a variety of We explored a variety of

image processing image processing techniques that can techniques that can significantly improve the significantly improve the robustness of visual robustness of visual servoing systemsservoing systems

These techniques can be These techniques can be implemented in modern implemented in modern robot vision control robot vision control systemssystems

Techniques such as Techniques such as these will make machine these will make machine vision in robots a reality in vision in robots a reality in the near futurethe near future