How to Achieve Accurate 3D Robotic Vision · 7/21/2011 · Types of 3D Vision Camera System Vision Methodology Required Comments Single Stationary 2D, 2½D, 3D 2½: ONLY if height

An educational webinar sponsored by

Universal Robotics and Yaskawa Motoman

Robotics

Welcome to Accurate

Robotic 3D Vision

Speakers

Hob Wubbena, Director of Marketing, Universal Robotics • Engineering, technical planning and product marketing for Hewlett-Packard

and Agilent Technologies for 25 years

• Aerospace & defense, telecommunications, test & measurement, electronic manufacturing, and the chemical process industries

• 3 patents, numerous marketing awards, ~12 published articles

• B.S. Civil Engineering, University of Wisconsin; Masters Business, Denver University

Erik Nieves, Technology Director, Yaskawa Motoman Robotics • Management and engineering, for Motoman Robotics for 20 years; currently

focused on corporate strategic technology roadmap and emerging applications

• Standards Development & Education Committees for the Robotics Industries Association (RIA), “Ask the Experts” forum, RIA

• Many published robot technology articles including control and metrology

• B.S. Mathematical Physics, Southwestern Adventist University

7/21/2011 Page 2 Copyright © 2011 Universal Robotics

Experts

Aditya Nawab, R&D Manager, Universal Robotics • Extensive design experience in computer vision, hardware/software

integration, autonomous vehicle development and industrial robotics

• Worked on numerous Department of Defense and NASA robotic projects

• His work at Universal focuses on dexterous manipulation, force control, sensory-motor coordination, forward and inverse kinematics, dynamics of articulated manipulators, and R&D project management

• B.S. Mechanical Engineering, Florida Atlantic University; M. S. Mechanical Engineering, University of Florida

Greg Garmann, Technology Leader, Yaskawa Motoman

Robotics • Technology Leader at Yaskawa America, Motoman Robotics Division,

and has been involved in automation for more than 25 years.

• Developed vision capabilities for robotics guidance using 2D and 3D technologies

• B.S. Computer Engineering, Wright State University


AGENDA: Accurate Robotic 3D Vision

• Introduction to Robotic 3D Vision

• Types of 2D, 2½D & 3D robotic vision systems

• Elements of Accuracy

• Choosing a 3D vision system

• Q&A


Robotic 3D Vision Introduction

• Beyond scope of webinar:

– 3D vision (movies) anaglyphs for depth perception

– Photometric stereo from telecentric camera

– Depth From Focus (DFF)

– 3D Mosaicking

– Vision analysis tools (blob analysis, face recognition)

– Time of Flight (TOF)

– Laser line projector

• Robotic vision delivers real-time data about objects

– Part Inspection

– Vision Guidance (X, Y, Z position & Rx, Ry, Rz pose)

• 3D accuracy requires both intrinsic camera calibration and hand-eye calibration

Future webinar


Repeatability vs. Accuracy

• Repeatability is important for automated tasks where robot picking and placing to same locations

• Relative Accuracy is important for random tasks where the spatial location is constantly changing:

– Random box depalletization

– Random box moving

– Random part picking

– Random bin picking

Page 6 Copyright © 2011 Universal Robotics 7/21/2011

Intrinsic Camera Calibration

Calibration process

• Camera sensor information

• Fiducial in full FOV

• 15 varying fiducial images

Data in 3 coordinate systems

• Image, Camera, Object

Image rectification (ΔY)

• Removes perspective and/or lens distortions

If multiple cameras

• Computes disparity, distance, 3D coordinates

• Aligns image on common plane

Page 7 Copyright © 2011 Universal Robotics

Z

X

Y

Image

Object Camera

Z

X

Y

Z

X

Y

Focal

length

FOV

Field of View

7/21/2011

3D Vision & Robotic 3D Vision

3D Vision: stereo vision with depth (z) perception resulting from the disparity (Δx) of different images of the same object

3D Accuracy: resolution in the Δz depth of an object viewed from stereoscopic vision. It is based on quality and geometry of cameras. Δy distance difference between the cameras (placement and orientation) is corrected

during intrinsic camera calibration, resulting in a rectified image.

The Δx disparity of each camera's image of same pixel point in space is computed through 3D software algorithms.

Robotic 3D Vision • 3D Vision with vision guidance

for a robot delivers in real-time: – Position (X, Y, Z)

– Pose/orientation (Rx, Ry, Rz)

– Interactively, with tool offsets

– Choice of robot tool affects vision requirements (vacuum and grippers

can work within 2-3 mm)


Hand–Eye Calibration

• First, calibrate cameras

• Then Hand-Eye calibration enables data transform across coordinate systems

– Robot “hand” tool guided by camera “eye”

• RESULTS FOR: – VISION GUIDANCE

– Part Inspection (optional)

– 3D Part Model creation (optional)







• Q&A


Types of 2D, 2½D & 3D Vision Systems

•Single Camera •Single camera with known

calibration plate, OR multiple

images to obtain depth, OR add

TOF/laser

•Move camera OR add 2nd camera

with shape-matching (model) OR

TOF/laser with surface-matching

(point cloud)

•Vision data in same plane •Z data added •Rx, Ry data added

•X, Y, Rz(angle) only; no Z •Postion X, Y, Z, Rz (angle) only •Position X, Y, Z, & Pose Rx, Ry, Rz

•No tilt (Rx, Ry) •No tilt (Rx, Ry) •Allows for part tilt

•Camera distance constant •Camera distance can change •Camera distance can change

Rz

X

2D

Rz

X

Z

2½D

Rz

X

Z

Rx

3D #1 #2


Types of 3D Vision

Camera System Vision Methodology Required Comments

Single Stationary

2D,

2½D,

3D

2½: ONLY if height is

known & fixed. Height can

be variable with TOF

3D: Shape-match

Good if plane is fixed

Single Moving 3D 3D: with shape-match or

surface-match

Can measure depth,

Slower than Stereo

Binocular Stereo 3D Shape-match (model) or

surface-match (point cloud)

Surface-match requires

TOF or laser

Multiple Stereo 3D Mosaicking

Shape-match (model)

Surface-match requires

TOF or laser,

Slower than binocular


Single Stationary Camera

• Using 3D Model

• Shape-Based 3D Matching

– Speed up by using only a region of interest, eliminate unnecessary edges

• Surface-Based 3D matching


Single Camera, Multiple Positions

• Parts must be stationary for measurement (e.g. Stacked parts)

• May require multiple images of same object from different camera locations

• Good for vision guidance and part inspection


Binocular Cameras

• Only one image required

• Faster than multiple image approach

• Can find randomly located objects

• Use of standard off-the-shelf cameras

• Can provide full field of view of robot operating envelope







• Q&A


3D Vision Accuracy and Robot Operating Envelope Geometry

Camera 2

Z =

ob

ject

dis

tan

ce

b=baseline distance

f=focal

length

P2 (x2,y2,z2)

Object

Δz

Virtual

Image 1

Virtual

Image 2

Camera 1

3D Accuracy, Δz, is affected

by: •Distance, b, between pair of

cameras

•Distance, Z, from object to

cameras

•Disparity, Δd, offset in x between

image points, P1 & P2

•Focal length, (pixels, not mm)

Δz = Z2

· Δd

f · b where f (pixels) = f of Lens (mm) * Camera Horiz. Resolution

(px) / CCD Horiz. sensor width (mm)

Z

X

P1 (x1,y1,z1)


Robot Work Envelope

3D Camera Positions for Objects on Flat Surface

±150mm depending on part & envelope

dimensions

2nd pair orthogonal ~100mm apart @ 45˚

Parts / Boxes on Flat Surface

Robot Envelope Depth

Robot

Envelo

pe

Heig

ht

1st pair cameras just above robot at ~100mm apart, 45˚ below horizontal

NOTE: Rectangular objects require about 200px by 200px for determining position

& pose

NOTE: Irregular objects require about 300px by 300px for determining

position & pose

Ro

bo

t


Robot

Operating Envelope

Camera Positions for Objects in Bin

1st pair cameras just above robot at ~100mm apart, 60˚ below horizontal

±150mm depending on part & envelope

dimensions

2nd pair orthogonal ~100mm apart @ 60˚

Parts or Boxes in Bin

Envelope Depth

Robot

opera

ting

envelo

pe H

eig

ht

NOTE: Rectangular objects require about 200px by 200px for determining position

& pose

NOTE: Irregular objects require about 300px by 300px for determining

position & pose

Ro

bo

t


Robot Accuracy & Repeatability

• Accuracy: how close a robot can reach a commanded 3D position

• Accuracy varies with robot speed, robot reach, and with payload

• Accuracy of a robot is determined by three elements of the system:

– Resolution of the control system

– Error operating the robot arm under closed loop servo operation

– Imprecision of mechanical linkages, gears & deflections under load

• Repeatability: the ability to duplicate an action or a result every time

• See ISO 9283


Typical Robot Repeatability

• Plotted over 100 robots from 3 companies

– 2Kg to 1,200Kg

– SCARA, material handling, welding, assembly…

• Repeatability loosely a function of:

– Payload (Kg)

– Reach (M)

– Speed (˚/sec)

• Best repeatability approaches limit line


Robot Accuracy

• Accuracy is typically worse than repeatability, not constant over workspace

• Full robot calibration:

– Accuracy = 2X – 4X of Repeatability; can approach 1X

• Typical robot calibration:

– Accuracy = 3X – 5X of Repeatability

Best (2X) Typ. (4X) Repeatability Accuracy Accuracy Payload, Reach, Speed

0.015mm 0.03mm 0.06mm 2kg 0.6M 375 deg/s

0.03mm 0.06mm 0.12mm 5kg 0.8M 270 deg/s

0.06mm 0.12mm 0.24mm 35kg 1.3M 170 deg/s

0.1mm 0.2mm 0.4mm 50kg 1.6M 170 deg/s

0.2mm 0.4mm 0.8mm 275kg 2.5M 90 deg/s


Overall 3D Robotic Vision System Accuracy System Accuracy results (go to Universal Robotics 3D Made Easy

Calculator at www.universalrobotics.com/calc


http://www.universalrobotics.com/calc

Camera Selection

Scalable vision systems enable a wide variety of cameras to fit the robot operating envelope geometry and accuracy needs


USB Webcams & GigE Camera

Selection

Working Distance from Cameras to Objects (Camera selection provides <1mm camera accuracy, resulting in a system accuracy of 1-3mm

within the workspace. Consult Universal Robotics for final configuration)

0.3 - 0.5M 1.0M 2.0M 2.5M

Medium Size Parts & Boxes (workspace < 1.0M wide x 1.0M deep x 1.0M high)

USB 1.0 MP, f3.7 USB 1.3 MP, f3.5 GigE 1.4 MP, f3.5 (0.5M workspace)

USB 1.3 MP, f4.5-f6 USB 2.0 MP, f3.5-4.5 GigE 1.2 MP, f3.5- 4.5 GigE 1.4 MP, f4.5 –f6

USB 2.0 MP, f6 – f8 GigE 1.2 MP, f6 - f8 GigE 1.4 MP, f8 -f12

USB 2.0 MP, f8 - f12 GigE 1.2 MP, f8 –f12 GigE 1.4 MP, f12 -f16

Boxes, Parts, & Pallets (workspace < 1.5M x 1.5M x 1.5M)

Consult with Universal Robotics

Consult with Universal Robotics

USB 1.3 MP, f6 - f8 USB 2.0 MP, f4.5 – f6

USB 2.0 MP, f6 –f8 GigE 1.4 MP, f8 – f12

7/21/2011






• Q&A


Attributes of Low-Cost 3D Vision

WORKSPACE: __ Robot operating envelope within 1.5M x 1.5M?

__ Cameras mounted within 2.5M of workspace?

ROBOT: __ Part-box picked up with vacuum or grippers? (sets 3mm accuracy)

__ Part or box weighs < 50 Kg? (robot size)

__ ≤10 parts-boxes/min or 600/hour? (can USB2.0 cameras)

PARTS OR BOXES: __ Part has visible edge and/or boundary? (enables shape-matching)

YES: White egg with black background; surface with reflectivity < 75

NO: Dull black molded rubber ball with black background

__ Part/box have CAD model? (shape-matching; no special lighting)

__ Only one part? (scalable to multiple parts, but additional work)


3D Vision at 2D Prices

• Standard off-the-shelf components scalable to multiple parts with world-class software

$-

$10

$20

$30

$40

$50

$60

1985 1990 1995 2000 2005 2010

Vis

ion

Syste

m $

K

3D Vision at 2D Prices

2D Machine Vision

3D Machine Vision


Robotic Vision Options

MotoSight 3D

Spatial Vision

2½ D Surface Traditional 3D

Machine Vision

Precise 3D

Inspection

Random Part Pick Automated Assembly Machine Vision Precise Part

Inspection/Guidance

$8-$10k $8k $12k - $18k $20k - $25k

Two Fixed Cameras One Camera on Robot

Arm

One – Multiple

Fixed or on Robot

One Camera on Robot

Arm

3D: complex parts 2½ D: planar parts only 3D: complex parts 2½ D: planar parts only

3D shape match 3D surface match 3D geometric

pattern match

3D geometric pattern

match

Stationary & Moving

Parts Stationary Parts

Stationary & Moving

Parts Stationary Parts

Three – Ten Parts One Part One Part One Part

1 - 5 mm accuracy

@ 0.5 – 2.5M

1 - 5 mm

@ 0.5 - 2.0M

~0.2 - 2 mm

@ 0.1 - 0.5M

~ 0.12 mm

@ 0.08M


Key Benefits of Robotic 3D Vision

• Locates 3D Objects in 3D Space

– Identifies 6 degrees of freedom X, Y, Z, Rx, Ry, Rz

– Not just 2D flat parts in space

• 3D Shape Matching

– Identifies complex objects

– Handles varying lighting conditions

– Identifies partially hidden objects

• Scalable Precision

– Choose just the right cameras for your needs

– Only ~3mm accuracy needed for vacuum pick or grippers

• 3D Vision at 2D Prices

– MotoSight 3D Spatial Vision U.S. List $10,000


Robotic 3D Vision Summary

• Use 3D where part complexity, location randomness, or motion require it

– Vision guidance

– Part inspection

• 3D is a mainstream solution; requires choosing right tool & approach

– Only pay for what you need; end effectors can manage from 2-3mm

• Motoman robots are some of the most accurate robots in the world

• Universal Robotics 3D Made Easy Calculator at www.universalrobotics.com/calc


http://www.universalrobotics.com/calc






• Q&A


End


3D Calculator Inputs

3D Calculator Inputs

5/31/2012


Ro

bo

t E

nve

lop

e H

eig

ht

Obj. Depth

Center (0, ½H, ½D)

Ro

bo

t 1st Pair Cameras

(0,H+Offset,0)

2nd Pair Cameras

(-½W, H+Offset,½D)

(½W, 0, D ) Robot Envelope Depth

Object

(-½W, 0, D )

Ob

jec

t

He

igh

t

2nd pair parallel cameras

~100mm apart; orthogonal to 1st

Center Edge (0,0,0)

Robot Work Envelope

1st pair cameras ~100mm

apart; pointed at center

Documents

How to Achieve Accurate 3D Robotic Vision · 7/21/2011 · Types of 3D Vision Camera System Vision Methodology Required Comments Single Stationary 2D, 2½D, 3D 2½: ONLY if height