Force and Visual Control for Safe Human−Robot Interaction...UPMC • Institut des Systèmes Intelligents et de Robotique Paris • 24 September 2015 Force Control Force and Visual

UPMC • Institut des Systèmes Intelligents et de Robotique Paris • 24 September 2015

Force and Visual Control forSafe Human−Robot Interaction

Bruno SICILIANO

www.prisma.unina.it


PRISMA Team

Bruno Siciliano • Luigi Villani • Vincenzo LippielloAlberto Finzi • Silvia Rossi • Franco Cutugno

Fanny Ficuciello • Fabio Ruggiero • Agostino De SantisRafik Mebarki • Antoine Petit • Jun Nishiyama • Alejandro Donaire • Aykut Satıcı

Francesca Cordella • Mariacarla Staffa • Daniela D'AuriaLuca Buonocore • Jonathan Cacace • Diana Serra • Mahdi Momeni • Andrea Fontanelli

Valeria Federico • Jonathan van der Meer

Force and Visual Control for Safe Human−Robot Interaction 2/35


Our Research Portfolio

Aerial RoboticsAssistive RoboticsCognitive Control of Robotic SystemsDual-Arm/Hand ManipulationForce ControlHuman−Robot InteractionInspection RoboticsLightweight Flexible ArmsMobile Dynamic Non-prehensile ManipulationMotion Generation and Biomechanical Analysis

for Human FiguresRedundant ManipulatorsRobotic Surgery TechnologySensory-Motor Synergies of Anthropomorphic

HandsService RoboticsVisual Servoing

Fund

ing

of 8.

5 M€



Physical human-robot interaction Force control

Impedance control Force control

Visual control Position-based visual servoing Pose estimation using vision

Interaction control using vision and force Position-based visual impedance control Pose estimation using vision and force

Experiments

OutlineForce and Visual Control for Safe Human−Robot Interaction 4/35


Human−Robot InteractionForce and Visual Control for Safe Human−Robot Interaction 5/35

Traditional industrial robotics Segmented workspace for machines and humans

Intelligent machines working in contact with humans Haptic interfaces and teleoperators Cooperative material handling Power extenders Rehabilitation and physical training Entertainment

Human–robot interaction (HRI) Cognitive issues [cHRI]: perception, awareness, mental models Physical issues [pHRI]: safety, dependability Ethical issues: motivations and critics about HRI, acceptability


Physical Human−Robot InteractionForce and Visual Control for Safe Human−Robot Interaction 6/35

Risks for robots interacting with humans Heavy moving parts and objects transported Sensory data reliability Level of autonomy/unpredictable behaviours

Solutions for collaborative human–robot operation Design of non-conventional actuators (passive safety) Interaction control (active safety) Dependable algorithms for supervision and planning Fault tolerance Need for quantitative metrics


Force ControlForce and Visual Control for Safe Human−Robot Interaction 7/35

Motion control vs. force control Advanced industrial robotics and service robotics demand for control of interaction

between robot manipulator and humans Use of purely motion control strategy is candidate to fail (task planning accuracy) Control of contact force (compliant behaviour) Use of force/torque sensor (interfaced with robot control unit)


TaxonomyForce and Visual Control for Safe Human−Robot Interaction 8/35

Indirect vs. direct force control Indirect force control: force control via motion control (w/out explicit closure of force

feedback loop) Direct force control: force controlled to desired value (w/ closure of force feedback

loop)


Indirect Force ControlForce and Visual Control for Safe Human−Robot Interaction 9/35

Impedance control with inner motion control loop Force/torque measurements for linear and decoupled impedance Compliant frame between desired and EE frame (disturbance rejection)


ExperimentsForce and Visual Control for Safe Human−Robot Interaction 10/35

Set-up COMAU Smart 3-S robot Open control architecture ATI force/torque sensor 6-DOF spatial impedance

surface contact

low compliancehigh damping

high compliancelow damping

high compliancehigh damping


Extension to Cooperating RobotsForce and Visual Control for Safe Human−Robot Interaction 11/35

Dual-arm set-up 6ax robot 7ax robot

peg-in-hole assembly6-DOF impedance

control of absolute motionand internal forces

absolute & relative impedance absolute impedance human-object interaction


Service TasksForce and Visual Control for Safe Human−Robot Interaction 12/35

Set-up @ ARTS Lab, SSSA Pisa DEXTER cable-actuated robot arm

Compliance control Tasks of assisting disabled people


Visual ControlForce and Visual Control for Safe Human−Robot Interaction 13/35

Artificial intelligence vs. automatic control Vision capabilities bring robots nearer to human skills The information that can be extracted form images is very rich and can be used at

different levels of a hierarchical architecture

Data complexity

Highest level(Artificial Intelligence)

Visual information used to understand the world and decide

how to act

Bandwidth

Lowest level(Automatic Control)

Visual measurements usedin the control law

Visual servoing


TaxonomyForce and Visual Control for Safe Human−Robot Interaction 14/35

Direct visual servoing

Indirect visual servoing

Vision-basedcontrol

Robot

CameraVisual feedback≥ 100 Hz

Vision-basedcontrol

Motion control Robot

CameraVisual feedback

Joint feedback

≥ 20 Hz

≥ 100 Hz

dynamic look-and-move


Indirect Visual ServoingForce and Visual Control for Safe Human−Robot Interaction 15/35

3D object

Vision system

Computer

BUS

Estimated pose

Control unitMOVE


Position-based Visual Servoing (PBVS)Force and Visual Control for Safe Human−Robot Interaction 16/35

Position-based (3D) visual servoing scheme

Calibration dependent

Features extraction

Posecontrol

CameraRobot + motion control

Pose estimation

Desiredpose


PBVSForce and Visual Control for Safe Human−Robot Interaction 17/35

Pose control and pose estimation

Calibrated cameras Target objects of known geometry


Pose EstimationForce and Visual Control for Safe Human−Robot Interaction 18/35

Reconstruction of object pose from observed workspace

Observedworkspace

Image regionsegmentation

Image featuresextraction

Image acquisition

Pose reconstruction

Selection of image regions


Dynamic Image FeaturesForce and Visual Control for Safe Human−Robot Interaction 19/35

Pose estimation with dynamic image features selection Extended Kalman Filter (EKF) based on approximate object models


EKFForce and Visual Control for Safe Human−Robot Interaction 20/35

Extended Kalman Filter State-space model of the object motion

Output equation

Jacobian of the output equation

= input disturbance

= object pose

= measurement noise

~ image Jacobian

= image features parameters


Pose ReconstructionForce and Visual Control for Safe Human−Robot Interaction 21/35

Dynamic mode (interposing parts) A single BSP tree is generated on-

line, describing all the objects with respect to a common reference frame

For each camera, the visit algorithmis applied to the BSP tree

A Kalman filter is used for each object


Preselection AlgorithmForce and Visual Control for Safe Human−Robot Interaction 22/35

The preselection algorithm estimates the position of the projections of the object visible corners on

the image planes of each camera

~

~

Out of sight


Selection AlgorithmForce and Visual Control for Safe Human−Robot Interaction 23/35

A selection algorithm based on suitable quality indexes is employed to find the (sub)optimal set of (6 to 8) feature points

Some quality indexes for the cost function Spatial distribution (⇒ avoiding points concentration) Angular distribution (⇒ avoiding alignments) Camera distribution (⇒ triangulation) Anti-chattering (⇒ estimate noise reduction) Extraction reliability (⇒ robustness)


Features ExtractionForce and Visual Control for Safe Human−Robot Interaction 24/35

extracted corners

Out of sight


Visual Servoing ArchitectureForce and Visual Control for Safe Human−Robot Interaction 25/35

Robot CPU Servo CPU Power amplifiers

BUS VME

User interface modules

COMAU – C3G 9000 openRobot COMAU

SMART−3SFOLLOWER

Robot COMAU SMART−3S

LEADER

Board BIT 3

BUS AT

Board BIT 3

ControlPC

Control – PC (RePLiCS)

VisionPC

MATROX GENESIS

MATROX GENESIS

Vision – PC (VESPRO)

BUS AT

RS232SONY XC 8500 CE

SONY XC 8500 CE

(PDL2)



FOLLOWER

LEADER

L-CameraR-Camera

Object


Extension to GraspingForce and Visual Control for Safe Human−Robot Interaction 27/35

Visually guided grasping Object in unstructured

environment Visual servoing Tracking of object motion Good reaction to uncertainties


Interaction ControlForce and Visual Control for Safe Human−Robot Interaction 28/35

Interaction control using vision and force Vision provides global information on surrounding environment to be used for motion

planning and obstacle avoidance Force allows adjusting robot motion so that local constraints imposed by environment

are satisfied


Force and Visual ControlForce and Visual Control for Safe Human−Robot Interaction 29/35

Set-up @ DLR KUKA robot with force sensor and

camera embedded in the gripper Integration of vision and force

Visual feedback in gross motion Force feedback in fine motion


Control StrategyForce and Visual Control for Safe Human−Robot Interaction 30/35

Problem Control interaction of a robot manipulator with a rigid object of

known geometry but unknown position and orientation

Solution When robot is far from object

Position-based visual servoing is adopted The relative pose of the robot with respect to the object is estimated recursively using only

vision When robot is in contact with object

Any kind of interaction control strategy can be adopted (impedance control, parallel force/position control)

The relative pose of the robot with respect to the object is estimated recursively using vision, force and joint position measurements


Visual ImpedanceForce and Visual Control for Safe Human−Robot Interaction 31/35

Position-based visual servoing

Force feedback

Impedance

impedance


Pose Estimation Using Vision and ForceForce and Visual Control for Safe Human−Robot Interaction 32/35

Modified EKF State-space model unchanged Modified output equation (when )

Modified Jacobian



Pose estimation errors

vision vision & force

X


ConclusionForce and Visual Control for Safe Human−Robot Interaction 34/35

Interaction control of robot manipulators Indirect and direct force control Position-based visual servoing Position-based visual impedance control

Current and future work Interaction control in unstructured environments (presence of humans) Integration of new sensors (proximity sensors, joint torque sensors, …) Extension to dynamic manipulation (nonprehensile, deformable objects)



Thank You very much indeed for Your kind attention

[email protected]

Documents

Force and Visual Control for Safe Human−Robot Interaction...UPMC • Institut des Systèmes Intelligents et de Robotique Paris • 24 September 2015 Force Control Force and Visual