A NOVEL INTERFACE FOR HUMAN- ROBOT

INTERACTION IN INDUSTRIAL ENVIRONMENT

• Presenter:

Dinh Quang Huy, IGS

• Supervisor: Assoc. Prof. Seet Gim Lee,Gerald, MAE

• Co-supervisors: Prof. Nadia Thalmann, IMI-IGS

• Mentor: Assoc. Prof. Lin Weisi, SCE

2/16/2017 1

OUTLINE

• MOTIVATION

• LITERATURE REVIEW

• PROPOSED SOLUTION

• DEMO

• CONCLUSION

2/16/2017 2

MOTIVATION Robot Interface in Industry

2/16/2017 3

Figure 1. (a) Teach Pendant Robot Controller. (b) Robot Control with Computer

MOTIVATION

User-interface Problems

2/16/2017 4

Figure 2. Operator wearing protective equipments

The lack of an interface for Human-Robot interaction that can

enhance productivity and adapt to different industrial working

requirements.

LITERATURE REVIEW

2/16/2017 5

• Current Forms of Human-Robot

Interaction:

1. Anthropomorphism and direct interaction

• Hard and sometimes misleading

2. Indirect Interaction

Robot

User

Interaction

1. Floor-based

2. Hand-held device

3. Gesture

4. Cursor

Mediating Object

GUIs on Projection

Surface

LITERATURE REVIEW

2/16/2017 6

2009[1] 2011[2]

2014[3] 2015[4]

Figure 3. Some current human-mobile robot interface [1][2][3][4]

LITERATURE REVIEW

2/16/2017 7

2009[1] 2011[2]

2014[3] 2015[4]

• Two main limitations:

1. Normal Projector is very blurry for projecting information

2. Insecure with only one channel for communication

RESEARCH GAP AND CONTRIBUTIONS

Research Gap

• The lack of a formal framework for Human-Robot interaction has the following features:

Compensate for the industrial requirements

Project high-definition graphics in both indoor and outdoor environments

Solve the security problem.

Contributions

• Develop a completed framework for human-machine interaction for industry which includes

• A multimodal single-handed device that is user – friendly, natural and flexible for

industrial contexts.

• A combination of see-through and spatial augmented reality systems that is capable of

faciliating the interaction between human and robot.

2/16/2017 8

PROPOSED SOLUTION

2/16/2017 9

Laser Writer

USB Camera

Human Interface Device

See-through AR Glass

Figure 4. Implementation of the proposed Human-Robot Interface

PROPOSED SOLUTION

1. Mobile Robot with Laser Writer

2/16/2017 10

Figure 5. MAVEN Robot with Lase Projection System

Solve blurry problem for SAR system

Laser Writer

USB Camera

PROPOSED SOLUTION

1. Mobile Robot with Laser Writer

2/16/2017 11

Figure 6. Robot is projecting its upcoming actions: stop, move foward, rotate to the left

or to the right

PROPOSED SOLUTION

2. A Combination of See-through and Spatial

Augmented Reality

2/16/2017 12

Enhance the security of the interface

See-through AR

Spatial AR

PROPOSED SOLUTION

2. A Combination of See-through AR and SAR

2/16/2017 13

Figure 7. Robot receives inputs from the main user via see-through AR while

projecting information on the floor using SAR

PROPOSED SOLUTION

3. The Introduction of a Multimodal Handheld

Device.

2/16/2017 14

Suitable for the industrial requirements

Display for Laser Rangefinder

Spring-loaded Linear

Potentiometers for finger

displacement inputs

IMU for hand motion inputs

Near-infrared Laser

Rangefinder

Laser pointer

why multimodal interaction ?

why handheld device ?

Figure 9. Implementation of a novel wearable device

DEMO

2/16/2017 15

CONCLUSION

Academic Aspects

Design and Implement a framework that can overcome the current

limitations of previous interface for a sucessful human – machine

interaction in industrial contexts

Future work

Usability Testing

Applying the interface for other industrial tasks.

2/16/2017 16

2/16/2017 17

Reference [1] A roadmap for US robotics: From Internet to Robotics,2013 edition

[2] World Robotics 2013 – Industrial Robot, IFR 18, September, 2013

[3] Image taken from http://www.autoalliance.org/images/dmImage/SourceImage/AdvancedTech-Cobots.png

[4] World Robotics 2004. UNCE, IFR. United Nations, Geneva, 2004

[5] R. D. Schraft, Christian Meyer, ‘The need for an intuitive method for small and medium enterprises’, ISR – Robotics, 2006

[6] Pedro Neto, J.Norberto Pires, A. Paulo Moreira, ‘CAD-Based off-line robot programming’, 2010 IEEE Conference on

Robotics Automation and Mechatronics (RAM)

[7] Pedro Neto, ‘Off-line Programming and Simulation from CAD Drawings: Robot-Assisted Sheet Metal Bending’, Industrial

Electronics Society, IECON 2013 - 39th Annual Conference of the IEEE

[8] c. Sehoul, J.S. Damgaard, S. B0gh, O. Madsen, ‘Human-Robot Interface for Instructing Industrial Tasls using Kinesthetic

Teaching’, 44th International Symposium on Robotics (ISR), 2013

[9] Petar Kormushev, Dragomir N. Nenchev, Sylvain Calinon and Darwin G. Caldwell, ‘Upper-body Kinesthetic Teaching of a

Free-standing Humanoid Robot’, 2011 IEEE International Conference on Robotics and Automation (ICRA)

[10] Azin Aryania, Balazs Daniel, Trygve Thomessen and Gabor Sziebig, ‘New Trends in Industrial Robot Controller User

Interface’, 3rd IEEE International Conference on Cognitive Infocommunications, 2012

[11] Balazs Daniel, Peter Korondi, Trygve Thomessen, ‘New Approach for Industrial Robot Controller User Interface’,

Industrial Electronics Society, IECON 2013 - 39th Annual Conference of the IEEE

[12] Image taken from http://www.windowscentral.com/nokia-maps-get-augmented-reality-functionality

[13] Image taken from http://www.gizmag.com/ikea-augmented-reality-catalog-app/28703/

[14] Gunther Reinhart, Wolfgang Vogl, Ingo Kresse, ‘A Projection-based User Interface for Industrial Robots’, IEEE Symposium

on Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2007. VECIMS 2007.

[15] T. Brick and M. Scheutz. Incremental natural language processing for HRI. Proceeding of the ACM/IEEE international

conference on Human-robot interaction – HRI ’07, page 263, 2007.

2/16/2017 18

Multimodal Hand-controller

• What is multimodal interface?

Multimodal interfaces allow users to interact with computers using

multiple different modes or channels of communication (pointer, haptic

buttons, gesture recognition)

• Why multimodal interaction?

Communication between humans is a multimodal process [15].

Multi modality is believed to produce more reliable semantic meanings

out of error-prone input models and remove vague information [16].

• Why handcontroller?

Hand gesture communication channel is natural for human.

The operator can perform another task while wearing the device.

2/16/2017 19