Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
MULTI MODAL INTERFACE WITH AUGMENTED
REALITY FOR INDUSTRIAL APPLICATIONS
• Presenter:
Dinh Quang Huy, IGS
• Supervisor: Assoc. Prof. Seet Gim Lee,Gerald, MAE
• Co-supervisors: Prof. Nadia Thalmann, IMI-IGS
• Mentor: Assoc. Prof. Lin Weisi, SCE
4/21/2015 1
OUTLINE
• MOTIVATION
• LITERATURE REVIEW
• PROPOSED SOLUTION
• PRELIMINARY RESULTS
• FUTURE PLAN
4/21/2015 2
MOTIVATION
Table 1. Annual growth rates of U.S.
companies for the period 2004-2011[1]
4/21/2015 3
• Robots are now being used in many fields such as Manufacturing, Medical
Treatment, Healthcare, Service Applications, Space or Defencse.
Figure 1. Worldwide annual supply of
industrial robots 2003 – 2016 [2]
PROBLEM STATEMENT
Robots are not popular in small medium enterprises (SMEs)
There are two main reasons
4/21/2015 4
High -
Investment
User-Interface
Problems
Figure 2. Example of small lot size processes: welding(left), cutting(right)
PROBLEM STATEMENT
High - investment
o The high – investment is rapidly vanishing (25% from 1990 to 2006 [5])
Industrial user – interface limitations
4/21/2015 5
o User-unfriendly and unnatural
Figure 4. An operator wearing
protective equipments Figure 3. Smart robot Teach Pendant
PROBLEM STATEMENT
4/21/2015 6
o Programming methods are time-consuming and require a lot user experience
which is difficult to apply to high mix, low volume production
OperatorGenerated Program
Specify points and orientations
Add process-related events
Robot Path
Figure 5. Traditional approach to program a robot o Cannot generate and return any type of visual feedback from the robot to the user
Figure 6. Human – robot cooperation
Task simulation
Task monitoring
Trust and accountability are important
considerations for cooperation[17]
Shift from robot – centric approach to
human - centric approach
MOTIVATION
4/21/2015 7
Adaptable and Reconfigurable interface
Perception for Unstructured Environments
Human must be able to recognize and interpret robot’s understanding
Image taken from A roadmap for US robotics: From Internet to Robotics,2013 edition
PROBLEM SUMMARY
• What prevent robots from becoming popular in
industrial contexts?
High - investment
User – interface limitations
User-unfriendly and unnatural
Programming methods are time-consuming and require a lot user
experience
Cannot generate and return any type of visual feedback from the robot
to the user
4/21/2015 8
LITERATURE REVIEW
o Human – robot interaction using Computer – Aided Design tool [6][7]
• The robot is controlled and programmed using a computer with a 3D model simulation
software
4/21/2015 9
Figure 7. CAD tool
o Intuitive with simulation
capability
o Require user - experience
o CAD calibration error
o Costly
Pros
Cons
LITERATURE REVIEW
Human – robot interface using kinesthetic teaching [8][9]
• Human user pilots the robot through the intended trajectory
4/21/2015 10
Figure 8. Operator piloting the manipulator t a
coordinate[8]
o Simple and more natural
o Time – consuming
o No visual feedback
Pros
Cons
RESEARCH GAP AND CONTRIBUTIONS
Research Gap
• The lack of a formal framework for Human-Robot interaction has the following features:
Enhance productivity and adapt to different industrial working scenarios.
Programming needs to be more intuitive, secure with a higher level of abstraction.
The human needs to “say less and the robot do more”.
Contributions
• Develop a completed framework for human-machine interaction for industry which includes
• A multimodal single-handed device that is user – friendly, natural and flexible for
industrial contexts.
• An AR system is capable of increasing the understanding between human and robot using
visual feedback.
• Refine and evaluate the proposed framework to identify essential interface’s factors that are
critical for industrial needs.
4/21/2015 11
Crane Tasks
Ship Hull Tasks
Proposed HRI in Mixed Reality with Multi
Modal Interface
12
Human User Robot
Mapping Hand-
Motion Gestures &
Actions to Robot
Actions
Computer
Generated Objects
Superimposed
Over Real World
Industrial Environment
Computer Graphics
Real World Image
Finger Action & Hand Gesture
Laser Mark
Visual Feedback
with Task
Simulation
4/21/2015
Human & Robot Interactions In Mixed Reality
13Human User Industrial Environment
Task SpaceComputer Graphics
Real World Image
Finger Action & Hand Gesture
Laser Mark
Visual Feedback with Task Simulation Computer Generated Objects,
Superimposed Over Real World
Mapping Hand-Motion Gestures & Actions to Robot Actions
Interaction
Human request Robot’s attention,
Robot confirms availability,
Human requests Robot to perform a task
Robot elaborates on Task requirements
and requests Human approval
Human amends Task details or Confirms
Robot executes Task
Human Monitors Robot Action
• A Robot Partner must recognize the Human’s intention and to add implicit constraints defined by the human.
• The human defines an incomplete Task, and the Robot elaborates on the Task.
• The High Level Robotic Assistance will enhance productivity at the risk of an appropriate proposal.
• Human confirmation/selection of Task options, is required of the Human.
4/21/2015
Proposed HRI Block Diagram
14
Computer
Graphics
Real World
Image
Finger Action &
Hand Gesture
Laser Mark
Human User
Task Space
IndustrialEnvironment
Graphics & Text
Generator
Finger Action &
Hand Gesture
Interpreter
Robot Motion
Command Generator
Robot/Environment
Status Parser
Knowledge
Decision Making
Task Constraints
Task SupportUser Interface
Viewer Head
Motion
Compensation
Mode?Direct
Assisted
4/21/2015
Proposed Secure Multimodal User Interface
15
Vision
Hand Action
Gesture & Command Recognition
10-DOF IMU Data
Analog Paddle & Buttons Status
Laser Pointer Track Recognition
Task Recognition
Task Simulation
Task ExecutionInteractions & ProcessesUser Command & Feedback
Task Space
Industrial
Environment
Transparent Display Glasses with Front
Facing Camera
Naked eye view
Augmented Image
Multi-modal paddle hand motion gesture and laser pointing
Paddle Action
Hand Motion
Laser Pointing
Hand Motion
4/21/2015
Multimodal Hand-controller
• What is multimodal interface?
Multimodal interfaces allow users to interact with computers using
multiple different modes or channels of communication (pointer, haptic
buttons, gesture recognition)
• Why multimodal interaction?
Communication between humans is a multimodal process [15].
Multi modality is believed to produce more reliable semantic meanings
out of error-prone input models and remove vague information [16].
• Why handcontroller?
Hand gesture communication channel is natural for human.
The operator can perform another task while wearing the device.
4/21/2015 16
Hand-controller input models Laser pointer
Laser pointer is useful when it comes to defining spatial
information such as robot task trajectory or object selection.
Haptic buttons
Provide tangible feeling while wearing gloves
Simple , high accuracy
4/21/2015 17
Figure 15. A clickable word
Hand-controller input models
Motion gesture mapping and recognition using inertial
measurement unit (IMU).
Vision-based gesture recognition often requires constant camera human
spatial information and consistent lightning [23].
If robots have too little autonomy, human operators will waste time
attending to robot instead of attending to their work tasks. If robot are
highly autonomous, situation awareness of activity is diminished[17].
The controlling mode could be a combination between these input
models
For example, haptic buttons can be useful to move the robot forward or
backward while gesture mapping is more convenient to rotate the robot.
4/21/2015 18
Augmented Reality System
Head Mounted Display (HMD)
Can perform functions similar to the teach pendant and be able to align
digital information to real information.
Task creation and simulation
Feedback Visualization
Input model mapping
4/21/2015 19
Figure 14. Task simulation example[18] Figure 15. Task creation example[19]
PREMINARY RESULTS
Hand-controller
4/21/2015 20
Figure 21. Hand-controller schematic
PREMINARY RESULTS
Hand-controller
Can control the simulated
robot using
• Haptic buttons
• Gesture
4/21/2015 21
Figure 22. Hand-controller prototype
FUTURE PLAN
Identify and design a AR module with importants components, and factors
that are critical for a sucessful human – machine interface for industrial
purposes
Interface Evaluation
• Use ten design heuristics for evaluation by Jacob Nielsen[22]
• Compare the performance of experienced users with novice users
• Calculate the average time needed to complete a task using the
interface for a set of users
4/21/2015 22
4/21/2015 23
Reference[1] A roadmap for US robotics: From Internet to Robotics,2013 edition
[2] World Robotics 2013 – Industrial Robot, IFR 18, September, 2013
[3] Image taken from http://www.autoalliance.org/images/dmImage/SourceImage/AdvancedTech-Cobots.png
[4] World Robotics 2004. UNCE, IFR. United Nations, Geneva, 2004
[5] R. D. Schraft, Christian Meyer, ‘The need for an intuitive method for small and medium enterprises’, ISR – Robotics, 2006
[6] Pedro Neto, J.Norberto Pires, A. Paulo Moreira, ‘CAD-Based off-line robot programming’, 2010 IEEE Conference on
Robotics Automation and Mechatronics (RAM)
[7] Pedro Neto, ‘Off-line Programming and Simulation from CAD Drawings: Robot-Assisted Sheet Metal Bending’, Industrial
Electronics Society, IECON 2013 - 39th Annual Conference of the IEEE
[8] c. Sehoul, J.S. Damgaard, S. B0gh, O. Madsen, ‘Human-Robot Interface for Instructing Industrial Tasls using Kinesthetic
Teaching’, 44th International Symposium on Robotics (ISR), 2013
[9] Petar Kormushev, Dragomir N. Nenchev, Sylvain Calinon and Darwin G. Caldwell, ‘Upper-body Kinesthetic Teaching of a
Free-standing Humanoid Robot’, 2011 IEEE International Conference on Robotics and Automation (ICRA)
[10] Azin Aryania, Balazs Daniel, Trygve Thomessen and Gabor Sziebig, ‘New Trends in Industrial Robot Controller User
Interface’, 3rd IEEE International Conference on Cognitive Infocommunications, 2012
[11] Balazs Daniel, Peter Korondi, Trygve Thomessen, ‘New Approach for Industrial Robot Controller User Interface’,
Industrial Electronics Society, IECON 2013 - 39th Annual Conference of the IEEE
[12] Image taken from http://www.windowscentral.com/nokia-maps-get-augmented-reality-functionality
[13] Image taken from http://www.gizmag.com/ikea-augmented-reality-catalog-app/28703/
[14] Gunther Reinhart, Wolfgang Vogl, Ingo Kresse, ‘A Projection-based User Interface for Industrial Robots’, IEEE Symposium
on Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2007. VECIMS 2007.
[15] T. Brick and M. Scheutz. Incremental natural language processing for HRI. Proceeding of the ACM/IEEE international
conference on Human-robot interaction – HRI ’07, page 263, 2007.
4/21/2015 24
Reference[16] S. L. Oviatt. Ten myths of multimodal interaction. In CACM, volume42(11):pages74–81, 1999.
[17] Clint Heyer, ‘Human-robot Interaction and Future Industrial Robotics Applications’, 2010 IEEE/RSJ International
Conference on Intelligent and Systems, Taipei, Taiwan
[18] Jens Lambrecht, Martin Kleinsorge, Martin Rosenstrauch and Jörg Krüger, ‘Spatial Programming for Industrial Robots
Through Task Demonstration’’, International Journal of Advanced Robotic Systems, 2013, Vol 10
[19] Jens Lambrecht and Jörg Krüger, ‘Spatial Programming for Industrial Robots based on Gestures and Augmented Reality’
2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,October 7-12, 2012. Vilamoura, Algarve,
Portugal
[20] G. Falcao, N. Hurtos, J. Massich, D. Fofi, ‘Projector-Camera Calibration Toolbox Report’,
http://code.google.com/p/procamcalib, 2009
[21] Samuel Audet and Masatoshi Okutomi, “A User-Friendly Method to Geometrically Calibrate Projector-Camera Systems”,
IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2009. CVPR Workshops 2009,
Page(s):47 – 54
[22] Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proc. ACM CHI'90 Conf. (Seattle, WA, 1-5
April), 249-256.
[23] Michael T. Wolf, Christopher Assad, Matthew T. Vernacchia, Joshua Fromm, and Henna L. Jethani, “Gesture-Based
Robot Control with Variable Autonomy from the JPL BioSleeve”, 2013 IEEE International Conference on Robotics and
Automation (ICRA) Karlsruhe, Germany, May 6-10, 2013.
4/21/2015 25