MIN FacultyDepartment of Informatics

Communicating Robot Motion Intent withAugmented Reality

Yiyao Wei

University of HamburgFaculty of Mathematics, Informatics and Natural SciencesDepartment of InformaticsTechnical Aspects of Multimodal Systems

18 November 2019

– Communicating Robot Motion Intent with Augmented Reality 1 / 19

OutlineMotivation A Brief Introduction Design AR Interface for Flying Robot References

1. Motivation2. A Brief Introduction

Current ApproachesVR vs AR vs MR

3. Design AR Interface for Flying RobotExperimentResultsLimitations and Future Work

4. References

– Communicating Robot Motion Intent with Augmented Reality 2 / 19

MotivationMotivation A Brief Introduction Design AR Interface for Flying Robot References

The Future is approaching...I Evolution of manufacturing industry, logistics, and constructionI And more...

Source: https://www.youtube.com/watch?v=1xfbwaAhgwhttps://www.press.bmwgroup.com/global/photo/detail/P90242725/smart-transport-robot-carrying-roller-containers-through-the-logistics-hall-at-bmw-group-plant

https://idsc.ethz.ch/research-dandrea/research-projects/archive/flying-machine-enabled-construction.html– Communicating Robot Motion Intent with Augmented Reality 3 / 19

Motivation (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

The future of work is very likely transformed by collaborativerobots.I Collaborative activities fundamentally depend on

interpredictability — the ability of team memebers to rapidlyunderstand and predict the attitudes and actions of the others[1].

I This is a challenging taskI Studies report that social cues is a way for human better

understand their movement intent and affective state [2, 3, 4].I Research shows that motion intent cueing can improve

interaction fludity and efficiency in collaboration betweenhumans and robots [7].

– Communicating Robot Motion Intent with Augmented Reality 4 / 19

Current ApproachesMotivation A Brief Introduction Design AR Interface for Flying Robot References

Most Common ApproachesI One approach is to implement advanced warning systems.I Another approach is to add anthropomorphic and zoomorphic

features.I Facial expressionsI Human-like gesturesI Natural languages

Source: https://www.newyorker.com/magazine/2017/10/23/welcoming-our-new-robot-overlords– Communicating Robot Motion Intent with Augmented Reality 5 / 19

Some downside of current methodsMotivation A Brief Introduction Design AR Interface for Flying Robot References

I Poor communicationI Hard to apply these findings to industrial robots arms or flying

robotsI Longer develop perid

– Communicating Robot Motion Intent with Augmented Reality 6 / 19

Comparison of VR, AR, and MRMotivation A Brief Introduction Design AR Interface for Flying Robot References

Source:https://rubygarage.org/blog/difference-between-ar-vr-mr

– Communicating Robot Motion Intent with Augmented Reality 7 / 19

Design AR Interface for Flying RobotMotivation A Brief Introduction Design AR Interface for Flying Robot References

Why Aerial Robots?I Drone robots have been developing rapidly in recent years.I They are ideal co-workers for logistics management in

manufacturing settings.I The ability of flying enable unique forms of assistance in a

variety of collaborative tasks [6].

– Communicating Robot Motion Intent with Augmented Reality 8 / 19

Experiment (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

I NavPoints: series of lines point to the target location withinformation about arrival times, wait times, and velocities ofeach line fragment

I Arrow: a big arrow represents the exact path that the robotplans to do

I Gaze: a flying eye as a representation of the aerial robot, withthe front information

I Utilities: a radar-like interface, indicating the location of therobot relative to the user

– Communicating Robot Motion Intent with Augmented Reality 9 / 19

Experiment (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

TechologiesI Robotic Platform: AscTec Hummingbird robotI ARHMD Platform: Microsoft HoloLensI Develop Platform: Unity

– Communicating Robot Motion Intent with Augmented Reality 10 / 19

ExperimentMotivation A Brief Introduction Design AR Interface for Flying Robot References

The TaskI Participants were tasked with assembling beaded strings at

workstations shared with an aerial robot.I The goal is to make as many beaded strings as possible in

exactly 8 minutes.I If the robot traveled to the same station as the participant,

they had to leave the workstation at least 2m away and wait forthe robot to leave

I post-questionnaireI Overall 30 mins long

– Communicating Robot Motion Intent with Augmented Reality 11 / 19

Design AR Interface for Flying Robot (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

Experiment Design5 x 1 between-subjectsI Five conditions: 4 models and one baseline conditionI Independent variable: type of AR feedbackI Dependent variables: task performance, work efficiency, intent

clarity, and robot usabilityBaseline condition:I no AR feedbackI Participants need to wear HoloLens and perform the same task

as in other conditions.

– Communicating Robot Motion Intent with Augmented Reality 12 / 19

Experiment (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

ParticipantsI 60 participantsI 40 males, 20 females, evenly distribute into 5 conditions

– Communicating Robot Motion Intent with Augmented Reality 13 / 19

ResultsMotivation A Brief Introduction Design AR Interface for Flying Robot References

Compared to the Baseline condition:I NavPoints, Arrow, and Gaze models show significant

efficiencies improvement.I NavPoints model is the best model for movement intent clarity.

– Communicating Robot Motion Intent with Augmented Reality 14 / 19

Limitations and Future WorkMotivation A Brief Introduction Design AR Interface for Flying Robot References

LimitationsI The design of information transmission is not straightforward

(Robot location → motion tracking system → AR feedback)I Participants report HoloLens is uncomfortableI The generalizability of the results may be limited due to the

experimental design, task, and measures

– Communicating Robot Motion Intent with Augmented Reality 15 / 19

Limitations and Future Work (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

Future WorkI Large sample sizeI Within-subjects designI Better information transmissionI Better measures for dependent variablesI Consider more aspects, e.g. gender effects

– Communicating Robot Motion Intent with Augmented Reality 16 / 19

Thank You for listeningAny question?

– Communicating Robot Motion Intent with Augmented Reality 17 / 19

ReferencesMotivation A Brief Introduction Design AR Interface for Flying Robot References

1. Gary Klein, Paul J Feltovich, Jeffrey M Bradshaw, and David DWoods. 2005. Common Ground and Coordination in Joint Activity.Organizational Simulation 53 (2005), 139–184.

2. hrystopher L Nehaniv, Kerstin Dautenhahn, Jens Kubacki, MartinHaegele, Christopher Parlitz, and Rachid Alami. 2005. AMethodological Approach Relat- ing the Classification of Gesture toIdentification of Human Intent in the Context of Human-RobotInteraction. In IEEE International Symposium on Robot and HumanInteractive Communication (RO-MAN’05). 371–377.

3. Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. 2003. ASurvey of Socially Interactive Robots. Robotics and AutonomousSystems 42, 3–4 (2003).

4. Jyotirmay Sanghvi, Ginevra Castellano, Iolanda Leite, André Pereira,Peter W McOwan, and Ana Paiva. 2011. Automatic Analysis ofAffective Postures and Body Motion to Detect Engagement with aGame Companion. In ACM/IEEE International Conference onHuman-Robot Interaction (HRI’11). 305–311.

– Communicating Robot Motion Intent with Augmented Reality 18 / 19

References (Cont.)Motivation A Brief Introduction Design AR Interface for Flying Robot References

5. Michael Walker, Hooman Hedayati, Jennifer Lee, and Daniel Szafir.2018. Communicating Robot Motion Intent with Augmented Reality.In HRI ’18: 2018 ACM/IEEE International Conference onHuman-Robot Interaction, March 5–8, 2018, Chicago, IL, USA.ACM, New York, NY, USA, 9 pages.https://doi.org/10.1145/3171221.3171253

6. Daniel Szafir, Bilge Mutlu, and Terrence Fong. 2015.Communicating Direction- ality in Flying Robots. In Proceedings ofthe ACM/IEEE International Conference on Human-RobotInteraction (HRI’15). ACM, 19–26.

7. Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. 2013.Legibility and Predictability of Robot Motion. In Human-RobotInteraction (HRI), 2013 8th ACM/IEEE International Conference on.301–308.

– Communicating Robot Motion Intent with Augmented Reality 19 / 19