Empathic Tele-Existence

EMPATHIC TELE-EXISTENCE: COLLABORATION ACROSS TIME AND SPACE

Mark BillinghurstUniversity of South Australia

January 16th 2018

Boston

Travelled 10,855 miles in 34.5 hours for a 20 minute talk …

Adelaide - 35o C/95o F

Boston - 1o C/35o F

“Communication is not only the essence of being human, but a vital property of life.”

- John A. Piece

A wide variety of communication cues used.

SpeechParalinguisticPara-verbalsProsodicsIntonation

AudioGazeGestureFace ExpressionBody Position

Visual

Object ManipulationWriting/DrawingSpatial RelationshipObject Presence

Environmental

Face to Face Communication

Teleconferencing Today

Limitations with Current Technology

• Lack of spatial cues• Person blends with background

• Poor communication cues• Limited gaze, gesture, non-verbal communication

• Introduction of artificial seams between physical/digital• Separation of task/communication space

Augmented Reality Teleconferencing

• Natural spatial cues• Blend physical and virtual• Rich communication cues

To Make This Science Fiction Fact..• Display

• Unobtrusive AR display• Tracking

• Robust AR tracking• Capture

• Real time volumetric capture of people• Networking

• High bandwidth, wireless networking

Nearly 20 Years of Research ...

1999Live video texture

2003Volumetric Video

2008Live Broadcast

2016Live Volumetric Capture

2016

https://www.youtube.com/watch?v=7d59O6cfaM0

Are We There Yet?• Display

• Vuzix, Lumus• Tracking

• SLAM, ARKit, ARCore• Capture

• Microsoft, Mimesys, DoubleMe• Networking

• Gb/s, 5G

Vuzix Blade

Mimesys

Going Beyond Being There • Better than Face to Face conferencing

• Changing body scale • Moving inside another’s body• Interacting with virtual content• Capture/replay of conferencing• Making multiple copies of yourself• Sharing enhanced communication cues

Mixed Space Collaboration (2017)

• Make 3D copy of real space• AR user in real world, VR user in 3D copy of real space• Share virtual body cues (head, hands, gaze information)

Real World Virtual World

Virtual Communication Cues

• AR/VR displays• Gesture input (Leap Motion)• Room scale tracking

Multi-Scale Collaboration

• Changing the user’s virtual body scale

From Tele-Presence to Tele-Existence• Tele-Presence

• Enabling you to feel that remote people are part of your space

• Tele-Existence• Enabling you to feel like you are in a remote place

Handheld Tele-Existence Examples

Head Worn Camera Views

• Camera + Processing + Display + Connectivity• Ego-Vision Collaboration (But with Fixed View)

Shared Sphere – 360 Video SharingShared

Live 360 Video

Host User Guest User

Theta S360 Camera

Hi-res Camera

Epson BT-200See-through HMD

Oculus RiftHMD

Leap Motion

Empathic Computing

Systems that allow us to share what we are seeing, hearingand feeling with others..

Empathy Glasses

• Combine together eye-tracking, display, face expression• Implicit cues – eye gaze, face expression

++

Pupil Labs Epson BT-200 AffectiveWear

AffectiveWear – Emotion Glasses

• Photo sensors to recognize expression• User calibration• Machine learning• Recognizing 8 face expressions

Empathy Glasses in Use

• Eye gaze pointer and remote pointing• Face expression display

https://www.youtube.com/watch?v=CdgWVDbMwp4

Technology Trends• Advanced displays

• Wide FOV, high resolution

• Real time space capture• 3D scanning, stitching, segmentation

• Natural gesture interaction• Hand tracking, pose recognition

• Robust eye-tracking• Gaze points, focus depth

• Emotion sensing/sharing• Physiological sensing, emotion mapping

Emotion Sensing HMDs are Coming• Looxid

• EEG/Emotion sensing• Eye tracking

• Emteq• EMG sensing

• Neurable• EEG/BCI for VR

• PhysioHMD (MIT Media Lab)• GSR, PPG, Emotion Sensing

Technology Trends

• Advanced displays• Real time space capture• Natural gesture interaction• Robust eye-tracking• Emotion sensing/sharing

Empathic Tele-Existence

Empathic Tele-Existence

• Move from Observer to Participant• Explicit to Implicit communication• Experiential collaboration – doing together

Future Research• Many directions for future research

• User evaluation• Communication models • Novel interactions• Scaling up• Emotion capture/sharing• Space/body capture• Possible applications

Conclusions• AR Reality Conferencing

• New types of seamless remote collaboration• From Tele-Presence to Tele-Existence

• Trend toward Empathic Computing• Sharing what you see, hear and feel

• Many directions for future research• Capture/sharing emotion, interface design, etc..

www.empathiccomputing.org

@marknb00

mark.billinghurst@unisa.edu.au