Lecture 6: Collaborative AR Lecture 6: Collaborative AR Applicationspp

Mark BillinghurstgHIT Lab NZ

University of Canterbury University of Canterbury

Collaboration: What’s that ?

Definitionf n a joint work ; between several peoples who f.n. a joint work ; between several peoples who

generate the creation of a shared task

Collaborative activitiesCollaboration: Work, Leisure

CSCW (Computer Supported Collaborative Work) CSCW (Computer Supported Collaborative Work) Groupware

Collaboration: Taxonomy

Time/Space [Ellis91]

Time

post, email, faxbulletin boardasynchronous

pos , e a , abu e boa d

Shared editor

meeting, brainstorming,

seminar

Spacesynchronous

chat, teleconference

Spacecolocated distant

Today’s TechnologyVideo Conferencing

lack of spatial cueslimited participants2D collaboration

Collaborative Virtual Environmentsseparation from real worldreduced conversational cues

Beyond Video Conferencing

2D Interface onto 3DVRML

Projection ScreenCAVE, WorkBench

Volumetric Displayp yscanning laser

Virtual RealityVirtual Realitynatural spatial cues

Beyond Virtual RealityLessons from CSCW

SeamlessEnhance Reality

Immersive Virtual Realityseparates from real worldreduces conversational cues

Collaboration in the Future ?

Remote Conferencing

Face to face Conferencing

Central ClaimAugmented Reality techniques can be used to provide spatial cues that significantly enhance face-to-face and remote collaboration on three-dimensional tasks.

Construct3D [Kaufmann 2000]

Collaborative geometry education toolg yDifferent learning modes (teacher, student, exam)Tangible interaction personal interaction panelTangible interaction – personal interaction panel

Collaborative Augmented Reality

Seamless InteractionNatural Communication

Att ib tAttributes:VirtualityAugmentationCooperationIndependenceIndividuality

Seamless CSCWSeam (Ishii et. al.)

spatial, temporal, functional discontinuity

Types of Seams FunctionalFunctional

- between different functional workspaces

Cognitiveg- between different work practices

Functional Seams

Cognitive Seams

Effect of SeamsFunctional Seams:

Mediated differs from F-to-F Conversation- Loss of Gaze Information- Degradation of Non-Verbal Cues

Cognitive Seams:Learning Curve EffectsUser Frustration

Unanswered Questions

Does seamlessness enhance performance?Wh AR h ll b ?What AR cues can enhance collaboration ?How does AR collaboration differ ?What technology is required ?...

Collaborative AR Interfaces

Face to Face CollaborationWebSpace, Shared Space, Table Top Demo, Interface p , p , p ,

Comparison, AR Interface Comparison

Remote CollaborationSharedView, RTAS, Wearable Info Space, WearCom, AR

Conferencing, BlockParty

T l I fTransitional InterfacesMagicBook

Hybrid InterfacesAR PRISM, GI2VIS

Face to Face Collaboration

Communication CuesA wide variety of communication cues used.

Visual

SpeechParalinguistic

Audio GazeGestureFace Expression

sua

ParalinguisticParaverbalsProsodicsIntonation

Body Position

Object ManipulationWriting/DrawingSpatial RelationshipObject Presence

EnvironmentalObject Presence

In computer supported collaboration, however, it is often hard for users to exchange non-verbal communication gcues, even when they are co-located.

Face-to-face collaborationPeople surround a tableIt is easy to see each other

C i ti S T k S

Computer supported ll b ti

Communication Space Task Space

collaborationPeople sit side by sideIt i h d t h thIt is hard to see each other

Communication Space Task Space

Shared Space - Table Top DemoGoal

create compelling collaborative AR p ginterface usable by novices

Exhibit contentmatching card gameface to face collaborationface to face collaborationphysical interaction

Results2 500 3 000 2,500 - 3,000 usersObservations

bl i h h i fno problems with the interface- only needed basic instructions

physical objects easy to manipulatephysical objects easy to manipulatespontaneous collaboration

Subjective survey (157 people)Subjective survey (157 people)Users felt they could easily play with other people and interact with objectsj

Improvementsreduce lag, improve image quality, better HMDg, p g q y,

AR PadHandheld AR Display

LCD screenSpaceOrbCameraP h l Peripheral awareness

Support for Collaboration

Virtual Viewpoint Visualization

Face to Face CollaborationCompare two person collaboration in:

Face to Face, AR, Projection Display

TaskUrban design logic puzzleUrban design logic puzzle

- Arrange 9 building to satisfy 10 rules in 7 minutes

SubjectsSubjectsWithin subjects study (counter-balanced)12 pairs of college students12 pairs of college students

Face to Face Condition

Moving Model Buildings

AR Condition

Cards with AR ModelsCards with AR ModelsSVGA AR Display (800x600)Video see-through ARg

Projection Condition

Tracked Input Devices

Task Space Separation

Interface ConditionsFtF AR Projection

User Viewpoint Independent Private PublicpEasy to change Independent

Easy to changeCommonDifficult to change

Limited FOV

Interaction Two handedNatural object manipulation

Two handedTangible AR techniques

Mouse-basedOne-handedTime-multiplexedmanipulation

Space-multiplexedtechniquesSpace-multiplexed

Time-multiplexed

Hypothesis

Collaboration with AR technology will produce behaviors that are more like natural face-to-face collaboration than from using a screen-face collaboration than from using a screen

based interface.

MetricsSubjective

Evaluative survey after each conditionEvaluative survey after each conditionForced-choice survey after all conditionsPost experiment interviewPost experiment interview

ObjectivejCommunication measures

- Video transcriptionp

Measured ResultsPerformance

AR collaboration slower than FtF + Projectionj

CommunicationPointing/Picking gesture behaviors same in AR as FtFPointing/Picking gesture behaviors same in AR as FtFDeictic speech patterns same in AR as FtF

- Both significantly different than Projection conditiong y j

SubjectiveFtF easier to work together and understandFtF easier to work together and understandInteraction in AR easier than Proj. and same as FtF

Deictic Expressions

25%

30%

15%

20%

5%

10%

0%FtF Proj AR

Significant difference – ANOVA, F(2,33) = 5.77, P < 0.01No difference between FtF and AR

Ease of Interaction

S f d ffSignificant differencePick - F(2,69) = 37.8, P < 0.0001Move - F(2,69) = 28.4, P < 0.0001

Interview Comments“AR’s biggest limit was lack of peripheral vision. The interaction was natural, it was just difficult to see. In the projection condition you could see everything but the interaction was tough”Face to Face

Subjects focused on task space- gestures easy to see gaze difficult- gestures easy to see, gaze difficult

Projection displayInteraction difficult (8/14)

- not mouse-like, invasion of space

AR display – “working solo together”Lack of peripheral cues = “tunnel vision” (10/14 people)Lack of peripheral cues = tunnel vision (10/14 people)

Face to Face SummaryCollaboration is partly a Perceptual task

AR reduces perceptual cues -> Impacts collaborationTangible AR metaphor enhances ease of interaction

Users felt that AR collaboration different from FtFBut:

measured speech and gesture behaviors in AR condition is more similar to FtF condition than in Projection display

Thus we need to design AR interfaces that don’t reduce perceptual h l k f cues, while keeping ease of interaction

Collaborative AR

AR TennisVirtual tennis courtVirtual tennis courtTwo user gameAudio + haptic feedbackAudio + haptic feedbackBluetooth messaging

Research Questions

Does having an AR interface enhance the faceto face gaming experience?g g p

AR Experience

ConditionsA: Face to Face ARB: Face to Face non-ARC: Non Face to Face

MeasuresSurvey questionsRanking

Survey Questions1/ How easy was it to work with your partner?2/ How easily did your partner work with you?y y p y3/ How easy was it to be aware of what your partner was doing?4/ How enjoyable was the game?

Answered on a Likert scale from 1 to 71 = not very easy, 7 = very easy

Results

Rankings

User felt AR condition was much easier to see what their partner was doing (Q3)p g ( )Easier to work with partner in AR case (Q1)Easier for partner to work together in AR case (Q2)Face to Face AR ranked best

Observations and ConclusionsObservations

Tangible input proved to be very intuitiveUsers preferred audio over haptic cues as it provided greater awarenessPlayers adapted behavior to interface

ConclusionsCollaborative AR enhances face to face awarenessCollaborative AR enhances face to face awareness

Remote Collaboration

2008 - CNN

AR Conferencing

Moves conferencing from the desktop to the workspace

FeaturesHardware

SGI O2Virtual i-O HMDhead mounted camera

Softwarelive videoshared whiteboardvision based registration/tracking

Pilot StudyHow does AR conferencing differ ?

Taskdiscussing images12 pairs of subjects

Conditionsaudio only (AC)y ( )video conferencing (VC)mixed reality conferencing (MR)

Sample Transcript

Transcript Analysis

Users speak most in Audio Only conditionMR fewest words/min and interruptions/minMore results needed

Presence and CommunicationPresence Rating (0-100)

8090

100

40506070

0102030

Could tell when Partner was Concentrating14

AC VC MR

8

10

12

0

2

4

6

AC VC MR

Subjective CommentsPaid more attention to pictures Remote video provided peripheral cuesRemote video provided peripheral cuesIn AR condition

Difficult to see everythingDifficult to see everythingRemote user distractingCommunication asymmetriesCommunication asymmetries

A Wearable Conferencing Space

Featuresmobile video conferencingfull size imagesspatial audio/visual cuesinteraction with real worlddozens of users body-stabilized data

Initial PrototypeInternet TelephonySpatial Audio/VisualsSee-through HMDgHead TrackingWireless Internet Wireless Internet Wearable ComputerStatic Images

Software ArchitectureMulticast GroupsP B dPosition Broadcasting

10 kb/s per person

Audio Broadcasting 172 kb/s per person

Local sound spatializationDirectSound3DDirectSound3D

Graphics InterfaceDirectX/Direct3DDirectX/Direct3D

Pilot User Study

Can MR spatial cues aid comprehension?Task

recognize words in spoken phrases

ConditionsNumber of speakers

- 1,3,5 simultaneous speakers

Spatial/Non Spatial AudioVisual/Non visual cues

Spatial Sound

Spatial vs. Non Spatial Performance

4

5

2

3

Scor

e SS

NS

0

1

1 3 51 3 5

Num Speakers

Subjective Ratings

Ease of Understanding

56

5)

234

atin

g (1

-5

SS

NS

01

1 3 5

Ra

1 3 5

Num Speakers

3D Live: Virtual Viewpoint Generation

Virtual Viewpoint Generation

Interpolate between real camera views to generate a virtual camera viewgenerate a virtual camera view

AR Conferencing

Superimpose video of remote person over real world

System Architecture

Tangible Manipulation

Using real paddle to move virtual user

AR Remote Conferencing

Progression2D S i l C 3D2D to Spatial Cues to 3DIncreasing realism (visual/audio cues)

Multiscale Collaboration

The MagicBook

Goal: A collaborative AR interface supporting transitions from reality to virtual realitytransitions from reality to virtual realityPhysical Components

Real book

Display Elementsp yAR and VR content

Interaction MetaphorInteraction MetaphorBook pages hold virtual scenes

Milgram’s Reality-Virtuality continuum

Mixed Reality

Real Augmented Augmented VirtualEnvironment Reality (AR) Virtuality (AV) Environment

Reality - Virtuality (RV) Continuum

Milgram’s Continuum (1994)

Mixed Reality (MR)

Reality VirtualityAugmented AugmentedReality(Tangible Interfaces)

y(Virtual Reality)

Augmented Reality (AR)

Augmented Virtuality (AV)

Central HypothesisypThe next generation of interfaces will support transitions along the Reality-Virtuality continuum

Transitions

Interfaces of the future will need to support transitions along the RV continuumalong the RV continuum

Augmented Reality is preferred for:co-located collaboration

Immersive Virtual Reality is preferred for:experiencing world immersively (egocentric)sharing viewsrem te c llab rati nremote collaboration

MagicBook Metaphor

Features

Seamless transition between Reality and VirtualityR li l d i l iReliance on real decreases as virtual increases

Supports egocentric and exocentric viewsUser can pick appropriate view

Computer becomes invisibleComputer becomes invisibleConsistent interface metaphorsVirtual content seems real

Supports collaboration

Collaboration

Collaboration on multiple levels:Physical Objecty jAR ObjectImmersive Virtual Space

Egocentric + exocentric collaborationmultiple multi-scale usersp

Independent Views Privacy, role division, scalabilityPrivacy, role division, scalability

Technology

RealityNo technologyNo technology

Augmented RealityCamera – trackingSwitch – fly in

Virtual RealityCompass – trackingPress pad – moveSwitch – fly out

Scientific Visualization

Education

Hybrid Interfaces

Hybrid User InerfacesG l T AR l Goal: To incorporate AR into normal meeting

environmentPhysical Components

Real props

Di l El tDisplay Elements2D and 3D (AR) displays

Interaction MetaphorInteraction MetaphorUse multiple tools – each relevant for the task

Hybrid User Interfaces

1 2 3 4PERSONAL TABLETOP WHITEBOARD MULTIGROUP

Private Display Private DisplayGroup Display

Private DisplayPublic Display

Private DisplayGroup DisplayGroup Display Public Display Group DisplayPublic Display

Bridging Space (1)Office of the Future [Raskar98]

office environment office environment augmented with embedded front projection3D video conferencing

Bridging Space (2)Emmie [Butz99]

Shared virtual “ether“ metaphorIncorporate existing standard applications

B id i S (3)Bridging Space (3)

Studierstube (V2.0) [Schmalstieg2000]Similar multi-display ARMixed view applicationsExample: Storyboard design

MagicMeeting [Regenbrecht 2002]

Hybrid Interface for Face to Face CollaborationAR view, Projection Screen, Desktop, PDAj p

Tangible InteractionPhysical manipulatorsPhysical manipulators

Tangible Interactions

Cake Platter Rotation

R l P Cli i Pl Obj t Li htiReal Props – Clipping Plane, Object Lighting

Conclusions

Lessons Learned

Face to face collaboration AR f d i i VRAR preferred over immersive VRAR facilitates seamless/natural communication

R t C ll b tiRemote CollaborationAR spatial cues can enhance communicationAR f i i id f iAR conferencing improves video conferencingMany possible confounding factors

Future WorkWearable collaborative AR system

opportunistic collaborationjust in time training

Communication Asymmetries interface, expertise, roles

Usability Studiesymulti-user AR systemscommunication tasks