69
UIs and User Centred Design Techniques for AR + VR

User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Embed Size (px)

Citation preview

Page 1: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

UIs and User Centred Design Techniques for AR

+ VR

Page 2: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

What’s AR and VR?!Augmented Reality and Virtual reality.AR: technology that layers over our everyday VR: technology that transports us to a different world

Page 3: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

But They Must be Years Away?!https://vimeo.com/104034319

Page 4: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

How do we design applications for AR and VR?

Page 5: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

IxD for System vs IxD for FrameworkDesigners are even confronted with the obstacle of a methodology or framework to design the system itself and quickly iterate through prototypesSteps are being taken to address these concerns Few applications / frameworks although commercial + open source are helping

Page 6: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Evaluating VR UI’s Goals FormationNavigate the WorldLocate ObjectsPosition for InteractionDeciding on ActionManipulating ObjectsRecognise FeedbackInterpret FeedbackDecide on New Action

Page 7: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

VR Related Issues with EvaluationObjects can obscure and may break interaction cycleDifferent modes of design for navigation and for environment driven VR systemsExpert users see the modes blend together

Feedback should be multisensoryCan’t see things off screen or behind a wall etc. Environmental cues are key

Page 8: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Evaluation TechniquesWalkthrough each phase step by stepAsk necessary questions along the way ◇ Aim to uncover breaks in affordances◇ Questions guided to create generic design

principles (GDPs)Collect design issues and virtual environment featuresPrioritise them for development

Page 9: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

UCD Tips for AR Install on a familiar device

Choose a design scenario

Run in an appropriate setting

Build for two hands if on mobile

Choose the right audience

Challenge users with mental flow not with physical strain

Page 10: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Issues OverallInformation is often weakly exhibited that link design problems with design solutions in VR and AR.Even some of Norman’s interaction evaluation techniques can break down when dealt with navigating 3D space.Need a tailored methodology for dealing with emerging issues in hardware and interaction.

Page 11: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Virtual RealityUses, Current Tech and Design

Page 12: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Uses of VREducationWith the leaps in technology, virtual reality can be used to transport people to other planets, tourist destinations and the many jungles and oceans on earth.

Video GamesVirtual Reality allows players to be transported to other worlds and puts them in the middle of the action!

MedicalVirtual reality can allow surgeons to move throughout the body and diagnose problems that patients have. Virtual reality is also being used for therapy for PTSD veterans and phantom limb syndrome.

source: openmedical.orgsource: immersivevreducation.com source: military.com

Page 13: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

https://www.youtube.com/watch?v=kBpFx-ixBiM

Page 14: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

VR Technology available today

Page 15: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Designing for VR

Page 16: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

User Interface design in VRWhen designing interfaces in a virtual reality, there are some

considerations which must be taken into account.

● Is there a motion controller used in conjunction with the VR

system?

● What is the nature of the experience?

● Who is the interface being designed for?

● What are the perceptual limitations of the user?

Page 17: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Perceptual LimitationsPerceptual limitations occur in the areas that users find interaction difficult in their field of view.

30

55

60

20

12

40

comfortableco

mfo

rtabl

e72

102

Peripheral

Page 18: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Perceptual LimitationsPerceptual limitations occur in the areas that users find interaction difficult in their field of view.

30

55

60

20

12

40

comfortableco

mfo

rtabl

e72

102

Peripheral

0.5mRegion where

convergence can occur

Page 19: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

How do we design UIs with these limitations in mind?

Page 20: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

How do we design UIs with these limitations in mind?

Page 21: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Is this a good UI design or bad UI design in VR?

Page 22: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Augmented RealityUses, Current Tech and Design

Page 23: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Uses of ARNotificationAR headsets can notify you of social media, texts or email as you go about your daily life.

Video GamesAR allows players to have their world transformed in front of their very own eyes!

NavigationAR allows people to find their destination in a non- obtrusive way.

source: play.google.comsource: google.com source: nintendo.com

Page 24: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

AR Technology available today

Page 25: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Designing for AR

Page 26: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

User Interface design in ARWhen designing interfaces in a augmented reality, the considerations taken into account are similar to VR.

But there is one thing that is the utmost importance!

Obscurity / Opacity.

Page 27: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

User Interface design in VR

Continue 10m

Page 28: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

User Interface design in VR

Page 29: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

User Interface design in VR

Page 30: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Case Studies1.Design and Evaluation of Menu Systems for Immersive Virtual Environments - Bowman & Wingrave 2001

2.Experimental Evaluation of User Interfaces for Visual Indoor Navigation - Moller et al. 2014

Page 31: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Key Interaction Tasks VRNavigation

Selection

Manipulation

System Control

Page 32: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

TULIP Menu vs

Floating Menu vs

Pen and Tablet Menu

Page 33: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Pinch Gloves

Page 34: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Hardware for TULIP MenuPinch Gloves consist of a flexible cloth gloves augmented with conductive cloth sewn into the tips of each of the fingers.

When two or more pieces of conductive cloth come into contact with one another, a signal is sent back to the host computer indicating which fingers are being “pinched”

Virtual Research V8 head-mounted display (HMD) and the head and both hands are tracked using a Polhemus Fastrak tracking system

Page 35: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Evolution of TULIP

Scrolling Menu

Three Up Menu

Page 36: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Scrolling Menu

Page 37: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Three Up Menu

Page 38: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Pilot StudyEvaluating these two menu designs, users had to change a virtual object to match a target object.

3 parameters could be controlled: the object’s shape, color, and texture. Each of these corresponded to a top-level menu.

There were 3 shapes to choose from, 8 colors, and 6 textures – these corresponded to second-level menu items.

Test 4 users - “Think Aloud” - Informal Results

Page 39: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Pilot ResultsNeither design satisfied the desired requirements.

Users did prefer the Scrolling Menu BUT realized tasks could be completed with less steps using the Three Up Menu.

Three Up Menu hides options if there is more than 3.

Scrolling Menus prompt users to incorrectly attempt to push palms.

Lack of feedback when items selected and fatigue from hands being raised.

Page 40: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

RE-DESIGNT hree

U p

L abels

I n

P alm

Page 41: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Summative EvaluationCompare Ease Of Use

Compare Ease Of Learning

Efficiency

Comfort of all 3 Menus

Page 42: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Summative Evaluation26 users participated

Repeated object matching task

Completed a questionnaire containing demographic information and information about their experience with computers and VEs

Same equipment used again with added stylus for floating menus and pen and tablet menus

30 trials of each menu and no help provided after initial briefing

Page 43: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Floating Menus

Pen and Tablet Menus

Page 44: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ResultsAppears that the gloves were the hardest to learn initially, but performance was at reasonable levels for all three types within five trials.

Reason times for the pen and tablet menu are initially poor is that users were not told they needed to look at the tablet in their hand

Page 45: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Comfort LevelsThe main drawback of the pen and tablet system is the discomfort it causes users, which might be alleviated by adding an ergonomic handle.

Page 46: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Reflection On StudyCombining the efficiency, comfort, and preference information, it appears that both the pen and tablet menu and the TULIP menus performed well in the evaluation

Fifteen users expressed a preference for the TULIP interface, while nine preferred the pen and tablet, and only two preferred the floating menus

This evaluation reiterated some important heuristics from the traditional human-computer interaction literature. Menu systems for VEs need to have good feedback, affordances, and constraints, and items and their actions should be visible.

Page 47: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Evaluation of User Interfaces for Visual Indoor Navigation

Page 48: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Andreas Moller et al.Implemented a novel UI for visual localization, consisting of Virtual Reality (VR) and Augmented Reality (AR) views that actively communicate and ensure localization accuracy.

Beneficial for large buildings and navigating your way around.

The concept consists of a panorama-based view as a complement to Augmented Reality and proposes different visualizations for motivating users to record “good” query images.

Good query images are important

Page 49: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Comparing AR vs VRThe users would hold the phone up as seen in the figure and look at the phone in order to see the augmentation i.e. items superimposed onto their real life surroundings.

The virtual reality displays pre-recorded images of the environment (downloaded from a server) that are arranged to a 360◦ panorama on the mobile device

Page 50: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Enhancing Visual Localization

Visual localization can be very dependent on how the device is being held.

Four indicator types were proposed which were:- Text Hint- Blur- Colour Scale- Spirit Level

Page 51: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Initial TestingA questionnaire-based survey with mockup videos were used for early testing. Users did not actually travel through a building.

From initial user evaluations there was an inconsistency. Majority of people stated that the VR mode helped them orient themselves even if the location estimate of the system was incorrect yet the subjects still claimed they prefer the AR mode.

Users preferred the Text Hint and Spirit Level as a means to prompt user to provide a better quality of image.

Page 52: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

PrototypeA prototype was built using Android 2.3

Had both the AR and VR modes

Wizard Of Oz technique - experimenter would play a role

The navigation interface on the subject’s device was implemented with OpenGL ES 2.0.

For the automatic trigger, they used a FAST feature detector from the OpenCV framework for Android to detect the number of features in the camera’s live image.

Page 53: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ExperimentsSo the goal of the experiment was to verify results from earlier mock-ups. Three experiments took place to test:

1. Efficiency, perception and convenience of AR and VR under different accuracy conditions.

2. Effectivity of UI elements specific to vision-based localization.

3. Convenience and distraction of object highlighting.

12 participants took part most of which were students but none were involved in the project.

Page 54: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

HardwareSubjects used a Samsung Galaxy S II (4.3- inch screen, 8 megapixel camera)

The WOz app ran on a Samsung Nexus S (4-inch screen).

Both devices had a screen resolution of 480×800 pixels.

Page 55: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

UIs Implemented (subject)

Page 56: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

UIs Implemented (WoZ)

Page 57: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Experiment 1- Navigation using AR and VR

Subjects performed a navigation task in a university building on a path of 220 meters length, using both the AR and the VR mode. Each user traversed the path eight times and was asked to rely only on the given instructions.

Navigation instructions were fed into the subject’s phone by the experimenter (Wizard of Oz).

Sent the appropriate panoramas in VR mode (and directional arrows in AR mode) to the subject’s phone using the WOz interface.

Users were encouraged to think allowed

Page 58: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality
Page 59: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ResultsSubjects were in average 25 seconds faster to reach their destination with VR (averagely 2:39 minutes for the 220 m path) than with AR (averagely 3:04 minutes). The experiment also proved that AR was worse with regards to orientation errors.

Subjects found carrying the phone more convenient in VR. The required upright position for carrying the phone in AR was physically constraining

Page 60: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Experiment 2 Vision Specific UI ElementsSubjects performed a navigation task on the path shown in figure, but in opposite direction as in Experiment 1, so that the path was not already too familiar.

Three times during the walk, a relocalization procedure was simulated.

The experimenter triggered a spirit level visualization to appear on the subjects’ device. This indicator told subjects to collect enough features for relocalization.

Page 61: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ResultsReliable localization requires 100 to 150 features in the image. While the indicator was visible, the average number of detected features per frame rose from 42 to 101.

The experiment also showed that subjects preferred the lower carrying position for VR mode, compared to the upright pose for AR mode

Page 62: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Experiment 3 Object Highlighting MethodsThere were two ways to highlight objects: Frame and Soft Hightlight

The algorithm was optimized to detect square, feature-rich objects out of a uniform background. This applies to, e.g., a poster on a wall.

Subjects pointed at the posters using both highlighting visualizations. Feedback was afterwards collected by a questionnaire.

Page 63: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ResultsOn a Likert scale from -3 to +3, subjects indicated that Frame drew more attention to the poster (M = 3) than Soft Highlight (M = 1). The semi-transparency of Soft Highlight complicated readability of text on the poster. Regarding distraction, the visible contours of the Frame visualization were perceived as unstable.

Page 64: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Reflection on ExperimentsVR mode turned out to be advantageous in several ways. Contrary to initial feedback, AR UI appeared far more appealing in theory.

AR UI does have it strengths though; It can help to improve feature collection using the feature indicator i.e. spirit level prompt to probability of reliable re-localization.

Subjects reported that the frequent updates of the panorama images in VR mode were partly irritating, especially when not permanently looking at the screen

Further studies are required in the field to strengthen these UI concepts, particularly with regards to AR and seeking more reliable localization.

Page 65: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

Soooo what’s the activity? Considering the revised User Centred Design Model, we’re going design some augmented reality applications.

● Navigation● Notification System● Photo and Video ● Messaging System

Page 66: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

thanks!ANY QUESTIONS?

Page 67: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ReferencesAlger, M., 2015. Visual Designs for Virtual Reality.Alger, M., n.d. VR Interface Design Manifesto.Alger, M., n.d. VR Interface Design Pre-Visualisation Methods.Bowman, E., Wingrave, C. (2001) "Design and Evaluation of Menu Systems for Immersive Virtual Environments", in Proceedings Of The Virtual Reality 2001 Conference (VR'01) (VR '01), IEEE Computer Society.Broll, W., Shafer, L., Hollerer, T., Bowman, D., 2001. Interface with angels: the future of VIR and AR interfaces. IEEE Computer Graphics and Applications 21, 14–17. doi:10.1109/38.963455De Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., Poulovassilis, A., 2010. Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. British Journal of Educational Technology 41, 69–85.Denis, J.-M., 2015. From product design to virtual reality. Google Design.DesigningInteractions_8.pdf, n.d.Faaborg, A., n.d. Designing for virtual reality and the impact on education. TEDx Talks.Hsu, C., Shiau, H., 2013. The Visual Web User Interface Design in Augmented Reality Technology. Editorial Preface 4.

Page 68: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ReferencesJang, D., Kim, J.-S., Li, K.-J., Joo, C.-H., 2011. Overlapping and synchronizing two worlds. Proceeding GIS ’11 Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems 493–496.Kunz, A., Wegener, K., 2013. Towards natural user interfaces in VR/AR for design and manufacturing. Eidgenössische Technische Hochschule Zürich, Institute of Machine Tools and Manufacturing.Leap Motion VR Best Practices Guidelines.pdf, n.d.rabedik, n.d. AR Screen Hackathon Project.Möller, A., Kranz, M., Diewald, S., Roalter, L., Huitl, R., Stockinger, T., Koelle, M., Lindemann, P. (2014) "Experimental evaluation of user interfaces for visual indoor navigation", in Proceedings Of The SIGCHI Conference On Human Factors In Computing Systems (CHI '14), ACM: New York, NY, USA.Samsung Developer Connection, 2014. VR Design: Transitioning from a 2D to 3D Design Paradigm.Sutcliffe, A., n.d. Multimedia and Virtual Reality: Designing Multisensory User Interfaces. Pyschology Press, 2003.Sutcliffe, A, & Deol Kaur, K 2000, “Evaluating the usability of virtual reality user interfaces”, Behaviour & Information, n.d.Ventä-Olkkonen, L., Posti, M., Koskenranta, O., Häkkilä, J., 2014.

Page 69: User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

ReferencesInvestigating the Balance between Virtuality and Reality in Mobile Mixed Reality UI Design – User Perception of an Augmented City. Proceeding NordiCHI ’14 Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational 137–146.Wesolowski, M., n.d. Designing Next-Gen Virtual Reality Gaming Experiences. www.techcrunch.com.Sutcliffe, A, & Deol Kaur, K 2000, 'Evaluating the usability of virtual reality user interfaces', Behaviour & Information Technology, 19, 6, pp. 415-426, Academic Search Complete, EBSCOhost, viewed 12 February 2016.Uxmag.com, (2016) User Testing As A Design Driver : Looksery Created A Product For Users, Not Designers: Looksery Created A Product For Users, Not Designers | UX Magazine [online], available: https://uxmag.com/articles/user-testing-as-a-design-driver [accessed 20 Feb 2016].Uxmag.com, (2016) User Testing As A Design Driver : Looksery Created A Product For Users, Not Designers: Looksery Created A Product For Users, Not Designers | UX Magazine [online], available: https://uxmag.com/articles/user-testing-as-a-design-driver [accessed 292Feb 2016].

1