1. Touchless Circular Menus: Toward an Intuitive UI for Touchless Interactions with Large Displays Debaleena Chattopadhyay & Davide Bolchini AVI 2014 International Working Conference on Advanced Visual Interfaces Como (Italy) May 27-30, 2014

2. Public Spaces Sterile Operating Rooms Interactive TVs Large Displays Touchless Interaction Usage contexts 3. Problem Space 4. Touchless interaction with large displays still lacks a standardized user interface language for frequent user- operations, such as.. 5. Command Selection Menu Marking Menu, 1994 FlowMenu, 2000 Toolglass, 1993 rapMenu, 2008 Grab, 2013 Finger-Count Menu, 2011 6. Purpose of the Work 7. To propose and validate a novel form of command selection language appropriate for touchless interaction with large displays. 8. Design Space 9. Current Debate on UI naturalness Norman, D. A. (2010). Natural user interfaces are not natural. interactions, 17(3), 6-10. Wigdor, D., & Wixon, D. (2011). Brave NUI world: designing natural user interfaces for touch and gesture. Elsevier. O'hara, K., Harper, R., Mentis, H., Sellen, A., & Taylor, A. (2013). On the naturalness of touchless: putting the interaction back into NUI. ACM Transactions on Computer-Human Interaction (TOCHI), 20(1), 5. You are the controller Microsoft Most gestures are neither natural nor easy to learn or remember. Norman Natural is a design philosophy and a source of metrics enabling an iterative process to create a product. Wigdor & Wixon Naturalness is not something to be represented but is rather an occasioned property of action [..]. OHara et al. 10. Intuitiveness of an interface Prior knowledge Expertise Culture Sensorimotor Innate Specialized knowledge acquired with practice. Knowledge of people specific to culture. Knowledge acquired since childhood through interaction with the world. Reflexes and intrinsic behavior. unconscious application of prior knowledge Hurtienne, J., & Israel, J. H., 2007. Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. Proc. TEI,127-134, ACM. Intuitive Interaction Framework 11. How can we use sensorimotor knowledge to inform the design of intuitive touchless selection mechanisms? 12. Device less interaction Device based interaction Menu Invocation Mouse click, pen down, or touching surface ? Menu-Selection Delimiter Breaking contact with the interface Menu Shape ? Linear, radial ? Hand posture Hand posture Linear, radial 13. What successful features of device-based menus can be "ported" to the touchless environment? 14. Prior sensorimotor knowledge Some successful features of device- based menus Touchless Circular Menus (TCM) + Directional strokes in mid-air Menu-invocation: Reaching a region-of- interest. Menu-selection: Crossing Shape: Radial 15. Touchless circular menus relieve users from both recalling a precise vocabulary of hand-postures and strictly complying with them. 16. Demo 17. To select a menu option, the user makes a directional stroke. To give feedback, when successfully selected, the menu- option changes color. 18. To mitigate accidental invocation, TCM appear opposite to the users direction of approach. To cancel the invoked menu, the user may continue in her direction of movement, or move away from the menu. 19. To invoke the menu, the user must reach the region-of- interest (ROI) of the target. 20. Hierarchical TCM: To select a command, the user first selects a menu option, and then accesses the submenu by continuing her trajectory. 21. The Design 22. To trigger the contextual menu, a user must cross the region-of-interest (ROI) of a display object. The ROI can be of any symmetrical shape around the center of the target, with its size directly proportional to the techniques sensitivity. Menu Invocation 23. To select a command after triggering the menu, users cross it using a stroke in the commands direction. Until the crossing happens, users can cancel TCM by moving in any direction away from the triggered menu. To allow easy escape routes, we designed the structure of TCM as a semicircular array of options appearing at the top-left or the bottom-right corner of the target. Menu Selection 24. As users approach the menu, to give them orientation, a trace is drawn connecting the target and the users hand position. To improve users pointing performance, the menu options would increase in amplitude as users approached them. To provide further feedback, menu options changed color when selected. Menu Selection 25. Currently, the menu design scales up to two levels (5 x 5), with users performing continuous strokes. To operate the submenu, users change their track and cross another command. Due to the lack of precision and control of freehand movements, TCM require users to make inflections in their continuing trajectories, and thereby avoid accidental command- selections. Hierarchical TCM 26. Evaluation 27. Experiments How effectiveness and efficiency of TCM is affected by their triggering locations on the visual interface? 1 Vs. Touchless circular menus Linear menus using a grab gesture 2 Participants (N = 15) were sitting away from a large display. 28. Experimental Design Sample Demographics N = 15 All right-handed 4/15 females 8/15 had prior familiarity with touchless gestures 11/15 were below 30 years of age Time/participant: 1 hour 1Experiment 9 trials X 7 blocks X 15 participants = 945 trials Experiment 2 6 trials X 7 blocks X 15 participants = 630 trials 29. Apparatus Display Built by Fakespace, eight 50 projection cubes, each with a resolution of 1600 X 1200 pixels. 160 wide and 60 high display Total resolution: 15.3 M pixels. Motion Tracking Sensor Kinect for Windows UITS Research Technologies Advanced Visualization Lab 30. Outcome Measures Efficiency: Time on task | Successful Trigger Rate Effectiveness: Error Rate User Satisfaction: SUS | NASA TLX 31. 1Experiment Hypotheses H1: Triggering location will affect the efficiency of TCM. H2: Triggering location will affect the effectiveness of TCM 32. Triggering location of TCM significantly affected selection time and successful trigger rate. I felt I had to rush to select the menu option I was surprised that I could do so well. 1Experiment Results 33. Arm posture affects users control on their hand movements and the required effort. 1Experiment 2D plane locations relative to the users body and arm configuration affects the consumed endurance in touchless interactions Hincapi-Ramos, J. D., Guo, X., Moghadasian, P., & Irani, P. (2014). Consumed Endurance: A metric to quantify arm fatigue of mid-air interactions. Proc. CHI, 1063-1072, ACM. 34. Menu triggering locations will significantly affect the user experience of contextual touchless menus for large displays. Arm posture affects users control on their hand movements and the required effort. Contextual touchless menus for large displays will have different menu- triggering locations. 1Experiment 35. Hypotheses H3: We predicted TCM would be more efficient than linear menus. H4: We hypothesized that TCM would be easier to use than linear menus. Experiment 2 36. Experiment 2 Compared with linear menus, users were more efficient with TCM, and perceived lower overall workload. But TCM were less effective than linear menus. This is how I envision using touchless gestures. Results It was a lot of effort. 37. Experiment 2 Compared with linear menus using grab gestures, participants using TCM were twice faster in selecting commands and perceived lower workload. However, TCM caused 3% more errors than linear menus. 38. Experiment 2Why TCM were less effective? 1st gesture registration to invoke the menu 2nd gesture registration to select the menu Gesture relaxation 39. Experiment 2Why TCM were less effective? 1st gesture registration to invoke the menu 2nd gesture registration to select the menu No gesture relaxation 40. Future Work 41. Minimize errors by constraining users freehand movements after triggering the TCM. Investigate bimanual gestures to operate menu levels in hierarchical TCM. 42. Takeaways Menu triggering locations significantly affect the user experience of contextual touchless menus for large displays. Simple directional movements for selecting commands is more efficient and causes lower workload than (hand) posture-based techniques. TCM relieve users from learning posture-based commands, and shift the interaction complexity from users input to the visual interface. Thank you. We thank the study participants and UITS AVL. This research is partially supported by an IUPUI Research Support Funds Grant. 1 2 3 Efficient gesture relaxation techniques may have the potential to increase effectiveness of touchless interactions. 4