Enabling Localized Gestures via Acoustic Sensing on Smartwatches

Abstract In this paper, we present our on-going work on localized gestures (i.e., gesture localization and classification) by leveraging acoustic processing techniques. Our exploration focuses on hand gestures that produce non-vocal acoustics when it comes into contact with the user's body or nearby surfaces. The sounds made from the gestures can be recognized and localized simultaneously. We are working on such localized gestures that are expressive, feedback-rich and intuitive to use for a great breadth of smartwatch interactions. We present the interaction scenarios as well as discuss the challenges in implementing such localized gestures.

Author Keywords Localized gestures; smartwatch interaction; acoustic technology; noise delimitation.

ACM Classification Keywords H.5.2. Information interfaces and presentation (e.g., HCI): Input Devices and Strategies.

Introduction Smartwatches are becoming popular among consumers as they feature the ability to quickly glance at information when needed. Regardless of its miniaturized form-factors, smartwatch interactions are

Paste the appropriate copyright/license statement here. ACM now supports three different publication options:

• ACM copyright: ACM holds the copyright on the work. This is the historical approach.

• License: The author(s) retain copyright, but ACM receives an exclusive publication license.

• Open Access: The author(s) wish to pay for the work to be open access. The additional fee must be paid to ACM.

This text field is large enough to hold the appropriate release statement assuming it is single-spaced in Verdana 7 point font. Please do not change the size of this text box. Each submission will be assigned a unique DOI string to be included here.

Teng Han University of Manitoba Winnipeg, Canada hanteng@cs.umanitoba.edu Khalad Hasan University of Manitoba Winnipeg, Canada khalad@cs.umanitoba.edu Randy Gomez Honda Research Institution 8-1 Honcho Wako-shi, Japan r.gomez@jp.honda-ri.com

Keisuke Nakamura Honda Research Institution 8-1 Honcho Wako-shi, Japan k.nakamura@jp.honda-ri.com Pourang Irani University of Manitoba Winnipeg, Canada Pourang.Irani@cs.umanitoba.edu

commonly carried out with touch input. However, this “natural” input modality do not fit smartwatch interactions very well due to the device’s miniature form factors, as well as several well-known limitations such as screen occlusion and the “fat finger” problem [4]. Many solutions have been proposed to solve these problems, including techniques that enable gesture input around smartwatches.

Researchers have proposed gestural interaction on smartwatches leveraging spaces around the device. Examples include in-air gestures [6], on-body gestures [7], single-hand [2] and double-hand gestures [5] that have been presented in recent years’ CHI and UIST papers. Various sensing approaches have been investigated to support such gestural interaction. These include, but are not limited to, embedded cameras [5], IMU [6], external IR sensors [7] and capacitive fields [3]. However, none of the previous approaches are capable to combine around-device position tracking information with the detections of variance of gesture types. This shortage limits new interaction possibilities with smartwatches. Intuitively, users can benefit from finger position tracking in around-device space to complement finger gesture input such as click or pinch.

At the University of Manitoba HCI lab, we are collaborating with the Honda Research Institute Japan (HRI-JP) on investigating smartwatch interaction opportunities with localized gestures. We focus on extending smartwatch input with rich auditory sensing capabilities that simultaneously recognize and localize hand and finger gestures. Specifically, we consider the gestures that produce sounds, such as snapping or clapping, that can be recognized through sound source localization and classification. While building a

prototype to localize and recognize the sound sources generated via hand and finger gestures, we find several technical challenges. For instance, these gestures are prone to ambient noise and provide low accuracy in noisy environments. We propose different approaches to address these challenges to improve accuracy and robustness of localized gestures.

The paper contributes to the HCI community in two aspects. First, we systematically look at the smartwatch interaction space that benefits from using localized gestures. This opens opportunities to explore a rich set of interaction techniques to facilitate novel smartwatch input such as one-handed interaction, extending interaction space using 3D space around the device and rapid access to on-device content. Second, we discuss our implementation challenges such as accuracy and ambient noise to push the solution towards being robust and practical.

Localized Gestures We propose to use acoustic sensing approaches to detect localized gestures. We include three categories of gestures as shown in Figure 1. Examples include gestures that produce sounds by fingers on the same hand (i.e., in-air gestures), hand or finger contacting objects (i.e., on-object gestures) and hand or finger contacting body (i.e., on-body gestures).

These gestures are intuitive to use. Users are familiar with in-air gestures such as finger snapping, rubbing and flicking, as well as on-body gestures such as hand clapping. Users perform these gestures occasionally but not much in daily life. Therefore, they can be mapped to smartwatch commands without interfering with users' daily actions. However, we are aware of the

In A

ir

Snap Rub

Flick Pinch

On

Bod

y Clap Hand Rub

Bump Clasp

On

Obj

ect

Clap Scratch

Tap Punch

Figure 1: Hand-made gestures

issues with physical discomfort while performing gestures and their social acceptance in public places such as in a shopping mall or in a restaurant [1]. These issues could be further explored with users' preference on performing the gestures in different social contexts.

These gestures are rich and expressive. Users may consider using different sound sources. Tapping on different objects, as well as, on different body parts are likely to make different and distinguishable sounds. The gesture vocabulary can also be expanded by including acoustic features such as tempo and rhythm. There are short and long sounds from hand gestures that can be mapped for discrete and continuous operations on smartwatches, such as finger snapping and finger rubbing respectively.

The gestures provide both acoustic and tactile feedbacks. The sounds produced from hand gestures are unique and users require no visual feedback to perform and understand these gestures. Additionally, tactile cues associated with the gestures are generated from physical contacts with objects, body, and hand.

Interaction Scenarios While numerous scenarios are possible with localized gestures on smartwatches, we discuss those demonstrating novel ways to interact with smartwatches.

One-handed interaction Localized gestures can be performed with one-hand where the sound is generated with the hand wearing the device. This makes it possible to use the smartwatch in single-handed mode while the other hand is not available to interact with the smartwatch.

For instance, a user could leverage a physical surface such as a table as available input space for one-handed interaction. In such a scenario, the device can continuously track the directional scratching sound of a finger on a table, either from top to bottom or in reverse as shown in Figure 2. These actions can be associated to browse images in an image gallery.

Extended Input Space on Smartwatch Interactions on smartwatches are typically limited to the touchscreen space. However, localized gestures could be used to extend the limited input capabilities on smartwatches. An as example, operating a 2D pie menu with many items can be challenging to access with touchscreen on smartwatches. Instead, users can leverage localized gestures for accessing menu items on the smartwatch as shown in Figure 3. Users navigate (or highlight) main menu items by rubbing their finger at the corresponding directions. A snap gesture selects the highlighted menu item and a flick gesture switches back to the main menu items.

Rapid access with hotkeys In mobile contexts, hotkeys become very useful where users engage in short activity bursts. However, there are limited provisions for hotkeys on mobile devices and smartwatches. Localized gestures could be used as hotkey input to quickly access information on the smartwatches. For instance, a jogger can check heart rate by tapping his hand on the chest as shown in Figure 4. He can also check step counts by tapping on his thigh. Our prototype is capable of detecting these events based on both sound localization and classification information.

Figure 2: A user scratches fingers

on a table to scroll images

Figure 3: the user rubs fingers to select items from a 2D pie menu.

Figure 4: Taps on chest to enable

quick access to heartrate data.

Figure 5: Two-player pong

game.

Ad-hoc group play Localized gestures support co-located user activities for ad-hoc groups. Due to the limited interaction space on smartwatches, such collaboration are not suitable on smartwatch platform. In Figure 5, we illustrate a two-player pong game where users can handle the paddle movement by a localized gesture (i.e., finger snap). This enable smartwatch to be a device that supports multiplayer device.

Technical Challenges As with any acoustic input, localized gestures are affected by environment noise. Solving the noise issue is not trivial. In the following paragraph, we will discuss different approaches to delimitate noise to improve accuracy and robustness of such interactions.

Non-gestural sounds could be produced in a wide range of contexts such as while people are chatting, walking, or even closing a door. These are treated as random spatial discrete noise events. Our prototype has the ability to separate sound sources from various positions that happen simultaneously (see “Sound source localization and separation”). This allows the system to identify such non-gestural sounds and therefore reduce the interference from them.

Ambient noise is often continuous, such as the sound from air conditioning or computers that are constantly running. We first set thresholds of sound loudness and frequency to filter out part of these background noise. A second and more robust way is enhancing the sound signal by subtracting noise features in the frequency domain with a sample of background noise. This is achieved by embedding a trigger mechanism into the system. Specifically, a user shakes the watch twice to

awake the system, which then records sound for 2 secs. These sound waves are used as background noise samples, from which noise features are obtained. In this way, we are also able to record gesture sounds samples online to train and use the gestures in the same environmental context. In the future, we will work on adaptive noise enhancement techniques in dealing with difficult variant of noise types (i.e., colored noise)

Conclusion and Contribution to the Workshop In this paper, we present a novel approach of using non-verbal acoustic for smartwatch interactions. We present localized gestures that are produced when one hand or finger comes in contact with other hands or fingers. We showed that such gestures have potential to enhance interaction capabilities on the smartwatch. In addition, we discussed challenges of such acoustic sensing approaches on small device platforms. Currently we are implementing a prototype with rich auditory sensing capabilities that recognizes and localizes hand-made gestures.

We hope this position paper will inspire researchers in designing acoustic sensing technologies for mobile and wearable devices.

References 1. David Ahlström, Khalad Hasan, and Pourang Irani.

2014. Are you comfortable doing that?: acceptance studies of around-device gestures in and for public settings. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14), 193-202.

2. Chris Harrison and Scott E. Hudson. 2009. Abracadabra: wireless, high-precision, and

unpowered finger input for very small mobile devices. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). 121-124.

3. Jun Rekimoto. 2001. Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Proceedings of Fifth International Symposium on Wearable Computers, 21-27. IEEE.

4. Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air gestures around unmodified mobile devices. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14). 319-329.

5. Wouter Van Vlaenderen, Jens Brulmans, Jo Vermeulen, and Johannes Schöning. 2015. WatchMe: A Novel Input Method Combining a

Smartwatch and Bimanual Interaction. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). 2091-2095.

6. Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 3847-3851.

7. Anusha Withana, Roshan Peiris, Nipuna Samarasekara, and Suranga Nanayakkara. 2015. zSense: Enabling Shallow Depth Gesture Recognition for Greater Input Expressivity on Smart Wearables. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). 3661-3670.