Embed Size (px)
Citation preview
Full Terms & Conditions of access and use can be found athttps://www.tandfonline.com/action/journalInformation?journalCode=tbit20
Behaviour & Information Technology
ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: https://www.tandfonline.com/loi/tbit20
User-defined gesture interaction for immersive VRshopping applications
Huiyue Wu, Yu Wang, Jiali Qiu, Jiayi Liu & Xiaolong (Luke) Zhang
To cite this article: Huiyue Wu, Yu Wang, Jiali Qiu, Jiayi Liu & Xiaolong (Luke) Zhang (2019)User-defined gesture interaction for immersive VR shopping applications, Behaviour & InformationTechnology, 38:7, 726-741, DOI: 10.1080/0144929X.2018.1552313
To link to this article: https://doi.org/10.1080/0144929X.2018.1552313
Published online: 02 Dec 2018.
Submit your article to this journal
Article views: 162
View Crossmark data
Citing articles: 1 View citing articles
User-defined gesture interaction for immersive VR shopping applicationsHuiyue Wu a, Yu Wangb, Jiali Qiua, Jiayi Liua and Xiaolong (Luke) Zhangc
aThe School of Communication and Design, Sun Yat-sen University, Guangzhou, People’s Republic of China; bDepartment of Medical Oncology,Sun Yat-sen University Cancer Center, State Key Laboratory of Oncology, South China, Collaborative Innovation Center for Cancer Medicine,Guangzhou, People’s Republic of China; cCollege of Information Sciences and Technology, Pennsylvania State University, University Park, PA,USA
ABSTRACTGesture elicitation studies, which are a popular technology for collecting requirements andexpectations by involving real users in gesture design processes, often suffer from gesturedisagreement and legacy bias and may not generate optimal gestures for a target system inpractice. This paper reports a research project on user-defined gestures for interacting withimmersive VR shopping applications. The main contribution of this work is the proposal of amore practical method for deriving more reliable gestures than traditional gesture elicitationstudies. We applied this method to a VR shopping application and obtained empirical evidencefor the benefits of deriving two gestures in the a priori stage and selecting the top-two gesturesin the a posteriori stage of traditional elicitation studies for each referent. We hope that thisresearch can help lay a theoretical foundation for freehand-gesture-based user interface designand be generalised to all freehand-gesture-based applications.
ARTICLE HISTORYReceived 9 September 2018Accepted 20 November 2018
KEYWORDSGestural interaction; user-centered design; elicitationstudy; virtual reality; onlineshopping
1. Introduction
Virtual reality (VR) is a very popular technology that hasbeen extensively used in the field of human–computerinteraction (HCI) for a variety of purposes in recentyears. One of the applications in which VR has beenwidely used to simulate physical presence in real-worldscenarios is VR-based online shopping. During the pastfew years, many efforts have been made to investigatethe potential of VR technologies for online shopping.As a result, many systems were developed, such as virtualsupermarkets (Josman et al. 2009; Josman et al. 2014), avirtual interactive shopper (Hadad 2012), and a virtualmall (Rand, Weiss, and Katz 2009). Virtual-reality-based online shopping systems can deliver efficient infor-mation about products and not only positively affect cus-tomers’ attitudes but also influence offline sales(Altarteer et al. 2017).
However, most existing VR-based shopping appli-cations were implemented in a desktop virtual environ-ment in which traditional input devices such as amouse (Cardoso et al. 2006), keyboard (Klinger et al.2006), and handle (Carelli et al. 2009) were used by par-ticipants to navigate and collect items. The usability pro-blems that are caused by the traditional WIMP-basedinterface paradigm (Window, Icon, Menu, and PointingDevice), such as low input/output bandwidth, low degreeof interactive freedom, discrete response and feedback,
and the requirement of accurate input, have substantiallydegraded customers’ interactive experiences. With therapid development of VR technology and new inter-action styles that draw on people’s knowledge, experi-ence and mental models that are built in real-worldscenarios, the optimal use of end-users’ sensory channelshas become a challenge for user interaction with VR-based shopping systems.
One way to address the challenge is to support free-hand-gesture-based immersive VR interaction foronline shopping systems. Freehand gestures, which area flexible input modality, have been extensively appliedin user interaction in many HCI applications, such asVR/AR (Feng et al. 2013; Piumsomboon et al. 2013),computer games (Kulshreshth and LaViola Jr. 2014),wearable computing (Tung et al. 2015; Shimon et al.2016; Gheran, Vanderdonckt, and Vatavu 2018),smart homes (Takahashi et al. 2013; Wu, Wang, andZhang 2016), robots (Yang, Park, and Lee 2007;Obaid et al. 2012) and UAVs (Pfeil, Koh, and LaViolaJr 2013; Peshkova et al. 2017(b)). Recent advances insensor technologies, biocybernetics techniques and mar-kerless motion capture technologies (Yang et al. 2012;Rautaray and Agrawal 2015; Cheng, Yang, and Liu2016; Cai et al. 2017) have also expanded the appli-cation scope of freehand gestures in user interactiontools in real-world settings.
© 2018 Informa UK Limited, trading as Taylor & Francis Group
CONTACT Huiyue Wu [email protected]
BEHAVIOUR & INFORMATION TECHNOLOGY2019, VOL. 38, NO. 7, 726–741https://doi.org/10.1080/0144929X.2018.1552313
Although empirical evidence for the benefits of free-hand gesture interaction has been obtained, open issuesand unanswered questions remain in the design of free-hand gesture interfaces. For example, how can we deter-mine the optimal gestures for a target system in practice?And what is the most suitable mapping between a ges-ture and the corresponding target task? Gesture elicita-tion study, which is a technique that has emerged fromthe field of participatory design, has attracted increasingattention and been widely used to collect end-users’requirements and expectations regarding the target sys-tem in the gesture design of various HCI applications.However, this approach suffers from the legacy bias pro-blem (Morris et al. 2014) and the disagreement problem(Wu et al. 2018) in practice and may often becomecaught in local minima and fail to identify gesturesthat may be better suited for specified system tasks.
Recently, efforts have been made to address theseissues by applying priming and production techniques(Morris et al. 2014; Chan et al. 2016; Hoff, Hornecker,and Bertel 2016; Chen et al. 2018) in gesture design toask participants to design at least three gestures for aspecified task. However, the practical effectiveness ofthis method might be limited because participantsfound it difficult to design so many gestures at a timefor each task, especially when they were not familiarwith the gesture design space. This paper reports ourresearch on user-defined freehand gestures for immer-sive VR shopping applications. The main differencebetween our work and prior elicitation studies lies inthe consideration of deriving two gestures in the a prioristage and selecting the top-two gestures in the a poster-iori stage of traditional elicitation studies for each task.Experimental results demonstrate that the proposedmethod can effectively alleviate gesture disagreementand offset end users’ legacy bias in traditional gesture eli-citation studies.
The remainder of this paper is structured as follows:First, we review related work and introduce our researchmethodology. Next, we describe our experimentalstudies, report their results, and discuss our findings.Finally, we conclude the paper by discussing the contri-butions of our research and possible future researchdirections.
2. Related work
Our work primarily concerns research that is related tothe application of freehand gestures in VR shoppingapplications and elicitation studies on gesture-basedinteraction in HCI. Thus, our review focuses on workin these areas.
2.1. Freehand-gesture-based interaction for VRshopping applications
Hands are our most dexterous body parts and are heavilyused in both the real world and virtual environments toperform various interactive tasks such as manipulationof virtual objects (Song et al. 2012; Feng et al. 2013; Alke-made, Verbeek, and Lukosch 2017), navigation in virtualenvironments (Tollmar, Demirdjian, and Darrell 2004;Sherstyuk et al. 2007; Verhulst et al. 2016), and systemcontrol (Kölsch, Turk, and Höllerer 2004; Colaco et al.2013).
In addition to the abovementioned studies, freehandgestures have also been explored in specific domainssuch as online shopping. For example, Badju and Lund-berg (2015) developed a system in which 10 gestureswere used to perform such tasks as item manipulationand menu navigation. Similarly, Altarteer et al. (2017)designed 5 freehand gestures to help consumers interactwith a hand-gestural VR interface for luxury brandonline stores. According to Badju and Altarteer et al.,freehand-gesture-based interaction can substantiallyimprove customers’ shopping experiences by enablingthem to try on new clothes or mix and match acces-sories without being physically present in a real shop-ping mall.
Recently, immersive virtual reality combined withnew sensors (e.g. Kinect and Leap Motion) and free-hand-gesture-based interaction technologies began arevolutionary shift away from the traditional WIMP-based desktop virtual environment (Badju and Lundberg2015; Altarteer et al. 2017) toward a more intuitive andnatural interaction paradigm for online shopping. Ver-hulst et al. (2016), for example, developed an immersivevirtual supermarket system in which patients with cogni-tive impairments were asked to collect various items witha shopping cart by using different interaction techniques.Their results demonstrate that human body gestures aremore natural and enjoyable than the traditional game-pad, while the latter is more efficient in terms of taskcompletion time.
2.2. Elicitation studies on gesture-basedinteraction in HCI
Although freehand gesture interaction has becomeincreasingly popular in recent years, there remains alack of general design guidelines and established con-ventions. Many of the gestures that are discussedabove were designed by professional HCI researchersand arbitrarily associated with corresponding targetfunctionalities (Yee 2009). End-users have little oppor-tunity to participate in the design process. In addition,
BEHAVIOUR & INFORMATION TECHNOLOGY 727
once the gestural system has been developed, end-usersusually have little opportunity to design personalisedgestures according to their own preferences. Therefore,such systems may not be able to recognise gestures thatend-users naturally perform and, consequently, face arisk of poor system usability and low user acceptance.To address the problem that is raised above, researchersproposed including an elicitation study technique ingesture design.
Figure 1 shows a typical procedure of traditional eli-citation studies for deriving a suitable gesture for aspecified target task. First, end-users are individuallyshown the desired visual effect of a specified targettask in the intended system, which is also called a refer-ent, and are asked to design a freehand gesture forachieving that effect. Then, system designers collect allgesture proposals from end-users and adopt the fre-quency ratio method to assign the gesture with thehighest frequency (top gesture) to the correspondingtarget task. Recently, elicitation studies have been widelyapplied to such emerging HCI devices and applicationsas surface computing (Wobbrock, Morris, and Wilson2009; Kurdyukova, Redlin, and André 2012; Buchanan,Bourke Floyd IV, and Holderness 2013; Grijincu,Nacenta, and Kristensson 2014; Valdes et al. 2014),mobile interaction (Kray, Nesbitt, and Rohs 2010;Ruiz, Li, and Lank 2011; Seyed et al. 2012; Chan et al.2016), virtual/augmented reality (Piumsomboon et al.2013; Connell, Kuo, and Piper 2013; Lee et al. 2015),large displays (Morris 2012; Nebeling et al. 2014; Roveloet al. 2014; Lou et al. 2018), ubiquitous computing(Chen et al. 2018), wearable devices (Tung et al. 2015;Shimon et al. 2016; Gheran, Vanderdonckt, and Vatavu2018), in-vehicle information systems (Döring et al.2011; Angelini et al. 2014), a humanoid robot (Obaidet al. 2012), and smart-home applications (Locken
et al. 2012; Kühnel et al. 2011; Vatavu 2012; Zaiţi, Pen-tiuc, and Vatavu 2015; Dong et al. 2015; Wu, Wang,and Zhang 2016).
Via such an approach, system designers can gain abetter understanding of the laws of end-users’ prefer-ences and personalised behaviours and the types ofgestures that can match end-users’ mental models.Moris et al. (2010), for example, compared user-defined and researcher-authored surface gestures andfound that user-defined gestures are more acceptablethan those that are created solely by professional sys-tem designers. Similarly, the findings by Nacentaet al. (2013) also demonstrated that user-defined ges-tures are easier to remember and learn and more inter-esting to use compared with predesigned gestures bysystem developers.
2.3. Limitations of current elicitation studies ongesture design
Despite the rapid proliferation of user elicitation studiesin recent years, there are potential pitfalls in practice(Figure 1): First, participants may not always recall abest gesture for a specified target task due to the limitedtime and experimental conditions. Furthermore, eventhe proposed gestures are often biased by participants’own preferences or experiences with prior user inter-faces, such as WIMP interfaces or touch-based userinterfaces (Morris et al. 2014). Recently, several research-ers (Morris et al. 2014; Chan et al. 2016; Hoff, Hornecker,and Bertel 2016) applied the priming and productiontechniques to offset legacy bias. In their method, partici-pants were required to design at least three gestures for aspecified target task. This method is effective, but hasvarious limitations. For example, Chan et al. (2016)claimed that some participants benefited little from it.
Figure 1. Procedure of traditional elicitation studies for deriving a top gesture for a specified target task.
728 H. WU ET AL.
One participant said that he already had a gesture inmind; hence, designing three gestures at a time madehim start grasping for straws.
Second, traditional elicitation studies often sufferfrom the gesture disagreement problem (Wu et al.2018). According to Wu et al. (2018), the chance thatusers will produce the same gesture for a task is below0.355 on average across three application domains ‒Car, VR, and TV. Therefore, it is unrealistic to expectthat participants can intuitively design the same gesturefor a specified target task. To address the gesture dis-agreement problem, Chen et al. (2018) and Wu et al.(2018) proposed offering a list of gesture choices forusers to choose in traditional elicitation studies. Afterseeing the offered set of gesture candidates, especiallygestures that are highly teachable, participants may bemore likely to change their minds and use them withthe benefits of hindsight. As a result, the disagreementproblem can be mitigated and the consensus among par-ticipants can be improved.
Third, in traditional elicitation studies, systemdesigners often use the ‘winner take all’ strategy to assignthe top gestures to corresponding target tasks. However,this method does not necessarily guarantee the popular-ity of the top gestures. According to Choi et al. (2014),66% of the top gestures needed to be altered in a posthoc evaluation of the resulting gesture set from the stan-dard elicitation study. Therefore, traditional elicitationstudies may often become trapped in local minima andfail to uncover gestures that may be better suited forspecified target tasks.
In summary, to design freehand-gesture-based inter-active systems that are more natural and user-friendly,we must consider the problems that are mentionedabove, explore how to get end-users more involved inthe design process, and analyse their outcomes (i.e. par-ticipants’ proposed gestures) more comprehensively ingesture elicitation studies. In this paper, we propose amore practical method for deriving more reliable ges-tures compared to the traditional gesture elicitationmethod. Our hypothesis is that deriving two gesturesin the a priori stage and selecting the top-two gesturesin the a posteriori stage of a traditional elicitationstudy may help mitigate the gesture disagreement pro-blem and offset end-users’ legacy biases.
3. Methodology
Two experiments were conducted to identify the afore-mentioned hypothesis. In the first experiment, we eli-cited freehand gestures from participants withoutlimiting them in terms of the gestures that they couldchoose. The derived gesture candidates were collected
and used as the basis for participants to choose in thesecond experiment, which aimed at investigatingwhether deriving two gestures in the first stage andselecting the top-two gestures in the second stage foreach referent can help alleviate the gesture disagreementproblem and offset the user’s legacy biases.
3.1. Requirement analysis and function definition
To develop a user-friendly VR shopping system, it isimportant to have a clear understanding of the usagecontext of the intended system (Ritter, Baxter, andChurchill 2014; Wu, Wang, and Zhang 2016), namely,we should identify the end-users who will use the finalsystem, the conditions under which they will use it,and for what they will use it.
To determine the set of most-necessary core tasks, wecollected interaction tasks for VR shopping applicationsfrom various e-commerce platforms. In addition, weinvestigated previous research on gesture-based VRapplications. Then, we conducted a brainstorming ses-sion in which 10 master shoppers were invited to discussthe common tasks for such a system. The aim of thebrainstorming session was to refine and validate the tar-get tasks and identify the problems that might arise inthe subsequent experiments. The session was conductedfor approximately 4 h in a usability lab. By collecting andgrouping functions that were proposed in previousstudies (Cardoso et al. 2006; Klinger et al. 2006; Josmanet al. 2009; Rand, Weiss, and Katz 2009; Carelli et al.2009; Hadad 2012; Josman et al. 2014; Badju and Lund-berg 2015; Verhulst et al. 2016; Altarteer et al. 2017), wegenerated a set of 22 common functions for a freehand-gesture-based VR shopping system.
Next, we recruited 32 participants (14 males and 18females) to a semi-structured interview. Their ageswere between 25 and 48 (M = 30.13, SD = 2.518). Theycame from various professions and included program-mers, marketers, housewives, and students and pro-fessors from a university. All participants had at least 5years of online shopping experience. During the inter-view, the 22 collected tasks were used as the basic infor-mation and participants were asked the followingquestions:
. What are the most-necessary core tasks in a freehand-gesture-based VR shopping system?
. Which tasks are appropriate to perform by using free-hand gestures?
. What are the advantages and disadvantages of usingfreehand gestures in such a VR shopping system? and
. What is the maximum number of freehand gesturesthat a VR shopping system should support?
BEHAVIOUR & INFORMATION TECHNOLOGY 729
According to the results that were collected from thesemi-structured review, the expected maximum numberof freehand gestures that can be used in a VR shoppingsystem is between 6 and 15 (M = 10, SD = 2.94). As aresult, we obtained the 12 most-necessary core tasks, asshown in Table 1, in which they are ranked accordingto popularity. A core task was considered only if atleast 50% participants chose it.
Table 1 lists these 12 most-necessary core tasks. Thetwo left columns of the table identify each task with asequential number and a task name. The two right col-umns indicate how popular each task was amongparticipants.
In this section, we collected requirements from actualend-users and determined the most-necessary tasks thata freehand-gesture-based VR shopping system shouldprovide. The results of this section laid the foundationfor our subsequent experimental studies.
3.2. Experiment 1
In this experiment, we collected further informationabout the most-necessary core tasks for VR shoppingenvironments that are derived from the previous sectionand asked participants to design two freehand gesturesfreely for each task without any hints. Our focus wason understanding the preferred gesture behaviours ofparticipants for various VR shopping tasks and applyingthe findings to inform freehand-gesture-based VR shop-ping interface design.
3.2.1. ParticipantsWe recruited 32 participants with an equal male-femaleratio for our experiment. Their ages were between 23 and41 (M = 27.94, SD = 1.390). They came from various pro-fessions and included programmers, marketers, house-wives, and students and professors from a university.All participants had more than 5 years of online shop-ping experience. However, none of them had any priorexperience with freehand-gesture-based interactive tech-nologies for VR shopping applications.
3.2.2. ApparatusThis experiment was conducted in a usability lab. Thetest environment had a PC, a Leap Motion sensor, andan HTC Vive (Figure 2). It also hosted a system thatwe developed in advance for processing freehand gestureinputs from the Leap Motion sensor and delivering VRscenarios to the HTC display.
3.2.3. ProcedureBefore the experiment, all 32 participants were brieflyinformed of the objective of the experiment and com-pleted a consent process. During the experiment, theywere told to use the proposed gesture-based VR shop-ping system to perform the 12 target tasks that are listedin Table 1.
Different from previous standard elicitation studiesthat required participants to design only a single gesturefor a specified target task (Nielsen et al. 2004; Kray, Nes-bitt, and Rohs 2010; Ruiz, Li, and Lank 2011; Kühnelet al. 2011; Locken et al. 2012; Kurdyukova, Redlin,and André 2012; Vatavu 2012; Piumsomboon et al.2013; Grijincu, Nacenta, and Kristensson 2014; Wu,Wang, and Zhang 2016; Zaiţi, Pentiuc, and Vatavu2015), we asked participants to design two gestures foreach target task after they heard the interactive instruc-tions from the experimenter, who was sitting behindthem in this experiment.
Table 1. Desired functionalities.No. Task name Frequency Percentage
1 Select an object 32 100%2 Rotate an object 32 100%3 Try on clothes 32 100%4 Previous colour 32 100%5 Next colour 32 100%6 Change to a smaller size 30 94%7 Change to a larger size 30 94%8 Enlarge an object 24 75%9 Shrink an object 24 75%10 View product details 24 75%11 Add to cart 22 69%12 Close the window 16 50%
Figure 2. Experiment 1 setup: (a) the proposed freehand-ges-ture-based VR shopping system and (b) the VR shoppingenvironment.
730 H. WU ET AL.
To counterbalance the possible order effects of the 12target tasks, we used a Latin square to determine theorder of tasks for each participant. To collect partici-pants’ design rationales, a ‘think-aloud’ method wasused in this experiment. All 32 participants wererequired to articulate why they chose a specific gesturefor a specified target task. To avoid the legacy bias pro-blem, we didn’t provide hints to participants. After allparticipants finished the test, we asked them to answera short questionnaire on their demographic data, includ-ing age, gender, and profession. The experiment lastedapproximately half an hour.
3.2.4. ResultsIn this section, we explain how we classified and groupeduser-defined gestures and present the agreement scoresamong gestures that were derived from participants.
3.2.4.1. Data processing.With 32 participants and 12 VRshopping tasks, we collected 768 (32 × 12 × 2) freehandgestures in total. After that, we invited 5 researcherswith expertise in gesture interaction and interface designto group and merge gestures for each target task in abrainstorming session. Two steps were conducted to cat-egorise the user-defined gesture set in the brainstormingsession: First, gestures that had the same shape and/ortrajectory were grouped into a single gesture. Second,for gestures with similar characteristics in terms ofshape and/or trajectory, the 5 researchers replayed thecorresponding video files that we collected during the eli-citation process and discussed whether and how to groupthem based on the verbal explanations (mental models)of participants in the experiment. As a result, weobtained 165 groups of identical gestures from the par-ticipants. Among them, 80 groups of identical gestureswere the first choices of participants and the remaining85 groups were the second choices of participants.
3.2.4.2. Agreement scores. Based on the collected user-derived gestures, we calculated the agreement score foreach target task via the updated consensus formula ofVatavu and Wobbrock (2015). The agreement score isdefined in Equation 1.
AR(r) = |P||P| − 1
∑Pi#P
|Pi||P|
( )2
− 1|P| − 1
(1)
where P is the set of all proposed gestures for task r, |P|the size of the set, and Pi subsets of identical gesturesfrom P. The agreement score was used to measure howlikely end-users are to choose the same gesture for aspecified task: the higher the agreement score is, the
more likely it is that the same gesture will be selected(Figure 3).
The average agreement scores of participants’ first-and second-choice gestures for the 12 VR interactivetasks are 0.181 (SD = 0.091) and 0.175 (SD = 0.089),respectively. According to the recommendation ofVatavu and Wobbrock (2015) for interpreting the mag-nitudes of agreement scores, the average agreementscores that were obtained in this study are very smallin magnitude. The results of our experiment demon-strate lower agreement score magnitudes comparedwith previous elicitation studies, such as the averageagreement scores of 0.221 for mobile interaction(Ruiz, Li, and Lank 2011), 0.242 for surface computing(Wobbrock, Morris, and Wilson 2009), 0.362 for TVcontrol (Vatavu 2012), 0.417 for augmented reality(Piumsomboon et al. 2013), and 0.430 for 3D objectmanipulation (Buchanan, Bourke Floyd IV, and Hol-derness 2013).
Using a matched-paired t-test, no significant differ-ence was found between the first- and second-choice ges-tures in terms of consensus by participants (t11 = 0.245, p= 0.811). Nevertheless, we found that the rankings of tar-get tasks correlated significantly with the first- andsecond-choice gestures (Spearman’s ρ(N = 12) = 0.725, p= 0.024 < 0.05).
For participants’ first-choice gestures, the highestagreement score, namely, 0.325, was obtained whendesigning gestures for Task 4: Shrink an object. Intotal, 3 freehand gestures were produced, of which thetop gesture, namely, Both hands moving from the outerleft and right to the centre middle, was chosen by 14 par-ticipants and the second-top gesture, namely, Perform apinch-in gesture with the thumb and index finger, by 12.In comparison, the lowest agreement score, namely,0.083, was obtained in designing gestures for Tasks 5and 6. In total, 10 freehand gestures were produced foreach task; however, the top gesture was chosen by only8 participants and the second-top gesture by only 6 ineach case.
For participants’ second-choice gestures, the highestagreement score, namely, 0.358, was obtained in design-ing gestures for Task 1: Select an object. In total, 4 free-hand gestures were produced by 32 participants, ofwhich the top gesture, namely, Grab, was chosen by 16participants and the second-top gesture, namely, Tapwith index finger, by 12. In comparison, the lowest agree-ment score, namely, 0.042, was obtained in designinggestures for Task 7: Try on clothes. In total, 12 freehandgestures were produced, of which the top gesture,namely, Drag onto one’s body, was chosen by only 6 par-ticipants and the second-top gesture, namely, Drag ontoan imaged mannequin, by only 4.
BEHAVIOUR & INFORMATION TECHNOLOGY 731
3.2.5. Discussion of experiment 1In this experiment, we asked participants to design twogestures for a specified target task without hints or phys-ical limitations. As discussed above, the results of thisexperiment demonstrated a lower consensus comparedto previous standard elicitation studies. Due to theserious gesture disagreement problem and the low con-sensus among participants, we could not determine anoptimal gesture set on which the majority of participantsagreed for interacting with the VR shopping application.Therefore, we conducted a post hoc evaluation in Exper-iment 2 in which all derived gestures were re-examinedand refined.
3.3. Experiment 2
Although we have learned about the most-necessary coretasks that a freehand-gesture-based VR shopping systemshould support and the commonly preferred freehand ges-tures by participants for those tasks, it remains unclearwhether those gestures would perform well in practicebecause of the open-ended nature of the elicitation studythat we used in the previous experiment. Therefore, in
this experiment,we aimedat further validating thepopular-ity and usability of the user-designed gestures. Specifically,we hope to validate the hypothesis that was stated above.
3.3.1. Participants and apparatusFor consistency, the same 32 participants who wereinvolved in the previous experiment were asked to par-ticipate in this experiment in the same usability lab asdescribed in Experiment 1. Compared to Experiment 1,in which participants were required to design two ges-tures for each VR interactive task, participants in thisexperiment were asked to choose two gestures for eachtarget task verbally. Therefore, participants were shownthe application scenarios and target tasks via text, pic-tures, and animated GIF images on PowerPoint slideson a laptop (Figure 4). Similar to Experiment 1, we didnot provide hints to participants.
3.3.2. ProcedureSimilar to the previous experiment, all 32 participantswere briefly informed of the experimental objective andrequirements and completed a consent process prior tothe experiment. Then, participants were shown a list of
Figure 3. Agreement scores of 12 target tasks in Experiment 1. The horizontal axis represents agreement score (AR). Tasks are orderedon the vertical axis in descending order of their agreement scores for the first-choice gestures.
732 H. WU ET AL.
the 12 target tasks for VR shopping and the 165 groupsof identical gestures that were derived from participantsin the previous experiment.
During the experiment, participants were presented aset of gesture candidates for each target task on Power-Point slides. Compared with previous elicitation studiesthat asked participants to choose only one gesture foreach task (Kühnel et al. 2011; Locken et al. 2012; Choiet al. 2014; Wu, Wang, and Zhang 2016; Chen et al.2018), we asked participants to choose two gestures foreach task in this experiment because we speculate thatvarious gestures that would be ignored by participantsdue to the legacy bias under standard elicitation studiesmay be revealed under the force of moving beyondsimple, legacy-inspired techniques in traditional userinterfaces. Similar to Experiment 1, a ‘think aloud’method was used to record why participants chose aspecific gesture for a specified interaction task. Theexperiment lasted between 20 and 40 min.
3.3.3. ResultsIn this experiment, we provided participants with 165groups of identical gestures that were collected fromExperiment 1, including 80 groups of first-choice ges-tures and 85 groups of second-choice gestures. Afterthis experiment, we collected 132 groups of identical ges-tures, including 61 groups of first-choice gestures and 71groups of second-choice gestures.
3.3.3.1. Agreement scores. Similar to the previous exper-iment, we calculated the agreement scores for the 12 VRinteractive tasks via the consensus formula of VatavuandWobbrock (2015). The results are shown in Figure 5.
The average agreement scores of participants’ first-and second-choice gestures for the 12 target tasks are0.267 (SD = 0.093) and 0.193 (SD = 0.096), respectively.Similar to Experiment 1, the average agreement scores
that were obtained in this experiment are very small inmagnitude.
Compared with Experiment 1, the first-choice ges-tures led to 38.3% higher agreement than the second-choice gestures. Using a matched-pair t-test, we founda significant difference between the first- and second-choice gestures (t11 = 2.617, p = 0.024). In addition, wefound that the rankings of the target tasks correlated sig-nificantly with the first- and second-choice gestures(Spearman’s ρ(N = 12) = 0.781, p = 0. 010 < 0.05).
For participants’ first-choice gestures, the highest agree-ment score, namely, 0.483, was obtained in designing ges-tures for Task 2: Rotate an object. In total, 4 freehandgestures were proposed by 32 participants, of which thetop gesture, namely, Twist, was chosen by 22 participantsand the second-top gesture, namely, Rotate an imagineddial, by 6. In comparison, the lowest agreement score,namely, 0.108, was obtained in designing gestures forTask 7: Try on clothes. In total, 8 freehand gestures wereproduced, of which the top gesture, namely, Drag ontoone’s body, was chosen by only 8 participants and thesecond-top gesture, namely, Pat one’s chest, by only 6.
For participants’ second-choice gestures, the highestagreement score, namely, 0.358, was obtained in design-ing gestures for Task 1: Select an object. In total, 4 free-hand gestures were produced, of which the top gesture,namely, Grab, was chosen by 16 participants and thesecond-top gesture, namely, Tap with index finger, by12. In comparison, the lowest agreement score, namely,0.075, was obtained in designing gestures for Tasks 8and 9. In total, 10 freehand gestures were produced byparticipants for each task; however, the top gesture waschosen by only 8 participants, and the second-top ges-ture by only 4 in each case.
3.3.3.2. Examination of participants who changed theirminds between two experiments. Next, we examinedthe number of participants who changed their mindson the first-choice gestures between two experiments.Figure 6 shows the percentage of participants who chan-ged their minds on the first-choice gesture for each taskafter seeing an offered set of gesture candidates that wereproposed by other participants. On average, 28.1% of theparticipants switched their first-choice gestures. Thisresult that almost one in three participants changedtheir first-choice gestures, which otherwise would besubsequently collected and sorted as the top gesturesfor the target tasks provides empirical evidence for theexistence of the legacy bias problem in freehand-ges-ture-based natural user interaction.
3.3.3.3. Examination of the top gestures between twoexperiments. According to Table 2, only 5 out of 12 top
Figure 4. Experiment 2 setup.
BEHAVIOUR & INFORMATION TECHNOLOGY 733
gestures (41.7%) remained after Experiment 2, includingGrab for Task 1 – Select an object, Twist for Task 2 –Rotate an object, Drag onto one’s body for Task 7 – Tryon clothes, Drag into an imagined shopping cart icon forTask 10 – Add to cart, and Swipe away for Task 12 –Close the window. In comparison, the remaining 7 topgestures for Tasks 3, 4, 5, 6, 8, 9, and 11 changed in Exper-iment 2. The change rate of the top gestures between the
two experiments is 58.3%. This is similar to the finding ofChoi et al. (2014): a 66% change rate of the top gesturebetween different experiments was observed.
Taking a closer look at the top gestures between twoexperiments, we find that the 5 top gestures for Tasks3, 4, 5, 6, and 11 that were derived from Experiment 1became the second-top gestures for tasks that wouldotherwise be excluded in Experiment 2 due to their low
Figure 5. Agreement scores of 12 target tasks in Experiment 2. The horizontal axis represents agreement score (AR). The tasks areordered on the vertical axis in descending order of their agreement scores for the first-choice gestures.
Figure 6. Numbers of participants who changed their minds on the first-choice gestures between two experiments.
734 H. WU ET AL.
frequencies. Interestingly, the 4 new top gestures forTasks 5, 6, 8, and 9 in Experiment 2 were proposed asthe second choices by participants in Experiment 1,while none of 32 participants had ever mentionedthem as first-choice gestures. This supports our hypoth-esis that requiring participants to design two gestures fora referent is necessary and can help reveal more mean-ingful results and, consequently, alleviate the legacybias problem in traditional gesture elicitation studies.
However, if we take the top 2 gestures into account foreach task in Experiment 2, except for Tasks 8 and 9, all thetop gestures that were derived in Experiment 1 areincluded in the top-2 gesture set in Experiment 2. There-fore, the change rate can be decreased to 16.7% from58.3%. Hence, 83.3% (10/12) will remain after Experiment2, although participants may change their minds on theirfirst-choice gestures. This finding supports our hypothesisthat selecting the top-two gestures for each referent may
help alleviate the gesture disagreement problem and, con-sequently, improve the consensus among participants.
3.3.3.4. User-defined gesture set. Using the gestures thatwere collected from participants in the previous twoexperiments, we generated a set of user-defined freehandgestures for VR shopping applications (Figure 7).
3.3.3.5. Examination of the social acceptance of the user-defined gesture set. Next, we examined the social accep-tance of the user-defined gestures from Experiment 2, i.e.whether the gestures were deemed to be appropriate byparticipants in the context in which they were carriedout (Montero et al. 2010). The average social acceptancerates of the top gestures and the second-top gestures are66.15% and 53.65%, respectively; hence, the user-definedgestures are accepted by more than half of the partici-pants. For the top gestures, the highest social acceptance,
Table 2. Top 2 user-defined gestures for the 12 target tasks between two experiments (ranked by first-choice gestures).
Target task
Experiment 1 Experiment 2
Top 2 gesturesFirst
choicesSecondchoices Totals Top 2 gestures
Firstchoices
Secondchoices Totals
1. Select anobject
Grab 14 16 30 Grab 14 16 30Tap with index finger 10 12 22 Tap with index finger 14 12 26
2. Rotate anobject
Twist 14 10 24 Twist 22 4 26Draw a circle with index finger 6 8 14 Draw a circle with index finger 2 14 16
3. Enlarge anobject
Perform a pinch-out gesturewith the thumb and indexfinger
14 10 24 Both hands moving from the centremiddle to the outer left and right
14 6 20
Both hands moving from thecentre middle to the outer leftand right
12 12 24 Perform a pinch-out gesture with thethumb and index finger
12 10 22
4. Shrink anobject
Perform a pinch-in gesture withthe thumb and index finger
14 10 24 Both hands moving from the outerleft and right to the centre middle
14 6 20
Both hands moving from theouter left and right to thecentre middle
12 12 24 Perform a pinch-in gesture with thethumb and index finger
12 10 22
5. Change to asmaller size
Swipe down 8 2 10 Tap on an imagined size panel 14 8 22Both hands moving from theouter left and right to thecentre middle
6 14 20 Swipe down 10 6 16
6. Change to alarger size
Swipe up 8 2 10 Tap on an imagined size panel 14 8 22Both hands moving from thecentre middle to the outer leftand right
6 14 20 Swipe up 10 6 16
7. Try onclothes
Drag onto one’s body 10 6 16 Drag onto one’s body 8 8 16Drag onto an imaginedmannequin
4 4 8 Pat one’s chest 6 10 16
8. Previouscolour
Swipe right 8 6 14 Tap on an imagined colour palette 12 4 16Index finger pointing left 2 8 10 Clockwise twist an imagined colour
wheel6 8 14
9. Next colour Swipe left 8 6 14 Tap on an imagined colour palette 12 4 16Index finger pointing right 2 8 10 Counterclockwise twist an imagined
colour wheel6 8 14
10. Add to cart Drag into an imagined shoppingcart icon
10 12 22 Drag into an imagined shopping carticon
14 10 24
Drag into one’s pocket 10 4 14 Drag into one’s pocket 8 4 1211. Viewproductdetails
Tap twice with index finger 10 2 12 Pull down an imagined curtain 10 6 16Index finger long pointing 3 s 8 10 18 Tap twice with index finger 8 16 24
12. Close thewindow
Swipe away 12 6 18 Swipe away 16 2 18Grab an imagined window andthrow it away
6 6 12 Pull up an imagined curtain 4 12 16
BEHAVIOUR & INFORMATION TECHNOLOGY 735
namely, 93.75%, was obtained in designing gestures forTask 1 – Select an object. In total, 30 participants selecteda Grab gesture for this task; among them, 14 participantschose it as the first-choice gesture and 16 participantschose it as the second-choice gesture. For the second-top gestures, the highest social acceptance, namely,81.25%, also was obtained in designing gestures forTask 1. In total, 26 participants selected a Tap withindex finger gesture for this task; among them, 14
participants chose it as the first-choice gesture and 12participants chose it as the second-choice gesture(Figure 8).
According to the column data in Table 2, the totalnumber of participants who chose the top-2 gestures astheir first choices for a specified task ranged from 14(Task 7) to 28 (Task 1) in Experiment 2 (M = 10.9, SD= 2.1) and from 12 (Tasks 8 and 9) to 28 (Task 1) forthe second-choice gesture (M = 8.3, SD = 2.3). Hence,
Figure 7. User-defined gesture set for VR shopping applications.
Figure 8. Examination of the social acceptance of the user-defined gestures.
736 H. WU ET AL.
the top-2 gesture set in Experiment 2 contains 70% of thefirst-choice gestures and 51.9% of the second-choice ges-tures of participants. Next, according to the row data inTable 2, the total number of participants who preferredto use one of the top-2 gestures for a specified task ran-ged from 12 (Task 10) to 30 (Task 1) in Experiment 2 (M= 9.6, SD = 2.3). Hence, on average, over 60% of end-users accept one of the top-2 gestures. If we take bothof the top-2 gestures into account, this number willincrease. This demonstrates relatively high social accep-tance of the derived top-2 gesture set in Experiment 2.
3.3.4. Discussion of experiment 2In this experiment, we provided a set of gesture candi-dates that were derived from Experiment 1 for partici-pants to choose for each target task. The experimentalresults support our hypotheses that (1) requiring partici-pants to design two gestures for each task can help offsetlegacy bias. This is manifested in the result that almostone in three participants changed their first-choice ges-tures and, consequently, 4 s-choice gestures that wereproposed in Experiment 1 were chosen as the new topgestures (see Tasks 5, 6, 8, and 9) in Experiment 2; and(2) selecting the top-2 gestures for each task may allevi-ate the gesture disagreement problems. This is supportedby the reduction in the number of groups of identicalgestures from 165 to 132, the lowering of the changerate of top gestures from 58.3% to 16.7%, the improve-ment of the average agreement scores from 0.181 to0.267 and 0.175 to 0.193 for first- and second-choice ges-tures, respectively, and the high social acceptance of thederived top-2 gesture set.
3.4. Mental model observations
In this section, we discuss how participants’ mentalmodels affect their design and choices of freehand ges-tures for VR shopping applications. Although we werecareful not to provide hints during the study, partici-pants often reasoned based on their prior interactionexperiences. Several characteristics of the participants’mental models were observed:
(1) The participants preferred to mimic interactions inreal-world scenarios in the gesture elicitationstudy. This could explain why the agreement scoresof target tasks 1–4 were always ranked higher thanthose of the other 8 target tasks in both experiments(see Figures 3 and 5).
(2) The participants were inclined to borrow gesturesthat had already been used in other designs ordomains, such as gestures in traditional WIMPinterfaces and touch-based interfaces, rather than
invent new gestures. The participants’ verbal expla-nations of their gesture design and choice rationalesduring the two experiment could be used as indi-cators of the underlying mental models, for example,‘I’d like to use a Pull down gesture for Task 11 ‒Viewproduct details just like I usually do to activate adrop-down list on my smart phone’. Other examplesinclude ‘Tap with index finger, just like a mouseclick on my computer’ for Task 1 ‒ Select an objectand ‘Tap twice with index finger, just like a doubleclick’ for Task 11 ‒ View product details.
(3) The participants preferred to take advantage of con-text-based 2D interactive components for tasks thatwere related to information navigation or retrievaland the interactive modes were analogous to thosethat are used in the traditional WIMP paradigm.For example, 22 participants used an index fingerto tap on an imagined size panel for Tasks 5 and 6in Experiment 2 (including the first- and second-choice gestures). Similarly, 16 participants used anindex finger to tap on an imagined colour palettefor Tasks 8 and 9.
(4) The participants imagined that they have magicpowers and superpowers when interacting with theimmersive virtual shopping mall; therefore, theycould take advantage of interactions beyond thereal world in the gesture elicitation study. Forexample, 16 participants used a Pat one’s chest ges-ture (including the first- and second-choice ges-tures) for Task 7 ‒ Try on clothes, with theunderlying mental model that the selected targetclothes would fly over them. Another example isthe designed gesture for Task 10 ‒ Add to cart: par-ticipants imagined that they could grasp anythingand put it into their pockets.
4. Discussion: implications for design
Based on the results that were obtained from the twoexperiments, we derived several guidelines for gesturedesign.
4.1. Taking advantage of legacy bias from prioruser interfaces and interaction techniques
Although participants in our experiment had no priorexperience using freehand gesture interaction in VRshopping applications, they showed a considerableamount of legacy bias from using traditional WIMPinterfaces or touch-based interfaces. Based on the resultsof the two experiments, the legacy bias had both positiveand negative effects on gesture design and the consensus
BEHAVIOUR & INFORMATION TECHNOLOGY 737
among participants. It showed positive effects on varioustasks, including tasks of virtual object manipulation (e.g.Tasks 1–4) and tasks that are strongly associated withdirection or order (e.g. Tasks 5 and 6). For these typesof target tasks, participants can use their familiar actionsin real-world scenarios directly or borrow gestures thathave already been widely used in other designs (e.g.touch-based user interfaces) rather than inventing newgestures. The clear association of these tasks with con-cepts, objects, or actions that participants have alreadyencountered may also make it easy for them to learnthe borrowed gestures quickly. In addition, legacy-inspired gestures are easier to guess and learn by partici-pants and easier to implement by system developers(Wobbrock et al. 2005). Therefore, system designersshould effectively take advantage of the positive effectsof the legacy bias in gesture design.
Our study also identified negative aspects of the legacybias. An example in which it negatively influenced con-sensus is Task 8 ‒ Previous colour. In total, 8 gestureswere proposed as the first-choice gestures by participantsin Experiment 1. Proposed gestures for this task includedTap on a colour-block on an imagined colour palette(WIMP paradigm), Clockwise twist an imagined colourwheel (physical knob in the real world), Swipe right(touch-based interface), and Swipe left (left arrow on aremote control). A similar situation arose for Task 9 ‒Next colour: participants drew upon a large variety ofprior legacy biases for such actions. In such situations,system designers may become confused regardingwhich gesture to choose as the most suitable one forthe target task. A feasible solution is to provide a toolkit(Wu, Wang, and Zhang 2016) that participants can useto design personalised gestures. As a result, the require-ments and preferences of various stakeholders’ can bebalanced.
4.2. Deriving two gestures for each referent mayalleviate the gesture disagreement problem andimprove consistency
Traditional elicitation studies suffer from the gesture dis-agreement problem (Wu et al. 2018) when deriving asingle gesture for each referent from end-users in the apriori stage and are apt to become trapped in localminima when the top gesture for each referent is beingselected by system designers in the a posteriori stage.Our study demonstrates that the average agreementscores of participants’ first-choice gestures in Exper-iment 1 and Experiment 2 are only 0.181 and 0.267,respectively. Statistical results demonstrate very smallmagnitudes via both agreement score analysis andKappa consistency testing. However, the situation will
be improved if we derive two gestures for each referentfrom end-users and select the top-2 gestures for eachreferent. The benefits are identified as follows:
(1) We observed a decrease in the number of identicalgesture groups after the second experiment. Partici-pants narrowed down their choices from 165 groupsin Experiment 1 to 132 groups in Experiment 2,which is a decrease of 20%. Specifically, the numberof first-choice gestures decreased from 80 to 61,which represents a 23.8% drop, and the second-choice gestures from 85 to 71, which represents a16.5% drop. As a result, the average agreementscore of the first-choice gestures increased to 0.267from 0.181 and the second from 0.171 to 0.193.
(2) In the second experiment, almost one in three par-ticipants changed their minds and resorted to choos-ing a new gesture for each target task after they wereoffered a list of gesture candidates that were col-lected in the first experiment. As a result, variousoutstanding or highly teachable gestures attractedincreasing attention from participants and it iseasy for system designers to determine the most suit-able gesture for a specified target task, for example,the Twist gesture for Task 2 ‒ Rotate an object.
(3) The change rate of the top gestures between the twoexperiments dropped from 58.3% to 16.7%. Ten outof the twelve top gestures that were produced inExperiment 1 were included in the top-2 gestureset in Experiment 2. Therefore, our method can alle-viate the problem of inconsistent outcomes by run-ning multiple elicitation studies.
(4) In addition to mitigating the legacy bias problem,requiring participants to design two gestures foreach task can also help designers learn more aboutuser acceptance, for example, to examine which ges-tures (e.g. the second-choice gestures) users are will-ing to expend effort to learn, in addition to the first-choice gestures, with which they are most familiar.Rather than the strict one-size-fits-all method thatis used in traditional elicitation studies to assignthe gesture with the highest frequency to a targettask, our method of selecting the top-2 gestures foreach task can help identify additional potentiallymeaningful gestures and, consequently, lead to ahigh level of social acceptance.
5. Conclusions and future work
Although freehandgestures have beenwidely used in a var-iety of applications, freehand-gesture-based interaction isstill in its infancy. There is a lack of general design
738 H. WU ET AL.
guidelines for determining the optimal gesture set for aspecified application scenario. To improve system usabil-ity, gesture elicitation has become a popular method forderiving intuitive and natural gestures from end-users fora specified set of system tasks in recent years. However,traditional gesture elicitation studies suffer from varioususability problems, such as vocabulary coherence (Pesh-kova et al. 2017(a)), gesture disagreement (Wu et al.2018) and legacy bias (Morris et al. 2014), and tend tobecome caught in local minima. Specifically, differentresults (different gestures) may result from different ges-ture elicitation studies that address the same referents;therefore, a methodological update of the way in whichelicitation studies are currently conducted might beneeded. This study addresses the problem of performinggesture elicitation studies and analysing their outcomes(i.e. participants’ proposed gestures) to optimize the setof recommend gestures.
The main contribution of this study lies in the 2-stepmethodology of deriving a set of more reliable user-defined gestures for VR shopping applications. We pro-vide empirical evidence for the benefits of designing twogestures for a referent in the a priori stage and selectingthe top gestures for each referent in the a posteriori stageof an elicitation study. Based on the experimental results,we also developed several design guidelines for freehand-gesture-based interfaces. We hope that our research willprovide valuable a reference for freehand-gesture-basedinterface design.
Similar to previous elicitation studies, this studyfocuses on understanding end-users’ mental models forgesture design, regardless of technology. An interestingstep for future work is to evaluate the recognition accu-racy and interaction efficiency of the resulting user-defined gesture set in a comparison experiment. Weare also interested in developing a prototype gestureinterface for VR shopping and examining such factorsas how freehand-gesture-based immersive VR inter-action affects customers’ presence and consequentlyaffects their shopping experiences in practice.
Acknowledgments
The authors would like to thank the anonymous reviewers fortheir insightful comments. This work was supported by theNational Natural Science Foundation of China under grantnumber 61772564, 61202344 and the funding offered by theChina Scholarship Council (CSC).
Disclosure statement
No potential conflict of interest was reported by the authors.
Funding
This work was supported by the National Natural ScienceFoundation of China under grant number 61772564,61202344 and the funding offered by the China ScholarshipCouncil (CSC).
ORCID
Huiyue Wu http://orcid.org/0000-0001-7027-518X
References
Alkemade, R., F. J. Verbeek, and S. G. Lukosch. 2017. “On theEfficiency of a VR Hand Gesture-Based Interface for 3DObject Manipulations in Conceptual Design.”International Journal of Human-Computer Interaction 33(11): 882–901.
Altarteer, S., V. Charissis, D. Harrison, and W. Chan. 2017.“Development and Heuristic Evaluation of Semi-Immersive Hand-Gestural Virtual Reality Interface forLuxury Brands Online Stores.” In AVR 2017, 464–477.
Angelini, L., F. Carrino, S. Carrino, M. Caon, O. A. Khaled, J.Baumgaartner, A. Sonderegger, D. Lalanne, and E.Mugellini. 2014. “Gesturing on the Steering Wheel: AUser-Elicited Taxonomy.” In AutomotiveUI’14, 1–8.
Badju, A., and D. Lundberg. 2015. “Shopping Using GestureDriven Interaction.” Master’s Thesis, Lund University, 1–105.
Buchanan, S., C. H. Bourke Floyd IV, and W. Holderness.2013. “Towards User-Defined Multi-Youch Gestures for3D Objects.” In ITS’13, 231–240.
Cai, Z. Y., J. G. Han, L. Li, and S. Ling. 2017. “RGB-D DatasetsUsing Microsoft Kinect or Similar Sensors: A Survey.”Multimedia Tools and Applications 76 (3): 4313–4355.
Cardoso, L., R. da Costa, A. Piovesana, M. Costa, L. Penna, A.Crispin, J. Carvalho, et al. 2006. “Using VirtualEnvironments for Stroke Rehabilitation.” In IWVR’06, 1–5.
Carelli, L., F. Morganti, B. Poletti, B. Corra, P. Weiss, R.Kizony, V. Silani, and G. Riva. 2009. “A Neurovr BasedTool for Cognitive Assessment and Rehabilitation of Post-Stroke Patients: Two Case Studies.” Studies in HealthTechnology and Informatics 144: 243–247.
Chan, E., T. Seyed, W. Stuerzlinger, X. D. Yang, and F. Maurer.2016. “User Elicitation on Single-Hand Microgestures.” InACM CHI’16, 3403–3411.
Chen, Z., X. C. Ma, Z. Y. Peng, Y. Zhou, M. G. Yao, Z. Ma, C.Wang, Z. F. Gao, and M. W. Shen. 2018. “User-definedGestures for Gestural Interaction: Extending From Handsto Other Body Parts.” International Journal of Human-Computer Interaction 34 (3): 238–250.
Cheng, H., L. Yang, and Z. C. Liu. 2016. “Survey on 3D HandGesture Recognition.” IEEE Transactions on Circuits andSystems for Video Technology 26 (9): 1659–1673.
Choi, E., S. Kwon, D. Lee, H. Lee, and M. K. Chung. 2014.“Towards Successful User Interaction with Systems:Focusing on User-Derived Gestures for Smart HomeSystems.” Applied Ergonomics 45: 1196–1207.
Colaco, A., A. Kirmani, H. S. Yang, N. W. Gong, C. Schmandt,and V. K. Goyal. 2013. “Mine: Compact, Low-Power 3D
BEHAVIOUR & INFORMATION TECHNOLOGY 739
Gesture Sensing Interaction with Head-Mounted Displays.”In ACM UIST’13, 227–236.
Connell, S., P. Y. Kuo, and A. M. Piper. 2013. “AWizard-of-OzElicitation Study Examining Child-Defined Gestures with aWhole-Body Interface.” In IDC’13, 277–280.
Dong, H. W., A. Danesh, N. Figueroa, and A. E. Saddik. 2015.“An Elicitation Study on Gesture Preferences andMemorability toward a Practical Hand-GestureVocabulary for Smart Televisions.” IEEE Access 3: 543–555.
Döring, T., D. Kern, P. Marshall, M. Pfeiffer, J. Schöning, V.Gruhn, and A. Schmidt. 2011. “Gestural Interaction onthe Steering Wheel – Reducing the Visual Demand.” InCHI’11, 483–492.
Feng, Z. Q., B. Yang, Y. Li, Y. W. Zheng, X. Y. Zhao, J. Q. Yin,and Q. F. Meng. 2013. “Real-Time Oriented Behavior-Driven 3D Freehand Tracking for Direct Interaction.”Pattern Recognition 46: 590–608.
Gheran, B. F., J. Vanderdonckt, and R. D. Vatavu. 2018.“Gestures for Smart Rings: Empirical Results, Insights,and Design Implications.” In DIS’18, 623–635.
Grijincu, D., M. A. Nacenta, and P. O. Kristensson. 2014.“User-Defined Interface Gestures: Dataset and Analysis.”In ITS’14. 25–34.
Hadad, S. Y. 2012. “Rehabilitation Tools along the RealityContinuum: From Mock-Up to Virtual InteractiveShopping to a Living Lab.” In ICDVRAT’12, 47–52.
Hoff, L., E. Hornecker, and S. Bertel. 2016. Modifying gestureelicitation: Do kinaesthetic priming and increased pro-duction reduce legacy bias? In TEI’16, 86–91.
Josman, N., R. Kizony, E. Hof, K. Goldenberg, P. Weiss, and E.Klinger. 2014. “Using the Virtual Action Planning-Supermarket for Evaluating Executive Functions in Peoplewith Stroke.” Journal of Stroke and CerebrovascularDiseases 23 (5): 879–887.
Josman, N., A. E. Schenirderman, E. Klinger, and E. Shevil.2009. “Using Virtual Reality to Evaluate ExecutiveFunctioning among Persons with Schizophrenia: AValidity Study.” Schizophrenia Research 115: 270–277.
Klinger, E., I. Chemin, S. Lebreton, and R. M. Marié. 2006.“Virtual Action Planning in Parkinson’s Disease: AControl Study.” CyberPsychology & Behavior 9: 342–347.
Kölsch, M., M. Turk, and T. Höllerer. 2004. “Vision-BasedInterfaces for Mobility.” In Mobile and UbiquitousSystems: Networking and Services (MOBIQUITOUS’04),86–94.
Kray, C., D. Nesbitt, and M. Rohs. 2010. “User-DefinedGestures for Connecting Mobile Phones, Public Displays,and Tabletops.” MobileHCI’10, 239–248.
Kühnel, C., T. Westermann, F. Hemmert, and S. Kratz. 2011.“I’m Home: Defining and Evaluating a Gesture Set forSmart-Home Control.” International Journal of Human-Computer Studies 69: 693–704.
Kulshreshth, A., and J. J. LaViola Jr. 2014. “Exploring theUsefulness of Finger-Based 3D Gesture Menu Selection.”In Proceedings of the ACM Conference on HumanFactors in Computing Systems (CHI’14), 1093–1102.
Kurdyukova, E., M. Redlin, and E. André. 2012. “StudyingUser-Defined iPad Gestures for Interaction in Multi-Display Environment.” IUI’12, 93–96.
Lee, G. A., J. S. Choi, J. Wong, C. J. Park, H. S. Park, and M.Billinghurst. 2015. “User Defined Gestures for AugmentedVirtual Mirrors a Guessability Study.” In CHI’15, 959-964.
Locken, A., T. Hesselmann, M. Pielot, N. Henze, and S. Boll.2012. “User-centred Process for the Definition of Free-Hand Gestures Applied to Controlling Music Playback.”Multimedia Systems 18 (1): 15–31.
Lou, X. L., R. Peng, P. Hansen, and X. D. A. Li. 2018. “Effects ofUser’s Hand Orientation and Spatial Movements on FreeHand Interactions with Large Displays.” InternationalJournal of Human-Computer Interaction 34 (6): 519–532.
Montero, C. S., J. Alexander, M. Marshall, and S. Subramanian.2010. “Would You Do That? – Understanding SocialAcceptance of Gestural Interfaces.” In MobileHCI, 275–278.
Morris, M. R. 2012. “Web on the Wall: Insights from aMultimodal Interaction Elicitation Study.” In ITS’12, 95–104.
Morris, M. R., A. Danielescu, S. Drucker, D. Fisher, B. Lee, M.C. Schraefel, and J. O. Wobbrock. 2014. “Reducing LegacyBias in Gesture Elicitation Studies.” Interactions 21 (3):40–45.
Morris, M. R., J. O. Wobbrock, and A. D. Wilson. 2010."Understanding Users’ Preferences for Surface Gestures."In GI ’10 Proceedings of Graphics Interface 2010 (261-268)
Nacenta, M. A., Y. Kamber, Y. Z. Qiang, and P. O. Kristensson.2013. "Memorability of Pre-designed & User-definedGesture Sets". In CHI ’13 Proceedings of the SIGCHIConference on Human Factors in Computing Systems(1099-1108)
Nebeling, M., A. Huber, D. Ott, and M. C. Norrie. 2014. “Webon the Wall Reloaded: Implementation, Replication andRefinement of User-Defined Interaction Sets.” In ITS’14,15–24.
Nielsen,M.,M. Störring, T.Moeslund, andE.Granum. 2004. “AProcedure for Developing Intuitive and Ergonomic GestureInterfaces for HCI.” Gesture-based Communication inHuman–Computer Interaction, 105–106.
Obaid, M., M. Haring, F. Kistler, R. Buhling, and E. Andre.2012. “User-Defined Body Gestures for NavigationalControl of a Humanoid Robot.” In ICSR’12, 367–377.
Peshkova, E., and M. Hitz. 2017(a). “Coherence Evaluation ofInput Vocabularies to Enhance Usability and UserExperience.” In Proceedings of the ACM SIGCHISymposium on Engineering Interactive ComputingSystems, 15–20
Peshkova, E., M. Hitz, D. Ahlström, R. W. Alexandrowicz, andA. Kopper. 2017(b). “Exploring Intuitiveness of Metaphor-Based Gestures for UAV Navigation.” In 26th IEEEInternational Symposium on Robot and Human InteractiveCommunication (RO-MAN), 175–182
Pfeil, P., S. Koh, and J. J. LaViola Jr. 2013. “Exploring 3DGesture Metaphors for Interaction with Unmanned AerialVehicles.” IUI’ 13, 257–266.
Piumsomboon, T., M. Billinghurst, A. Clark, and A. Cockburn.2013. “User-Defined Gestures for Augmented Reality.”CHI’13, 955–960.
Rand, D., P. T. Weiss, and N. Katz. 2009. “TrainingMultitasking in a Virtual Supermarket: A NovelIntervention After Stroke.” American Journal ofOccupational Therapy 63: 535–542.
Rautaray, S. S., and A. Agrawal. 2015. “Vision Based HandGesture Recognition for Human Computer Interaction: ASurvey.” Artificial Intelligence Review 43 (1): 1–54.
Ritter, F. E., G. D. Baxter, and E. F. Churchill. 2014.Foundations for Designing User-Centered Systems: What
740 H. WU ET AL.
System Designers Need to Know About People. London:Springer.
Rovelo, G., D. Vanacken, K. Luyten, F. Abad, and E. Camahort.2014. “Multi-Viewer Gesture-Based Interaction for Omni-Directional Video.” CHI’14, 4077–4086.
Ruiz, J., Y. Li, and E. Lank. 2011. “User-Defined MotionGestures for Mobile Interaction.” In ACM CHI’11, 197–206.
Seyed, T., C. Burns, M. C. Sousa, F. Maurer, and A. Tang. 2012.“Eliciting Usable Gestures for Multi-DisplayEnvironments.” In ITS’12, 41–50.
Sherstyuk, A., D. Vincent, J. J. H. Lui, and K. K. Connolly.2007. “Design and Development of a Pose-BasedCommand Language for Triage Training in VirtualReality.” In IEEE Symposium on 3D User Interfaces, 33–40.
Shimon, S. S. A., C. Lutton, Z. C. Xu, S. M. Smith, C. Boucher,and J. Ruiz. 2016. “Exploring Non-touchscreen Gestures forSmartwatches.” In CHI’16, 3822–3833.
Song, P., W. B. Goh, W. Hutama, C. W. Fu, and X. P. Liu.2012. “A Handle Bar Metaphor for Virtual ObjectManipulation with Mid-Air Interaction.” In ACM CHI’12,1297–1236.
Takahashi, M., M. Fujii, M. Naemura, and S. Satoh. 2013.“Human Gesture Recognition System for TV ViewingUsing Time-of-Flight Camera.” Multimedia Tools andApplications 62 (3): 761–783.
Tollmar, K., D. Demirdjian, and T. Darrell. 2004. “Navigatingin Virtual Environments Using a Vision-Based Interface.”InNordiCHI’04, 113–120.
Tung, Y. C., C. Y. Hsu, H. Y. Wang, S. Chyou, J. W. Lin, P. J.Wu, A. Valstar, and M. Y. Chen. 2015. “User-Defined GameInput for Smart Glasses in Public Space.” In ACM CHI’15,3327–3336.
Valdes, C., D. Eastman, C.Grote, S. Thatte, O. Shaer, A.Mazalek,B. Ullmer, and M. K. Konkel. 2014. “Exploring the DesignSpace of Gestural Interaction with Active Tokens throughUser-Defined Gestures.” CHI’14, 4107–4116.
Vatavu, R. D. 2012. “User-Defined Gestures for Free-Hand TVControl.” EuroITV’12, 45–48.
Vatavu, R. D., and J. O. Wobbrock. 2015. “FormalizingAgreement Analysis for Elicitation Studies: New Measures,Significance Test, and Toolkit.” In CHI’15, 1325–1334.
Verhulst, E., P. Richard, E. Richard, P. Allain, and P. Nolin.2016. “3D Interaction Techniques for Virtual Shopping:Design and Preliminary Study.” In VISIGRAPP’16, 271–279.
Wobbrock, J. O., H. H. Aung, B. Rothrock, and B. A. Myers.2005. “Maximizing the Guessability of Symbolic Input.” InCHI’05, 1869–1872.
Wobbrock, J. O., M. R. Morris, and A. D. Wilson. 2009. “User-Defined Gestures for Surface Computing.” CHI‘09, 1083–1092.
Wu, H. Y., J. M. Wang, and X. L. Zhang. 2016. “User-centeredGesture Development in TV Viewing Environment.”Multimedia Tools and Applications 75 (2): 733–760.
Wu, H. Y., S. K. Zhang, J. L. Qiu, J. Y. Liu, and X. L. Zhang.2018. “The Gesture Disagreement Problem in FreehandGesture Interaction.” International Journal of Human-Computer Interaction. doi:10.1080/10447318.2018.1510607.
Yang, C., Y. Jang, J. Beh, and D. Han. 2012. “GestureRecognition Using Depth-Based Hand Tracking forContactless Controller Application.” In IEEE InternationalConference on Consumer Electronics, 297–298.
Yang, H. D., A. Y. Park, and S. W. Lee. 2007. “Gesture Spottingand Recognition for Human-Robot Interaction.” IEEETransactions on Robotics 23 (2): 256–270.
Yee, W. 2009. “Potential Limitations of Multi-Touch GestureVocabulary: Differentiation, Adoption, Fatigue”.Proceeding of the 13th International Conference onHuman Computer Interaction, 291–300.
Zaiţi, I. A., S. G. Pentiuc, and R. D. Vatavu. 2015. “On Free-Hand TV Control: Experimental Results on User-ElicitedGestures with Leap Motion.” Personal and UbiquitousComputing 19: 821–838.
BEHAVIOUR & INFORMATION TECHNOLOGY 741
