GESTURE-BASED HEAD-MOUNTED AUGMENTED REALITY GAME

GESTURE-BASED HEAD-MOUNTED AUGMENTED

REALITY GAME DEVELOPMENT USING LEAP MOTION

AND USABILITY EVALUATION

Pu-Hsuan Chien and Yang-Cheng Lin Department of Industrial Design, National Cheng Kung University

No.1, University Road, Tainan, Taiwan

ABSTRACT

The environment of augmented reality created by head-mounted displays can provide players with an immersive gaming

experience and proper gestures can enable the integration of real-world objects with the virtual environment. This study

explores the usability of gesture operations applied to games and tries to find new applications using current mature gesture

sensors. This study develops a head-mounted augmented reality game based on gesture recognition, and evaluates their

usability and impact on the user’s interaction with the virtual interface. It uses the Unity engine and open-source glasses

designed by Leap Motion to develop an adventure game, and it contains elements of a first-person shooter game. There is

no need for any joystick or touch screen to control the game, as the game relies only on the player's hands to operate through

a controller-less interface. The results show that the system usability score is 79.4, which indicates a good usability level.

This study may be of importance in clarifying problems that may be encountered in the development of interactive-gesture

head-mounted display augmented reality games. Finally, this study offers suggestions for the challenges and future of

head-mounted augmented reality games.

KEYWORDS

Natural User Interface, Gesture, Leap Motion, Head-Mounted Displays, Augmented Reality Game

1. INTRODUCTION

Games are a daily pastime for many people, and various ways of playing can affect the mental state of players

(Shin, 2019). In the last decade, many human-computer interaction methods ranging from traditional keyboards

to motion sensors have provided new methods of interaction. As this technology advances, games can bring

players an immersive experience. Head-mounted displays (HMDs) can impart players with an immersive

gaming experience. The technology behind head-mounted virtual reality (VR) games is very mature, with

several devices based on them available in the market. However, HMD technology is not used as much in

augmented reality (AR) games — currently, most augmented reality (AR) games are on mobile platforms,

often requiring holding a mobile phone to play. In recent years, there has been an increasing research interest

on AR (Gugenheimer et al., 2019), and most of the research on the use of HMDs in AR games has focused on

either solving technical problems (Hua, 2017; Xiao & Benko, 2016) or on improving input factors (Xu et al.,

2019). In this study, a set of head-mounted display AR games that can be controlled with both hands has been

developed to bring players a new gaming experience.

There are many gesture recognition sensors available. Among them, Microsoft’s Kinect and Leap Motion

have much related research (Cabreira & Hwang, 2015, Vokorokos et al., 2016). Leap Motion allows users to

use gestures to control the operation and is currently widely used in sign language recognition, rehabilitation,

and interactive game development (Khademi et al., 2014, Potter et al., 2013). To realize the functions of gesture

recognition and AR at the same time, we use the AR HMD developed by Leap Motion in 2018 with

open-source glasses. Most of the components are 3D printed, so the price is lower than other head-mounted

AR glasses. This method broke through the original form of operation and increased the application of AR,

opening up more possibilities for human-computer interaction. Unlike the general Leap Motion operation

method, it places the sensor on top of the head so it can effectively recognize gestures when moving.

International Conferences Interfaces and Human Computer Interaction 2021; and Game and Entertainment Technologies 2021

149

To study the potential of this device, we developed an AR game, considering the player’s operation process

and the game interface. The major purpose of this study was to assess the usability of this kind of prototyping

game, so participants were asked to rate its usability. Finally, we put forward some problems that may be

encountered in the development of this type of game.

2. RELATED WORKS

The game development in this study involves many areas, such as the choice of gesture sensors, how the

gestures are used in games, and the game interface design. We consider that the application of gestures in

current gesture-based game operation has not been fully studied, so we have limited literature to consider in

developing games.

2.1 Gesture Sensor

To date, there have been many studies on the application of gesture sensors. Previous studies have also

compared the advantages and disadvantages of these sensors. Some researchers have used Leap Motion in

various fields. For example, Cabreira and Hwang conducted experiments on three sensors and analyzed 250

applications, of which more than half were game programs. Their analysis of 15 gestures on each platform

indicated that Kinect is more limited than other devices for finger and hand tracking (Cabreira & Hwang, 2015).

Leap Motion recognized a wider range of gestures. The most commonly used gestures for Myo were rotation

and waving, which could be used consistently in the various programs. Vokorokos et al. (2016) also conducted

experiments on three sensors in their research, using pointing, waving, rotating, and fist to determine the gesture

accuracy of the devices. Many tests were conducted on the sensors of the three platforms for each movement.

Kinect performed well in waving, while Leap Motion performed well in pointing, rotating, and punching.

It was fast and intuitive, but the sensing distance was limited. Myo was also very good in rotation, with the

advantage that the user can put their hands down to feel more comfortable. Because it uses muscle detection,

motion can be easily detected even if the hands are covered or the device is placed in a pocket. Based on the

above research, it can be determined that Kinect is very smooth under large-scale gesture sensing such as

waving hands. However, other gestures are not very accurate. Its advantages are that it can detect the skeleton

of the face and the whole body and that the detection range is wider. Leap Motion's research on AR applications

is far more advanced than those of other manufacturers. (Katahira & Soga, 2015; Wozniak et al., 2016). Leap

Motion provides open-source AR HMD glasses, the sensor can be placed in a head-mounted device, which

solves the original distance limitation of the sensing, but no current research on these glasses has yet appeared.

Based on the above research and viewpoints, and because of the good accuracy of Leap Motion and the

availability for purchase, this study uses Leap Motion as the sensing device for the AR glasses. Although the

sensing range is somewhat limited, changing the Leap Motion device above the HMDs can effectively address

this question.

2.2 Gesture Interaction Applied to Games

Existing research on the application of gestures in games, based on their recognition method, can be broadly

categorized into two types: visual recognition and motion sensor recognition. Visual recognition primarily

relies on Kinect, Leap Motion, and mobile phone lenses for gesture analysis (Pirker et al., 2017; Yeo et al.,

2015), while motion sensor recognition uses six-axis sensors or electromyographic signals and usually requires

the user to wear additional devices (Esfahlani et al., 2018; Lee et al., 2017).Regarding the research on the use

of gesture interactions in games, Silpasuwanchai and Ren (2015) held that in gesture interaction games, not all

operational methods are suitable for all players. Therefore, they used user feedback and user-elicitation to get

a clear understanding of the actions the players felt were most intuitive and found that various people

experience the game differently, and there were also differences in their actions, but there were some gestures

that players agreed on. For example, the preferred gesture for shooting was the gesture of using one hand to

draw a pistol.

ISBN: 978-989-8704-31-3 © 2021

150

There have been several studies that have investigated the application of Leap Motion in medical,

entertainment, and educational fields, such as two-handed drone operation (Fernandez et al., 2016) and hand

rehabilitation exercises (Alimanova et al., 2017). The above points out the potential of Leap Motion in the

natural user interface. However, little research has been done on the development of in-game applications.

Overall, we believe that it has great potential in gaming and entertainment, and we use Leap Motion with AR

HMDs to design and develop a game, which will help researchers study the application of human-computer

interaction in related game fields in the future.

2.3 Diegetic Interface

When only gestures are used, the interface plays a primary role in games. Previous studies have also pointed

out that the design of the interface affects the player’s immersion (Brown & Cairns, 2004, Qin et al., 2009).

The word “diegesis” was originally a term used in movies to refer to all the stories about the character.

Galloway applied this concept to the role of the game. The operation of the role of the game is regarded as

diegesis, and non-diegesis refers to the game outside the narrative world. He also distinguished between the

behavior of the player and the character. The behavior of the player is divided into diegetic and non-diegetic

elements, so according to the corresponding interface of its behavior classification, the heads-up display (HUD)

can be used. The HUD is mainly for players to see, so it is an interface superimposed on the game screen

(Galloway, 2006). Later writers have voiced the same view. Peacocke et al. studied the user interface of a

first-person shooter game based on Galloway’s theory. They compared the player’s preferences and immersion

between the diegetic and non-diegetic interfaces based on the display of the remaining ammunition, using a

5-point Likert scale to evaluate their immersion. He concluded that the digital display interface works best in

the diegetic interface (Peacocke et al., 2015). Fagerholt and Lorentzon’s theory offered a sounder theoretical

basis for interface design. They analyzed and classified first-person design game interfaces, constructed some

appropriate professional terms, and provided design guidelines to increase game immersion. However, they

deemed that to complete these design guidelines, it is still necessary to have a game prototype design for

evaluation (Fagerholt & Lorentzon, 2009). Because this study’s game is not a pure first-person shooter game,

we primarily divided the interface into the diegetic interface and non- diegetic interface and applied it to our

design theory.

3. GAME DESIGN AND DEVELOPMENT

Because this study uses HMDs, we think it is very suitable for first-person games. To allow players to be more

integrated into the game, we have created some stories and relatable characters, so it can be regarded as an

adventure game with first-person shooting elements. In this study, the player is acting as an adventurer and

must help the good dinosaurs defeat the evil dinosaurs within the time allotted for the mission.

The task of the game is to defeat the subordinates of the Demon Ass Dragon King. We established three

subordinates with different attributes. The player must switch to an attack with the relative attribute to damage

to the target. At the same time, the player must pay attention to supplement their magic value, or they will be

unable to attack. Play continues in this level until either the time expires or all monsters are defeated, at which

point the game ends.

The gameplay is different from typical games. Because there is no controller, all operations must be done

by hand. Operation of the game is as follows:

－Confirm/Next step: thumb up, right hand (Figure 1a).

－Shoot: make a pistol gesture with right hand (Figure 1b).

－Select: finger toggle button interface (Figure 1c).


151

(a) (b) (c)

Figure 1. (a) Confirm, (b) Shoot, (c) Select the menu

According to Galloway’s theory, interfaces can be divided into diegetic and non-diegetic interfaces for

characters and players. However, there is no relevant research on the interface for first-person head-mounted

AR games. Therefore, this study adopted expert interviews and semi-structured interviews. We used expert

interviews to understand some professional knowledge of game interfaces to be appropriate for the player’s

operation.

We interviewed Associate Professor Hu Minjun, an expert in the fields of computer vision and AR/VR.

The main questions that the interview was intended to clarify were (1) the current interface problems of HMD

games, (2) The way in which players operate games and the priority of visual communication, and (3) the

advantages and disadvantages between the diegetic and non-diegetic interfaces and the application suggestions.

After compiling the data from the interview, the following conclusions were drawn. First, we should note

the detection sensitivity. It is recommended not to use the most complex gestures in AR or VR. Second, visual

communication is more important than the method of operation because a good design can be used to

communicate the interaction and make up for the player’s lack of experience. Finally, the diegetic interface

gives players higher interactivity. If it is desired that the player be a first-person character experiencing the

game, the diegetic interface is very important, but the non- diegetic interface is used to assist it, and the two

should be used in conjunction with the situation. Therefore, we decided to use the diegetic interface as the main

interface first to minimize the HUD on the screen.

The interface plan is as follows:

－Time: diegetic interface (Figure 2).

－Health bar and mana: diegetic interface (Figure 2).

－Menu: diegetic interface (Figure 1c).

Figure 2. Health bar, mana, and time interface

Thus, the left hand is mainly to display the interface, and the right hand is used for attacking and selecting.

After the game starts, a non-player character guides the player into the game, through the story of the game,

and to the operation instructions. The thumb is used to attack the monster, and the designated button is used to

switch the attack attribute. When the mana is exhausted, the mana must be replenished as soon as possible, or

the player will be unable to attack. They need to complete the task within the allotted time, and completing the

task sooner will lead to a higher score.

ISBN: 978-989-8704-31-3 © 2021

152

Because we use it with HMDs and Leap Motion, we must first import some software development kit files

into Unity. We referred to Takahashi’s article to build the game environment, which includes hand calibration

and adjustment of the position of the glasses window. This is to allow the player to match the virtual hand in

Unity with the actual hand position during the game.

The game also needs related materials such as characters and interfaces. The 3D character model, its

skeleton, and animation were all created in Blender, and the material and rendering were done in Unity. The

special effects were drawn using Illustrator, with post-production in Photoshop, and the particle special effects

were done in Unity. Finally, after the program is created, the game detects the player’s hands, with the left

hand for the panel, and the right hand for attacking. We use Leap Motion to detect the skeleton of the hand;

when the distance between the thumb and the tip of the index finger reaches a certain value, an attack is

launched.

Figure 3. Game development screen: Thunder attribute attacks monsters

4. METHOD

This research is a relatively novel way of gaming, and Leap Motion’s input method is very different from

traditional operation. Most people have not used this type of input device. To understand the usability of the

device in the future and how players feel about the game. We evaluated the ease of use based on the system

usability score (SUS) and conducted semi-structured interviews to analyze subjective player feedback.

4.1 Participants

We recruited eight participants (four males and four females) ranging in age from 24 to 31 years old (M = 24.8,

SD = 1.6). All were right-handed. Regarding their gaming experience, all participants had experience with VR

games, five had experience with AR games, and all were in the habit of playing games. However, no one had

ever played a game using gesture controls.

4.2 Apparatus and Material

This research used the AR HMD Project North Star kit released by Leap Motion in 2018 (Figure 4). The helmet

was 3D printed and equipped with special polycarbonate lenses and two screens with a resolution of

1440 × 1600 pixels and a frame rate of 120 FPS. The screen image is reflected on the lens so the user can see

the real and virtual images at the same time. In addition, we combined headphones with the HMD so players

could hear sounds in the game. The game was developed using Unity rather than modifying an existing game.

A new game was chosen because existing games usually required a controller to operate and could not be used

on our equipment. The interface also needed to be developed by itself. The interface was designed based on

the results of the interview and with reference to the videos released by Leap Motion.


153

Figure 4. Leap Motion AR HMD with Project North Star kit

4.3 Procedure

The participants selected were instructed to play the game to the best of their ability. Because there were

instructions in the game, we did not teach the subjects separately. This was to allow players to play the game

smoothly just by watching the instructions. After the game started, NPCs explained the plot, operation methods,

and tasks to the players. Monsters had different attributes, and players needed to attack them with

corresponding attributes. All players had eight points of mana, and each attack consumed one point. After the

mana was exhausted, the player needed to click on the interface to replenish the mana. The game was over

after they killed all the monsters. After completing the task, the players filled in the SUS to evaluate the

usability of the game subjectively, and we conducted semi-structured interviews to obtain more in-depth and

specific suggestions.

Figure 5. The process of players operating the game

5. RESULTS AND DISCUSSION

The participants were asked to fill out the SUS. The SUS score is divided into six levels: A, 90–100 points;

B, 80–89 points; C, 70–79 points; D, 60–69 points; and F, 0–59 points. A qualified system should be higher

than 68 points. The SUS was originally intended to evaluate system usability, but it can also be used to

evaluated websites and games. Therefore, the word “system” in the questions was replaced by “game.” The

results show that our game SUS average score was 79.4 points, which falls in the C grade, which means that

the usability qualified. The questions were as follows, of which 2, 4, 6, 8, and 10 are reverse questions:

ISBN: 978-989-8704-31-3 © 2021

154

Table 1. SUS Questions and Score

Question Score STDev

1. I think that I would like to play this game frequently.

2. I found the game unnecessarily complex.

3. I thought the game was easy to use.

4. I think that I would need the support of a technical person to be able to play this game.

5. I found the various functions in this game were well integrated.

6. I thought there was too much inconsistency in this game.

7. I would imagine that most people would learn to play this game very quickly.

8. I found the game very cumbersome to play.

9. I felt very confident playing the game.

10. I needed to learn a lot of things before I could get going with this game.

8.4

6.9

7.8

7.2

7.2

9.1

8.8

8.1

6.6

9.4

0.52

0.72

0.64

0.83

0.64

0.52

0.53

0.89

0.74

0.46

The weighted means of each question are presented in Figure 6. All players thought that the rules of the

game were easy and they did not need to learn many things. Question 9 had the lowest score, which may be

due to players sometimes being inaccurate, which would lower their self-confidence.

In the post-experiment interviews, the responses were generally positive. This study yielded the following

results: Some players thought it was very interesting because they could see their hands operating the game.

Four subjects said they felt immersed and felt that the characters in the game seemed to exist in the real world,

and they could be even more integrated into the game than in VR games. The game instructions were simple

and easy to understand, which helped players enter the game. It is worth mentioning that the sound effects

played a helpful role in the game. Most players thought that the sound effects gave them significant feedback.

If they lacked these sound effects, they would not be able to control the game, and the music brought about

emotional changes, which were also indispensable.

However, our game is not without problems. Due to the limitation of Leap Motion’s detection range,

gestures may sometimes not be detected. Various angles should be attempted for detection. Regarding the

interface, some players suggested that the button interface should be larger and that a gap between the buttons

would make them easier to operate and reduce the chance of accidentally touching other buttons. In addition,

participants noted that the mana had to be filled too frequently. I interpret this to mean that the participants

probably did not like frequent operations.

Based on the information from previous interviews and the literature, we chose to use the diegetic interface.

The advantage is that it allows players to become more integrated into the game, but the disadvantage is that

some interfaces cannot be displayed immediately. Some players liked this interface because they thought it

made the screen concise, but it also required frequent operation of the interface. Some players also pointed out

that they preferred the HUD interface, that is, the non-diegetic interface, which helped them understand the

required information in real time, although it occupied part of the layout. It can be seen that players have

different preferences for the interface. The players’ immersion and preference for the interface need additional

study. Overall, we think this project is a success. Although there are some restrictions regarding sensor issues

that need to be resolved, all the players had a good gaming experience. Finally, we believe that the interface

should be able to successfully assist players. If players can fully master the game, their immersion will increase.

Therefore, future goals should focus on these projects.

6. CONCLUSION

Previous studies on the application of Leap Motion to games mostly focus on object manipulation, but this

research uses Leap Motion in the game mode of HMDs. There has thus far been relatively little research in this

area. This research adds many interface aids, such as buttons and the interface attached to the hand, which can

provide help for the development of gesture-related interactive games in the future. Our results show that even

without operating a controller, a player can still control the game and have an interesting gaming experience.

Players have indicated that this method of operation is very new. The lack of precision of the operation, if not

complete control of the game, will result in the loss of the player's sense of immersion. Thus, this input method

may be more suitable for the type of game that does not require precise operation rather than a competitive

type of game. Although this study has its limitations, this game can serve as a basis for relevant head-mounted

gesture-based game development.


155

We think there are two directions for development in the future: First, additional location trackers should

be added to ensure that every gesture of the player can be detected. We also think that if gestures can achieve

more precise operations, the game will have broader development opportunities, not limited to simple

minigames but also more gesture operations can be developed. Second, we plan to do further research on the

game interface and test diegetic and non-diegetic interfaces, including displaying the player’s health bar, menu,

time, and remaining ammo/mana, to understand the player’s preference for the interface and create an interface

suitable for HMD AR games.

REFERENCES

Alimanova, M. et al., 2017. Gamification of hand rehabilitation process using virtual reality tools: Using leap motion for

hand rehabilitation. In IEEE 2017 First International Conference on Robotic Computing (IRC). Taichung, Taiwan,

pp. 336-339.

Brown, E. and Cairns, P., 2004. A grounded investigation of game immersion. In CHI'04 extended abstracts on Human

factors in computing systems. Vienna, Austria, pp. 1297-1300.

Cabreira, A. T. and Hwang, F., 2015. An analysis of mid-air gestures used across three platforms. Proceedings of the 2015

British HCI Conference, Lincoln Lincolnshire, United Kingdom, pp. 257-258.

Esfahlani, S. S. et al., 2018. Validity of the Kinect and Myo armband in a serious game for assessing upper limb movement.

Entertainment Computing, 27, pp. 150-156.

Fagerholt, E. and Lorentzon, M., 2009. Beyond the HUD-user interfaces for increased player immersion in FPS games.

Master's thesis. Chalmers University of Technology, Göteborg, Sweden.

Fernandez, R. A. S. et al., 2016. Natural user interfaces for human-drone multi-modal interaction. In 2016 International

Conference on Unmanned Aircraft Systems (ICUAS), IEEE. Arlington, USA, pp. 1013-1022.

Galloway, A. R., 2006. Gaming: Essays on algorithmic culture (Vol. 18). U of Minnesota Press, Minnesota.

Gugenheimer, J. et al., 2019. Challenges using head-mounted displays in shared and social spaces. Paper presented at the

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems.

Hua, H., 2017. Enabling focus cues in head-mounted displays. Proceedings of the IEEE, 105(5), pp. 805-824.

Katahira, R. and Soga, M., 2015. Development and evaluation of a system for AR enabling realistic display of gripping

motions using Leap Motion controller. Procedia Computer Science, 60, pp. 1595-1603.

Khademi, M. et al., 2014. Free-hand interaction with leap motion controller for stroke rehabilitation. In CHI'14 Extended

Abstracts on Human Factors in Computing Systems. Toronto, Canada, pp. 1663-1668.

Lee, S. et al., 2017. User study of VR basic controller and data glove as hand gesture inputs in VR games. Paper presented

at the 2017 International Symposium on Ubiquitous Virtual Reality (ISUVR).

Peacocke, M. et al., 2015. Evaluating the effectiveness of HUDs and diegetic ammo displays in first-person shooter games.

Paper presented at the 2015 IEEE Games Entertainment Media Conference (GEM). Toronto, Canada, pp. 1-8.

Pirker, J. et al., 2017. Gesture-based interactions in video games with the leap motion controller. Paper presented at the

International Conference on Human-Computer Interaction.

Potter, L. E. et al., 2013. The leap motion controller: a view on sign language. Paper presented at the Proceedings of the

25th Australian computer-human interaction conference: augmentation, application, innovation, collaboration.

Adelaide, Australia, pp. 175-178.

Qin, H. et al., 2009. Measuring player immersion in the computer game narrative. Intl. Journal of Human–Computer

Interaction, 25(2), pp. 107-133.

Shin, D., 2019. How does immersion work in augmented reality games? A user-centric view of immersion and engagement.

Information, Communication & Society, 22(9), pp. 1212-1229.

Vokorokos, L. et al., 2016. Motion sensors: Gesticulation efficiency across multiple platforms. In 2016 IEEE 20th Jubilee

International Conference on Intelligent Engineering Systems (INES), IEEE. Budapest, Hungary, pp. 293-298.

Wozniak, P. et al., 2016. Possible applications of the LEAP motion controller for more interactive simulated experiments

in augmented or virtual reality. Paper presented at the Optics Education and Outreach IV.

Xiao, R. et al., 2016. Augmenting the field-of-view of head-mounted displays with sparse peripheral displays. Paper

presented at the Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems.

Xu, W. et al., 2019. Pointing and selection methods for text entry in augmented reality head mounted displays. Paper

presented at the 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

Yeo, H.-S. et al., 2015. Hand tracking and gesture recognition system for human-computer interaction using low-cost

hardware. Multimedia Tools and Applications, 74(8), pp. 2687-2715.

ISBN: 978-989-8704-31-3 © 2021

156

Documents

GESTURE-BASED HEAD-MOUNTED AUGMENTED REALITY GAME