AFFECTIVE GAMING: BEYOND USING SENSORS

Irene KotsiaA,B, Ioannis PatrasA and Spiros FotopoulosB

A. School of Electronic Engineering and Computer ScienceQueen Mary University of London, Mile end Campus, UK, E14NS

{irene.kotsia, i.patras} @eecs.qmul.ac.ukB. Electronics Laboratory, Department of Physics, University of Patras, Patras 26500, Greece

spiros@pelops.physics.upatras.gr

ABSTRACT

Affective, ‘emotional’ as widely known, gaming, consti-tutes the new frontier for game design and development. Theultimate goal is being able to read the emotional state of agamer and use it to change the game in such a way so asto provide to him/her a more immersive experience, a bettergameplay.

However, existing affective gaming approaches use spe-cialized sensors in order to extract behavioral cues, introduc-ing in that way a variety of challenges. The main issue tobe resolved is that of affecting the player’s immersion in thegame scenario by having an impact on his/her behavior, interms of the actions and emotions he/she displays.

In this paper we survey existing approaches in the field ofaffective gaming, briefly describing the sensors used to ex-tract behavioral cues (mainly physiological ones) and alsopresenting the commercial applications developed that em-ploy those sensors. In addition, we propose two effective,low-cost and easy to implement affective game scenarios, inwhich the behavior of a social group playing with a gamesmachine is studied. The proposed scenarios ‘use’ Kinect toextract the behavioral cues under examination, that can belater used to evoke specific emotions to the players and alterthe game’s objective and plot, providing in that way a morerealistic interaction between the player and the game.

Index Terms— Affective gaming, emotion recognition,action recognition, group interactions, group relationships,group behavior.

1. INTRODUCTION

The term affective gaming refers to the new generation ofgames in which the players’ behavior directly affects thegames objectives and gameplay. More precisely, the emo-tional state and actions of a player can be correctly recog-nized and properly used in order to alter the gameplot andoffer to the player an increased user experience feeling. Inother words, the emotions and actions of a player are of ex-

treme importance, as the behavioral cues extracted from themwill define the way the game will progress.

The ultimate goal of affective gaming systems is to creategames that will be ‘intelligent’ enough to understand whateach player feels at each specific moment, using behavioralcues obtained either in an obtrusive way (e.g. using neuro-physiological measurements), or in an unobtrusive one (e.g.observing his/her facial expressions, body pose and actions).However, existing scenarios employ sensors that limit thefreedom of movement, for example extracting neurologicalcues requires the player to be sitting in front of the com-puter (game console). As a result, a poor quality/less realisticimmersion of the player in the game environment may beachieved, due to the fact that the player may be conscious ofthe recording/measurement device and thus exhibit unusualbehavioral patterns (not corresponding to spontaneous be-havior). The players tend to prefer less intrusive methods ofphysiological input for their games [1]. Therefore, the ex-traction of the necessary behavioral cues in an affective gamescenario should be performed in a way that is not perceptibleto the player and does not limit his/her actions.

The introduction of Xbox Kinect has opened new avenuestowards this direction. Although we are now able to over-come many technical obstacles and realize in a more efficientway gestural interaction with a machine, several user-centricquestions remain unanswered, including how to sense and un-derstand group behavior.

In this paper we propose an affective gaming scenario thatoperates in a totally unobtrusive way. In the scenario underexamination state-of-the-art technology (Microsoft Kinect[2]) is used to extract behavioral cues using only ComputerVision techniques. The players are able to freely interact witheach other not being restricted by wearable sensors. Our aimis to understand individual and social group behavior.

The rest of the paper is organized as follows. In Section 2we briefly describe the approaches existing in literature. Moreprecisely, we briefly describe the hardware solutions proposedin order to extract behavioral cues and present the existinggame applications that employ them. In Section 3 we pro-

Proceedings of the 5th International Symposium on Communications, Control and Signal Processing, ISCCSP 2012, Rome, Italy, 2-4 May 2012

978-1-4673-0276-0/12/$31.00 ©2012 IEEE

pose two affective gaming scenarios (single and multi playerscenarios) that progress the state of the art beyond sensors andallow us to study individual and social group behavior in orderto create more realistic player-computer interaction systems.We finally draw conclusions in Section 4.

2. SENSOR-BASED AFFECTIVE GAMING

Affective gaming is still at its infant stage. Most of the ap-proaches followed in the past involve the theoretical analysisof affective game systems in order to describe their funda-mentals from a physiological point of view, to identify theirconceptual and technological capabilities and to propose sim-ple applications towards this direction. In order to extract be-havioral cues, they used specialized sensors, mainly physio-logical and neurological ones.

The first category includes devices that measure humansignals that are captured by sensors that require the player’sfull collaboration, making difficult, if not impossible, to ig-nore their presence. Representative examples include Elec-troencephalography (EEG) devices, that measure the brainelectrical activity and facial electromyography (EMG) thatmeasure the motion of the facial features. In both cases, theplayer has to wear a number of electrodes (to his scalp andfacial area, accordingly) in order for the system to extract theinformation needed.

The second category includes devices that measure hu-man signals signals that are easier to obtain and result inmore player-friendly scenarios. Less invasive devices in-clude electrocardiography (EKG)/ heart rate measurementdevices, that measure the number of heartbeats per unit oftime, strain/stress or respiration measurement devices, thatmeasure the number of breaths taken within a set amountof time, thermometers, that measure the temperature of thebody/body part and galvanic skin response (GSR) devicesthat measure skin conductivity. All of the above mentioneddevices involve a smaller set of sensors (in most cases a sin-gle one) that is attached at the human torso/wrist/finger andrestricts the player’s actions in a smaller extent. In Fig. 1 themost widely used specialized sensors are depicted.

Simple, prototypic in most cases, applications employingthe above mentioned sensors have been created to study therelationship between several physiological signals and con-cepts like game difficulty and player engagement, arousal andperception. Most precisely, in [3] the authors studied the rela-tionship between the game difficulty and the pressure appliedon the gamepad used to provide input, to use it as an indi-cation of the player’s state of arousal. In [4] the authors de-veloped a game that considered the relaxation/stress levels ofthe players. The relaxation level, measured using EEG, wasused to control the movement direction of a ball. In [5] theauthors described the fundamentals of affective gaming froma physiological point of view and measured the player’ heart-beat rate to use as an indication of the his/her engagement

Fig. 1. Examples of electroencephalography, electrocardio-graphy, respiration device, thermometer, galvanic skin re-sponse and facial electromyography devices.

with the game. The goal was to alter the gameplay in orderto control the player’s affective state and ensure his/her en-gagement. In [6] the authors created a social robot that eval-uated the outcome of a player’s action in a chess game. Themain goals were a) to study if the robot’s emotional behav-ior reflects what happens in the game b) to study if the users’perception of the game increases in such a case.

Mot recent works have been highlighting the importanceof emotions in affective gaming. In [7][8] the author dis-cuses the importance of emotions in affective gaming scenar-ios. More precisely, emphasis is given on enhancing real-ism and believability in generated ‘affective behaviors’ in thegame characters and players avatars. In [9] the authors pro-vide theoretical foundations for modeling emotions in gamecharacters, as well as practical hands-on guidelines to helpgame developers construct functional models of emotion.

In [10] the authors proposed a system for affect recogni-tion that combines face and body cues in order to achieve af-fect recognition. They employed Computer Vision techniques(Hidden Markov Models (HMMs), Support Vector Machines(SVMs) and Adaboost) to fuse the available facial (appear-ance, e.g wrinkles or geometric feature points) and body (sil-houette and color based model) cues. However, the databaseon which the experiments were conducted included record-ings of subjects sitting in front of a camera and reacting toa provided scenario. Therefore, although the subjects werefree to express theirselves, no real body pose information wasavailable. Moreover, the database included recordings of asingle person, thus not being suitable for group behavior re-search.

2.1. Existing commercial games

In this Section we will discuss existing commercial gamesthat use the physiological and neurological cues measured us-ing the sensors presented above. The commercial affectivegames that have been developed are the following:

• Bionic Breakthrough (1983), a bounce the ball into abrick wall game. The player wears a headband on hishead whose sensors are supposed to pick up any facialmovements or muscle twitches, in order to control themovements of the paddle and use is as input instead ofan ordinary joystick.

• Missile Command (1980), in which the player has todestroy moving targets. The heart beat rate of a playeris measured and used to change the nature of the chal-lenge the game presents. The aim is to keep engage-ment within an optimum range.

• Oshiete Your Heart (1997), a Japanese dating game.The heart beat rate and sweat level of a player is mea-sured. The goal is to use the measurements in order toinfluence the outcome of a date.

• Zen Warriors [11], a fighting game where, to performtheir finishing move, the player has to switch from fastpaced aggression, to a Zen-like state of inner calm.

• Brainball [4], in which the player’s relaxation status,measured using EEG, moves a physical ball across atable between two opponents. The player who is morerelaxed wins by moving the ball all the way to his/heropponent’s side.

• Left 4 Dead 2 (2009), a first person shooter video game.The player’s stress level, measured as the electric re-sponse of the player’s skin, determines the pace thegame. The goal is to make the game easier if the playeris too stressed.

3. PROPOSED SCENARIOS

3.1. System overview

In this Section we propose a set of affective game scenariosthat enables us to study human behavior in an individual andsocial group level. The proposed scenarios employ only Com-puter Vision techniques to extract behavioral cues, disposingin that way wearable sensors. More precisely, the scenariosunder consideration involve either an individual, or a smallsocial group of people residing in the same space playing witha games machine. For the purposes of this research the Mi-crosoft Kinect [2] games machine was used due to its low costand wide accessibility.

The experimental setup consists of four Kinect sensors(their topology is shown in Fig. 2) [12] in order to capturethe complete scene (360o) and create in that way a multicam-era scenario. This is of great importance as the presence ofmultiple players adds extra issues to be resolved, such as oc-clusions (either due to space limitations or caused by the co-existence of many people in the same space) and clutter. Byusing 4 Kinect sensors we can effectively extract the 3D scene

information and the accompanying texture information, for 4different viewpoints.

Fig. 2. A diagram of the proposed scenarios.

3.2. Single-player game

This scenario involves just one person playing with the gamesmachine. We propose to use facial expression, body pose andactions recognition to extract the behavioral cues. The in-formation facial expressions, body pose and actions reveal isfused to extract the emotional state the player is in within anaction context.

However, several computer vision problems have to betackled in such a scenario. More precisely, the first step in-volves the real-time detection of a player and of his/her bodyparts. Additionally, the problem of recognizing his/her ac-tions and emotions has been widely studied in the past, but in-volving a set of predefined classes under examination. There-fore, the introduction of spontaneity, as the player may ex-press himself/herself in any way that he/she wishes, consti-tutes an extra challenge. Moreover, the proposed scenarioemploys many cameras so as to extract information from dif-ferent viewpoints. Although the information from differentviewpoints aids in correctly detecting the player and recog-nizing his body pose and emotions, finding efficient methodsto fuse those pieces of information remains an open problem.

3.3. Multi-player game

This scenario involves a group of people residing in the samespace. The goal is to extract the social group behavior. Sev-eral extra challenges exist, for example several occlusions dueto space limitations but also to the presence of many peoplein the same space. The free interaction among the playersand the way that affects their subsequent emotions and statesis also a novel field of research. After having efficiently rec-ognized the emotional state of each individual player within

an action context as performed in the single-player game sce-nario, the next step is to study the players as members of asocial group, that is to identify the way each player interactswith each other and also to identify relationships built withinthis interaction framework. For example, do a player’s actionsreveal a friendly/aggressive mood towards the other players?Can we use the player’s actions to predict subsequent actions?When it comes to the whole group. do the players choose aleader, even subconsciously, whom they tend to mimic? Arecohesion (i.e. tendency of people to form groups) relation-ships formed? How are all of the aforementioned interactionsand relationships developed in time? Can we exploit the in-formation their dynamics have to offer?

3.4. Computer vision meeting psychology

Both scenarios involve many interdisciplinary fields. Besidesthe obvious Computer Vision techniques that have to be em-ployed in order to extract behavioral cues, input from psy-chologists has to be provided. More precisely, input frompsychologists is required in order to properly define the sce-nario under examination in terms of the emotions and actionsthat are more likely to be observed during the game. The poolof possible interactions among the players as well as the rela-tionships that they are most likely to form while in the gameshould also be defined. Input from psychologists is also re-quired to extract the ground truth concerning the emotionalstate of each player and to explain the way it affects his/heractions as well his/her interactions with other players and therelationships built among them as member of a social group.The challenging nature of the proposed scenarios regardingbehavior understanding, in combination with the lack of avail-able datasets, constitutes the proposed research a novel field,even for psychologists.

Understanding human behavior in such scenarios willenable the development of automatic schemes for model-ing, recognition and understanding of small social groupactions and prediction of their dynamics (i.e. how group ac-tions evolve through time). More versatile ways for human-computer interaction will be developed leading to the creationof more user-centered applications.

4. CONCLUSIONS

Although the introduction of Kinect has opened new avenuestowards the creation of more realistic human-computer inter-action systems, existing approaches rely on specialized sen-sors to extract the necessary behavioral cues for human be-havior understanding. In this paper, we propose two efficientsensorless affective gaming scenarios that enable us to studyhuman behavior of individuals and social groups, using lowcost and easily obtained state of the art devices.

5. ACKNOWLEDGEMENT

This work was supported by the EPSRC grant Recognitionand Localization of Human Actions in Image Sequences(EP/G033935/1) and by the GSRT grant Understading ofHuman Actions in Groups - UHAG (PE6 3329).

6. REFERENCES

[1] L. E. Nacke and R. L. Mandryk, “Designing affectivegames with physiological input,” Workshop on Mul-tiuser and Social Biosignal Adaptive Games and Play-ful Applications in Fun and Games Conf. (BioS-Play),2010. 1

[2] “Xbox kinect,” www.xbox.com/en-GB/kinect. 1, 3

[3] J. Sykes and S. Brown, “Affective gaming measuringemotion through the gamepad,” Human factors in com-puting systems (CHI), 2003. 2

[4] S. I. Hjelm, “Research + design: the making of brain-ball,” Interactions 10, 2003. 2, 3

[5] A. Dix K. Gilleade and J. Allanson, “Affectivevideogames and modes of affective gaming: Assist me,challenge me, emote me,” DiGRA, 2005. 2

[6] A. Pereira I. Leite, C. Martinho and A. Paiva, “icat:an affective game buddy based on anticipatory mecha-nisms,” 2008. 2

[7] E. Hudlicka, “Affective computing for game design,”Proc. of the 4 Intl. North American Conf. on IntelligentGames and Simulation (GAMEON-NA), 2008. 2

[8] E. Hudlicka, “Affective game engines: Motivation andrequirements,” Proc. of the 4 International Conf. onFoundations of Digital Games, 2009. 2

[9] E. Hudlicka and J. Broekens, “Foundations for mod-elling emotions in game characters: Modelling emotioneffects on cognition,” ACII MiniTutorial, 2009. 2

[10] H. Gunes and M. Piccardi, “Automatic temporal seg-ment detection and affect recognition from face andbody display,” IEEE Transactions on Systems, Man, andCybernetics - Part B, Special Issue on Human Comput-ing, vol. 39, no. 1, 2009. 2

[11] “Zen warriors,” www.play-ground.co.uk. 3

[12] A. Maimone and H. Fuchs, “Encumbrance-free telep-resence system with real-time 3d capture and display us-ing commodity depth cameras,” www.cs.unc.edu/ mai-mone/KinectPaper/kinect.html. 3