Technological Advancements in Affective Gaming: A ... · Affective gaming has to move away from the conﬁdes of a laboratory and be deployed in a normal environment [13], [22], [68]

1

Technological Advancements in AffectiveGaming: A Historical Survey

Thomas Christy & Ludmila I. KunchevaSchool of Computer Science

Bangor University, LL57 [email protected] & [email protected]

Abstract—Affective gaming (AG) is a cross-disciplinary areadrawing upon psychology, physiology, electronic engineeringand computer science, among others. This paper presentsa historical overview of affective gaming, bringing togetherpsychophysiological system developments, a time-line of videogame graphical advancements and industry trends, therebyoffering an entry point into affective gaming research. Itis proposed that video games may soon reach a peak inperceivable graphical improvements. This opens up the doorfor innovative game enhancement strategies, such as emotion-driven interactions between the player and the gaming envi-ronment.

Index Terms—Affective gaming, emotion, psychophysiology,physiology, input devices, computer graphics.

I. INTRODUCTION

Video games have become the leading global entertain-ment industry, overtaking the movie industry on sales andrevenue statistics [21]. Provoking profound emotions isdeemed to be of vital importance in video game design [63].

Emotion is a crucial component of human interactions,problem solving [62] and entertainment [67]. The role ofemotion in human interactions was discovered nearly twomillennia ago and has been documented scientifically forwell over a century [16], [36]. Affective Gaming (AG) is arelatively new field of research that exploits human emotionfor the enhancement of player’s experience during videogame play.

Conventionally, the video game attempts to elicit emotionby its story lines, characters and video effects. However,there is no precise means to assess if the expected emotionis being experienced. It is even more important to be ableto detect a change in the player’s emotional state as a resultof the game play. There is no provision for assessing suchchanges within the mainstream gaming practices. AG seeksto remedy this through the use of bespoke psychophysio-logical input sensors that read and interpret the changes inthe player’s emotion in real time.

Technically, an AG system acquires emotion-related sig-nals from a set of sensory input modalities, analyses thesignals, and provides data to the game engine [35]. Thegame is subsequently altered taking into account the typeand strength of the measured emotion. Figure 1 shows adiagram of a real-time AG loop.

An AG system is expected to stream real-time affect data,be robust, and most importantly be easy to use. The main

Fig. 1. The real-time affective gaming (AG) loop.

criticism to experimental research in affective computingthus far has been that it is done in a heavily controlled envi-ronment, which limits its chances for practical applications.Affective gaming has to move away from the confides of alaboratory and be deployed in a normal environment [13],[22], [68].

A particular challenge when trying to design a single allencompassing taxonomy of the field comes from the factthat it is comprised of several broad disciplines: physiology,psychology, electronic engineering and computer science.It should also be mentioned that results of research into AGconducted in commercial settings are rarely published.

In this study, we survey the historical technological de-velopments and research efforts that are attempting to bringpractical AG to reality, including academic and commercialcontributions to the field as a whole. The rest of the paperis organised as follows. Section II summarises the difficul-ties in measuring and classifying emotion. We argue thataffective gaming does not rely exclusively on the accuracyof emotion recognition. Any change of the emotional stateof the player that is detected and reflected in the game cancontribute to the pleasurable experience. Behavioural andphysiological modalities for affective data acquisition arepresented in Section III. We put the emphasis on the wealthof physiological modalities as the preferred input for AG.Section IV reviews studies that use physiological modalitiesin AG. Section V contains a historical perspective of thedevelopment of two of AG-related technologies: consoleplatforms and video graphics. Finally, Section VI concludesthe paper by listing the tasks and challenges set beforemodern AG.

DOI: 10.5176/2251-3043_3.4.287

GSTF International Journal on Computing (JoC) Vol.3 No.4, April 2014

©The Author(s) 2014. This article is published with open access by the GSTF

32

Received 03 Oct 2013 Accepted 22 Oct 2013

DOI 10.7603/s40601-013-0038-5

2

II. EMOTION AND AFFECTIVE GAMING

Can we recognise and classify emotion? Emotion isnotoriously difficult to quantify, measure or put into clear-cut categories [53]. The prevailing evidence from psy-chology and psychophysiology is that emotions do notnaturally form distinct clusters in data spaces extracted fromrepresentational cues, but are rather facets of a continuum.The relationship between the physiological measurementsand the emotional states they are supposed to identify iscomplex and ambivalent [22]. The list of difficulties facedby researchers in automatic emotion classification has beenwidely discussed in the literature [10], [22], [41]. It includesbut is not limited to

• We don’t know what to measure.• Emotions experienced by the subject may not corre-

spond well to the stimuli.• Different subjects may react with different emotions

to the same stimulus.• The presentation of emotion will differ between sub-

jects and also at different time moments for the samesubject.

• Emotion is not clear-cut and measurable, thereforethere cannot be “ground truth” data.

• There is no agreed protocol for stimulating and mea-suring emotion.

• There is no agreed protocol for testing emotion clas-sification systems.

• The classes of emotions are intricately related to oneanother.

A vast diversity of results have been reported as the out-come of emotion recognition experiments [11], [47], [65].Owing to the difficulties listed above, classification accu-racy in identifying categorical emotions was found to varyconsiderably, spanning a range between 51% and 92% [61].Along with the excitement, the literature contains cautiousor even sceptical views [55]. It may be that endeavours tolabel emotion accurately across different subjects might bean impossible quest using today’s technologies [13], [66],[79]. Employing interdisciplinary research effort has beenstrongly advocated [41], [68].

Video games are designed to entertain, and thus allowa margin of error in recognising a specific emotion. Theultimate aim of an affective game is to make the playeraware that the game recognises and interacts with theiremotional state throughout the course of the game. Occa-sional misclassifications will not have a crucial impact tothe player’s enjoyment or satisfaction.

III. AFFECT ACQUISITION MODALITIES

A variety of input modalities are used to gather affectivecues from the player. Figure 2 shows a diagram of the majortypes of modalities in affective computing.

A. Behavioural modalities

Behavioural affect detection is an extremely instinctiveapproach and is often used in video game research and

development [24]. When engaged in a dialogue with areal person, it is normal to express facial emotions andemotive body gestures, or body language, as a responseto the message being communicated. There are at least100 documented facial expressions to concisely describethe emotion or mood of a person [60]. One need onlyconsider research based on Autism and Asperger Syndrometo recognise the need for visual emotion interaction insocial and professional environments. However, when facedwith a video game display, such outward expressions ofemotion are not expected from the active player. Sponta-neous emotions may be displayed but the player will notbe compelled to react in the same way as in human tohuman interaction. Emotion detection using cameras hasbeen criticised for being inefficient in that it requires alot of processing power to sift through video streams [23].In addition behavioural affect detection presents cultural,gender, and age differences, making behavioural analysisdifficult [73].

Voice modulation is perceived as one of the impor-tant behavioural modalities for detecting emotion in videogames. However, it has been observed that some playersfeel uncomfortable talking to a video game [38].

B. Physiological modalities

The two types of physiological modalities in Fig. 2 comefrom the central nervous system and peripheral nervoussystem, respectively.

1) Central nervous system signals: It is unlikely that AGwill benefit from functional Magnetic Resonance Imaging(fMRI) or functional Near Infra-red Spectroscopy (fNIRS)in the near future. Electroencephalography (EEG), how-ever, is an important technology in modern neuroscience.Compared to fMRI, EEG has a worse spatial resolutionbut a much better temporal resolution [27]. The electricalpotentials related to emotion can be projected widely inan intricate pattern across the scalp, and can thereforeoverlap with potentials evoked by other activities. EEGhas been applied for classification of emotions in variouscontexts [8], [43], [82], [88] and is progressively becominga portable lightweight technology. Several commerciallyavailable EEG sets can be considered for affective gam-ing: OCZ Nia1, Neurosky Mindset2, Neurosky Mindwave3,Emotiv EPOC4. It is often assumed that the projections ofpositive and negative emotions in the left and right frontallobes of the brain make these two emotions distinguishableby EEG. Practice has shown that the granularity of theinformation collected from these regions through EEG maybe insufficient for detecting more complex emotions [8].

2) Peripheral nervous system signals: Psychophysiology(PPG) is the combination of innate human physiologicalresponses in relation to emotional changes. The correlationbetween emotion and physiology is well founded [67], [93],

1http://www.ocztechnology.com/nia-game-controller.html2http://store.neurosky.com/products/mindset3http://neurosky.com/Products/MindWave.aspx4http://emotiv.com



33

3

Input Modalities for Affective Computing

Peripheral Nervous System

Central Nervous System Naturally Observed

Computer Interaction

Physiological Behavioural

1 Electroencephalography (EEG)

2 functional MRI (fMRI)

3 functional Near-infra-red Spectroscopy (fNIRS)

4 Electrooculography (EOG)

5 Electromyography (EMG)

6 Electrodermal Activity (EDA)

7 Blood Volume

8 Blood Oxygenation

9 Respiration

10 Skin Temperature

Facial Expression 11

Eye Tracking/Blink Rate 12

Gesture 13

Posture 14

Voice Modulation 15

HCI Patterns 16

Dialogue with Agent/Tutor 17

Pressure on Mouse 18

Fig. 2. Main input modalities in affective computing.

[96], [98], and has been thoroughly explored throughoutpsychological studies [85]. Some of the most commonlyused physiological inputs are listed below.

• Electrodermal Activity (EDA) Fig. 2 (#6), also re-ferred to as Galvanic Skin Response (GSR), SkinConductance Response (SCR) or Psycho GalvanicReflex (PGR), measures the variance in electricalconductivity through the surface of the skin. EDAreadings are effected through the sympathetic nervoussystem, making it a good indicator of stress andanxiety. EDA suffers from latency, with a delay ofapproximately one second for a response to be evoked,followed by approximately three seconds for the effectto dissipate. It is among the most basic and low costphysiological modalities available, and is widely usedin physiological emotion recognition, including videogames [1], [17]. EDA is commonly read between twofingers on either hand, although is not limited to thisarea of the body [7]. EDA bodes well for adaptationinto video-game controllers.

• Blood volume (Plethysmograph) Fig. 2 (#7-8) is thevariation in heart rate, and is a good indicator of stressand anxiety. As a sign of its pervasive popularity,heart rate input became the pivotal component of atelevision game-show, called The Chair [83], [97]. Inthis show, contestants answered general knowledgequestions and were expected to maintain a calm heartrate to win money. The sensing devices suitable foraffective gaming come in the form of a clip, whichuses optical technology to measure simultaneouslyheart rate and blood oxygenation [81].

• Respiration Fig. 2 (#9) Emotion can influence breath-ing rates [6], [34]. The measuring device could bea respiration belt or sensors embedded into clothing.For example, a force feedback vest with embeddedbreathing rate sensors already features in the avid pro-gamers’ arsenal5. However, mainstream applicationscould be hindered by utilising garments to acquiredata.

• Temperature Fig. 2 (#10) Body temperature is affectedby emotion, specifically joy, anger and sadness [57],[58], and has been used for emotion recognition invideo games [7], [90]. Temperature sensors fall intotwo general types: contact and non-contact. Bothtypes are sensitive to movement, which can introduceinaccuracies in the data collected. Movement is animportant issue in the process of an active game play;hence, the positions of the sensors have to be chosencarefully.

• Electromyography (EMG) Fig.2 (#5) measures theelectrical activity produced by muscle movement. Ac-tivity patterns in muscles such as orbicularis oculi(eye) and zygomaticus major (smile) are often usedfor affect detection. However across the populous(including nationalities), people differ widely in termsof how they visually display emotion [66], [79]. Be-sides, EMG sensors may need to be placed at variousbody locations, which may compromise the player’scomfort.

5http://en.wikipedia.org/wiki/3rd Space Vest



34

4

3) Sensors and devices: Table I shows the early patents,designed to detect PPG signals. These systems formed thebedrock of modern PPG technologies.

Sensors and devices for AG can be roughly grouped intothee types.

• Standard sensors. Hardware used in AG researchhas been borrowed from psychological research orcommercial relaxation systems, such as the BioPac6,IOM7, etc. Commercial hardware for physiologicalacquisition is considered expensive and awkward touse [42]. For example, standard sensors give betterresults when they are held still. However, the devicescan regraded as a nuisance, being attached to theplayer’s fingers, ear, etc. This makes them unsuitablefor active video game play.

• Wearable sensors. Wearable sensors implies “bodyworn”, making long term physical contact with thebody [66]. In 1997, Picard and Healey [66] introducedan affective wearable system that stored physiologicaldata from Respiration, EDA, Blood Volume Pulse(BVP) and Electromyography (EMG), for later anal-ysis. Picard’s work would later become a beacon foraffective computing research [7], [32], [45]. Sensorscan be embedded into clothing, glasses, gloves, shoes,hats, helmets, jewellery, etc., making this an attractiveavenue for AG.

• Seamless contact sensors and devices. The sensorsin this group come into contact with the body for alimited time, for example through traditional interfacessuch as mouse and keyboard. To be considered seam-less, the user should not be aware of any interactionwith the sensor. For example, EDA could be measuredfrom electrodes embedded into the hand-grip of a con-sole controller or on a mouse. A comprehensive studyon devices of this type is provided by Reynolds [76],examples of which are

– Sentograph – measures and visualises user’s touchalong a two-dimensional space

– Touch Phone, touch mouse – measures gripstrength

– Sentic Mouse – senses directional input– Sqeekee – click pressure– IBM Emotion Mouse – gathers temperature, EDA

and somatic movement

Even though the core technology for physiological signalcapture is mature and well proven, hardware for affectivegaming is still not widely available.

Affective video game input must be comfortable andintuitive to use. If a sensor impedes the enjoyment and com-petitive edge that video game players expect, it is less likelyto be adopted. AG would benefit best from seamless contactsensors. It needs to move from expensive laboratory equip-ment to reliable and affordable consumer devices [13], [22],[68]. Affective sensors can be incorporated into already

6http://www.biopac.com/psychophysiology7http://www.wilddivine.com/iom-feedback-hardware/iom-active-

feedback-hardware/

adopted video game controllers [2], [7], [13], [50]. Valve8

and Sony 9 have both implied that EDA and HR couldsoon be incorporated into standard controllers. Producingbespoke low cost hardware for AG is feasible [7], [13],for example, by using recently released rapid prototypingplatforms10 and 3D printing. It is proposed that this type ofrapid prototyping is expected to shape the newly forminglandscape of AG.

IV. USING PHYSIOLOGICAL MODALITIES IN AG

By 2001, academic research into AG took off [5], [69],[76]. Table II lists the research contributions made withinthe field of AG, including the physiological sensors andvideo games used. Many sensors and sensor permutationshave been tested, along with several game genres.

Tables III and IV detail game variations in responseto changes in physiological data, published by Dekker &Champion [17].

TABLE IIIDEKKER & CHAMPION VISUAL EFFECTS AND PHYSIOLOGICAL

THRESHOLD CONDITIONS; USED IN THE MODIFIED VIDEO GAME

Half Life 2.

Emotion Game Change Criteria*

Comatose Sound (heart beat), HR + EDA < 0.8new enemy or boss or < 0.4 below avg

Bored Black & White HR < 0.8− avg

Calm Shader (White filter) HR > ×2+ avg

Worried Shader (Red filter) HR > avg ×2

Panic Shader (Bright red filter) HR > ×3 above avg+ FOV = 130

Berserk Shader (Red screen) HR > ×3.5 above avg

*Abbreviated terms: Heart Rate (HR), Electrodermal Activity (EDA). Shaders referto graphical overlay alterations

TABLE IVDEKKER & CHAMPION CONDITIONS APPLIED TO

PSYCHOPHYSIOLOGICAL THRESHOLD CRITERIA IN THE VIDEO GAME

Half Life 2.

Condition Criteria*

Stealth EDA 0.5 ≫ 0.7

Invisible HR < 0.5 and EDA < 0.5

Weapon damage HR & EDA ×40

Speed HR

Sound volume EDA2 × 0.8

Bullet time (Slow motion) EDA > cal ×3

Shake HR > 3.8 above avg

*Abbreviated terms: Heart Rate (HR), Electrodermal Activity (EDA).

In this example, visual and audible effects are introducedto the game when particular electrodermal activity (EDA)

8http://www.valvesoftware.com/9http://www.sonyentertainmentnetwork.com/10https://www.ghielectronics.com/, http://www.netmf.com/



35

5

TABLE IEARLY DAY DEVICES PRE-DATING AFFECTIVE GAMING TECHNOLOGIES.

[h] Year Author Acquired a patent for:1928 Schrawzkopf & Wodtke [80] An electrocardiograph.1928 Hathaway [30] A circuit design of an EDA device Apparatus for measuring psychogalvanic responses.1943 Raesler et al. [72] Psychogalvanometer; a fluid-aided EDA device.1944 Milne et al. [56] Psychometer; an easy to use device.1953 Koller [44] Cardio-Pneumo-Electrodermograph; an advance upon earlier machines.1953 Holzer & Marko [33] Arrangement for recording variations in the electrical resistance in the human body.1954 Mathison [51] Electropsychometer or Bioelectronic instrument (v1); a sponge-aided EDA device.1955 Golseth & LeGrand [29] Electronic Diagnostic Instruments; a portable EDA device.1956 Mathison [52] Second instalment of the Electropsychometer or Bioelectronic instrument (v2); a low cost EDA device.1958 Douglas [18] Psychogalvanometer; EDA plus phonographic recorder.1964 Ryan [77] An improved Novelty (Toy) lie detector.1965 Takagi [87] An EDA device.1967 Weidinger et al. [95] A portable heart monitoring device.1969 Tygart [91] System for FM transmission of cardiological data over telephone lines.1972 Burlyl R. Payne [64] Audible Psychogalvonometer (AP); smaller and low-cost EDA device.

and heart rate (HR) signal threshold combinations are met.Tijs et al. [89] use 7 physiological parameters to guide thespeed of Pacman (Table V).

TABLE VTijs et al. [89]: EFFECTS OF GAME SPEED OF Pacman, BASED ON

PSYCHOPHYSIOLOGICAL DATA INPUT

Physiological features* condition identified (action)

Mean SCL slow (speed up)

Number of SCR slow (speed up)

Mean HR slow (speed up)

Mean respitory slow (speed up)

CORR fast (slow down)

ZYG fast (slow down)

Key pressure slow (speed up) & fast (slow down)& normal (nothing)

*Abbreviated terms: Heart Rate (HR), Skin Conductance Level (SCL),Skin Conductance Response (SCR), corrugator supercilii muscle,

(CORR), zygomaticus major muscle (ZYG).

To illustrate the developmental potential of the physi-ological input modalities, we collated the accessible pub-lications explicitly devoted to affective gaming, and notedwhich ones address the engineering of bespoke psychophys-iological hardware. Figure 3 gives a bar chart of thedistribution of the publications over the years. The paperswhose main focus are psychophysiological sensors anddevices are marked in a lighter colour.

Even though the publication numbers are not high, anemerging trend could be identified. Affective input devicesare beginning to make their way to the centre stage ofaffective gaming.

Judging by the achievements and the potential of usingphysiological modalities in AG, the future is likely tosee smarter and more sophisticated implementations onprogressively smaller, more robust, reliable and noise-freedevices.

V. TECHNOLOGIES FOR AG

A. Generations of console platforms

Historically, the introduction of the electronic computerheralded the beginning of a new era [14]. No sooner

2002 2004 2006 2008 2010 20120

1

2

3

4

5

6

7

8N

umbe

r of

AG

pap

ers

Fig. 3. Time-line view of AG research publications. Publicationsaddressing bespoke psychophysiological hardware are shown in a lightercolour.

had the computer become affordable to the masses; videogames became an arising staple of popular entertainment[70]. Interactivity plays a pivotal role of video games.Typically, video games allow the player to interact with thevirtual environment using joysticks, joy-pads, keyboards,mice, trackballs, cameras, touch-screens, etc. Video gameconsoles became a dominant symbol of the mainstreamvideo game market. Several attempts have been madeto introduce AG to commercial environments [3], [15],[25], [26], [94]. The companies involved manufacturedthe necessary hardware and software to foster the use ofaffect in video games. However, none of the systems weresuccessful in the long term. Below we take a closer lookat the developmental context and the possible reasons forthe delayed progress.

Figure 4 shows a time-line representing the introductionand commercial duration of 8 video console generations.The thick black lines show the time span of each generationand the vertical dashed indicate the year of key commercialcontributions.

The first video game console, called the Magnavox



36

6

TABLE IIAFFECTIVE GAMING MODALITIES AND THE CURRENT CONTRIBUTORS, LISTED IN ORDER OF YEAR. MODALITIES INDICATE WHICH COMBINATION

OF PSYCHOPHYSIOLOGICAL OR BEHAVIOURAL AFFECT DETECTION SENSORS CONSIDERED AND GAME DETAILS WHICH GAMES AND

GAME-GENRES USED.

Modalities & year published* Game

EDA [5], 2001. Racing Dragon

HR, EDA, EMG, video [79], 2002. Puzzle

HR [28], 2003. Action based

Game pad pressure [86], 2003. Space Invaders

EMG [31], 2006. Juiced (THQ)

EDA,EMG [4], 2005. Cards (Skip Bo)

ECG, ICG, HR, HS, EDA, EMGtemperature, questionnaire [73], 2005. Pong

User control knobs [78], 2005. Generic

HR, EDA [54], 2006. Treasure Hunt (Source)

HR, EDA, facial EMG [74], 2006. Monkey Ball 2

EDA, HR [17], 2007. Half Life 2

EDA, ECG, HR [49], 2007. NHL 2003

HR, EOG, EDA, EEG,respiration, temperature [12], 2008. Tetris

Time, eye movement [37], 2008. Half Life

Audio (vocal cues) [39], 2008. Half Life Mod

EEG [71], 2008. Break-Out (Arkanoid)

HR, EDA, facial EMG [75], 2008. Monkey Ball 2 & James Bond 007

HR, EDA, respiration [89], 2008. Pacman

EDA, HR [19], 2009. Prey, Doom3 & Bioshock

EDA, HR, EMG, temperature [48], 2009. Pong & anagrams

EDA, EMG [59], 2009. Half Life 2 Mod

Control tilt, pressure [92], 2009. Need 4 Speed

EDA, respiration [46], 2010. Emoshooter (FPS)

EDA [90], 2010. Racing

(Use) EDA, (tried) HR, eye movement, EEG,pupil dilation, EOG, posture, gesture, voice,face expression, respiration [2], 2011. Left 4 Dead 2, Portal 2 & Alien Swarm

EDA, HR, pressure [7], 2011.temperature, gyroscope Racing Car

EDA, HR, temperature [13], 2013. Custom game

*Abbreviated terms: Heart Rate (HR), Heart Sound (HS), Electrodermal Activity (EDA), Electrooculography (EOG),Electroencephalography (EEG), Electromyography (EMG), First Person Shooter (FPS), Impedance Cardiogram (ICG).

Odyssey, was released in 1971, as shown in Fig. 4. In1982 Atari planned to release an input device calledthe Mindlink, but the company collapsed prior to its re-lease [94]. The Atari Mindlink was a head-mounted devicethat used Electromyography (EMG) to detect electricalchanges in the player’s forehead. The Mindlink was tobe used as an input device in lieu of a standard joystickcontroller. The device was abandoned when Atari wasredistributed in 1983 [94].

In 1984 CalmPrix was introduced with a psychophysio-logical system called CalmPute. CalmPrix was a car racingvideo game that used EDA signals to alter the speed ofthe racing car. It used a commercial EDA sensor called theMantra Mouse or GSR2 [25] as an input device. The Calm-Prix video game was meant to teach a relaxation technique,

using an extremely primitive racing car game (even for thetime). CalmPrix would have been pitted against games suchas the highly successful Mario Brothers (1983)11, Return ofthe Jedi and Elite (1984)12. Commercially, it didn’t standa chance.

Tokimeki memorial oshiete your heart was a provocative,alluring arcade game that was released in 1997. Unfortu-nately, the game was available only in Japan which limitedits global success. Tokimeki memorial oshiete my heart wasa curious cartoon dating game, far flung from the populargames of the day, such as Herc’s Adventure, BackyardBaseball and Claw13. It encouraged players to express

11www.IMDb.com - Highest Rated Video Games Released In 198312www.IMDb.com - Highest Rated Video Games Released In 198413www.IMDb.com - Highest Rated Video Games Released In 1997



37

7

1970 1975 1980 1985 1990 1995 2000 2005 2010 20150

1

2

3

4

5

6

7

8

9

Gen

erat

ions

1971, Magnavox Odyssey

1982, Mindlink (Atari)

1984, CalmPrix (CalmPute) 1997, Tokimeki memorial (Konami)

1998, Tetris 64 (Nintendo)

Fig. 4. Video game console time-line representing the introduction and commercial duration of each console generation, shown in thick blacksegments. Key commercial contributions are indicated with vertical dashed lines pinpointing the year of release. Given in brackets are the names ofthe relevant companies. CalmPrix and Tokmeki Memorial are plotted at the bottom of the figure because they were not console based.

affect towards cartoon characters (Manga art), which wasmeasured by EDA and Heart Rate (HR).

Tetris 64 was released on the Nintento 64 in 1998. It usedHR taken from a clip attached to the player’s ear to controlthe speed of the falling shapes. The game allowed up to fourplayers to compete on the same screen, making it possibleto alter the outcome of the competition using emotionalfeedback. Unfortunately, the game concept was rather old.Although Tetris remains a popular relic, it was competingagainst games such as Half Life, Crash Bandicoot, SonicAdventures, etc.14. In addition, it was reported that Nintendowas experiencing issues with its latest console. Amongthese were expensive cartridges, poor graphical renderingand a complex programming interface [20].

Other physiological devices have since been considered,such as the Wii Vitality. However, this device was notreleased due to speculation that it would not prove com-mercially successful [40].

B. Graphical advancements

Immersion is important for emotional experiences withinvideo game environments [9], [37]. The illusion of im-mersion and interaction with surroundings, materials andsubstances in modern video games cannot be achievedwithout the giant steps in video graphic hardware andsoftware. Graphic technology and video games evolvedsynchronously, from primitive monocohrome pixelated-blocks being controlled on a cathode ray tube (CRT) toadvanced visually realistic, fully immersive, simulated en-vironments, presented on ultra high definition (4K) organiclight emitting diode (OLED) liquid crystal displays (LCD).Detailing advanced graphical techniques, such as particlesystems, billboards, shadow-maps, etc., is beyond the scopeof this paper. Here we are interested in the exponentialtechnological growth in video graphics performance andits relationship with AG.

Companies such as Intel, Array Technologies Industry(ATI) and nVidia made vast efforts in the graphic ren-

14www.IMDb.com - Highest Rated Video Games Released In 1998

dering technology arena, through competitive video cardreleases. An advanced programming interface (API) calledOpenGL15 (Open Graphics Library) was introduced in1992. It offered a single graphical programming interfaceacross all popular technologies and platforms, therebyspeeding up game development. Notably, nVidia releasedthe Graphical Processing Unit (GPU) in 1999. Mega(10002) Texels (MT) are defined as the number of textureelements graphic hardware processors can manipulate periteration. Figure 5 demonstrates the exponential growth ofMT graphics hardware performance since 1997. In 1997,the maximum MT expected from a 3D rendering graphicprocessor was log

10(100), which grew exponentially to

log10(1875000) by mid-201316.

1996 1998 2000 2002 2004 2006 2008 2010 2012

2

2.5

3

3.5

4

4.5

5

5.5

6

6.5

Meg

atex

els

log 10

nVidia released GPU

Fig. 5. Graphic card technology advancement since 1997 beginning in1997 at 100 mega-texels (MT) and growing exponentially to 1,875,000 MTby mid-2013. The vertical axis is log10(y), where y represents maximumMT per year.

It is suggested that graphical advancements could soonreach a plateau. High-end video game systems are now

15http://www.opengl.org/about/16http://www.techspot.com/article/650-history-of-the-gpu/



38

8

capable of animating real-time graphic renderings that arebecoming visually close to that of real life video footage.In addition, video graphic displays have pixels that are sominute, they cannot be discriminated individually by thehuman eye, with pixel density greater than 300 pixels perinch. As the visual component of video games reaches apeak, more emphasis on player’s interaction and immer-sion is expected. Games are now appealing to a moresophisticated audience, with the average age of the frequentgame purchaser being 35 [21]. This calls for exploring newavenues in simulated intelligence and affect manipulation.

Commercial video game publishers are rumoured to beconsidering psychophysiological hardware, as part of theirnext generation of video game consoles [84]. This mayherald the birth of real commercial investment into AG.

VI. CONCLUSION

The general consensus is that using the player’s affectivestate to manipulate the video game adds to the enjoymentand immersion experienced. AG will form an importantrole in the future of human computer interaction. The tasksremaining for modern AG can be summarised as follows

1) Bring together expertise from the various fields thatAG draws upon: psychology, physiology, computerscience and engineering.

2) Create unobtrusive, robust and accurate devices forinputting physiological signals such as electrodermalactivity (EDA) and heart rate (HR) among many.

3) Develop state-of-the-art pattern recognition and ma-chine learning methodologies for reliable emotionrecognition as well as algorithms for detectingchanges in the player’s emotional state.

4) Move AG out of laboratories and into the developer’shands. A step in this direction would be the cre-ation of an open affect application protocol interface(OAAPI) for affect acquisition. When no affect acqui-sition hardware device is present the interface wouldemulate the required emotive techniques necessaryusing software. This would enable the developer tocode the game for affective video games with orwithout hardware specific knowledge.

We speculated that video graphic advancements are head-ing towards a plateau, which will likely shift the focustowards intelligent interactivity and affective gaming. Sucha shift is expected to open up commercial perspectives forAG.

Taking a look into the not so distant future. Equippedwith inventive affective input devices, using fast and pow-erful pattern recognition algorithms and realistic graphicalrendering technology, the game developer is facing the mostexciting challenge of all: creating the affective gaming loopto maximise the gamer’s entertainment pleasure.

REFERENCES

[1] M. Ambinder. Valve’s approach to playtesting: The application ofempiricism. Valve Software, 2009.

[2] M. Ambinder. Biofeedback in gameplay: How valve measuresphysiology to enhance gaming experience. Valve Software, 2011.

[3] Amtex. Tetris 64. Seta Corporation, November 1998.[4] C. Becker, A. Nakasone, H. Prendinger, M. Ishizuka, and

I. Wachsmuth. Physiologically interactive gaming with the 3D agentMax. In International Workshop on Conversational Informatics atJSAI, volume 5. Citeseer, 2005.

[5] D. Bersak, G McDarby, N. Augenblick, P. McDarby, D. McDonnell,B. McDonald, and R. Karkun. Intelligent biofeedback using an im-mersive competitive environment. Designing Ubiquitous ComputingGames Workshop, 2001.

[6] F.A. Boiten. The effects of emotional behaviour on components ofthe respiratory cycle. Biological Psychology, 49(1-2):29–51, 1998.

[7] A. Bonarini, F. Costa, M. Garbarino, M. Matteucci, M. Romero, andS. Tognetti. Affective videogames: The problem of wearability andcomfort. SpringerLink: Human-Computer Interaction. Users andApplications, 6764:649–658, 2011.

[8] D. O. Bos. EEG-based emotion recognition. [Online]url: http://emi.uwi.utwente.nl/verslagen/capita-selecta/CS-oude Bos-Danny, 2006.

[9] E Brown and P Cairns. A grounded investigation of immersion ingames. Extended abstracts of the 2004 conference on Human factorsand computing systems CHI 04, computings:1297, 2004.

[10] R.A. Calvo and S. DMello. Affect detection: An interdisciplinary re-view of models, methods, and their applications. IEEE Transactionson Affecitve Computing, 1(1):18–37, 2010.

[11] G. Chanel, J. Kronegg, D. Grandjean, and T. Pun. Emotion assess-ment: Arousal evaluation using EEGs and peripheral physiologicalsignals. Multimedia Content Representation, Classification andSecurity, pages 530–537, 2006.

[12] G. Chanel, C. Rebetez, M. Betrancourt, and T. Pun. Boredom,engagement and anxiety as indicators for adaptation to difficultyin games. In Proceedings of the 12th international conference onEntertainment and media in the ubiquitous era, pages 13–17. ACM,2008.

[13] T. Chisty and L.I. Kuncheva. A.M.B.E.R. Shark-Fin: An UnobtrusiveAffective Mouse. In ACHI 2013: The Sixth International Conferenceon Advances in Computer-Human Interactions, pages 488–495,2013.

[14] Jack Copeland. Colossus: The First Electronic Computer (PopularScience). Oxford University Press, USA, 1 edition, May 2006.

[15] Konami Corporation. Tokimeki memorial oshiete your heart. Arcade,1997.

[16] C. Darwin. The Expression of the Emotions in Man and Animals.John Murray, 1872.

[17] A. Dekker and E. Champion. Please biofeed the zombies: enhancingthe gameplay and display of a horror game using biofeedback. Proc.of DiGRA, pages 550–558, 2007.

[18] D.W. Douglas. Psychogalvanometer, April 1958.[19] A. Drachen and A. L. Pedersen. Correlation between heart rate,

electrodermal activity and player experience in first-person shootergames. Heart, pages 49–54, 2009.

[20] The Economist. Nintendo wakes up, 1996. Last accesed 23September 2013.

[21] (ESA). Essential facts about the computer and video game industry.ESA E3 2013, 2013. Last accessed 11 September 2013.

[22] S.H. Fairclough. Fundamentals of physiological computing. Inter-acting with Computers, 21(1-2):133–145, 2008.

[23] C. Fisher and P. Sanderson. Exploratory sequential data analysis:exploring continuous observational data. Interactions, pages 25–34,1996.

[24] B. Fulton. Beyond Psychological Theory: Getting Data that ImprovesGames. In Gamasutra, 2002.

[25] A. Gaggioli. Mantra mouse. gaggio.blogspirit.com/archive/2006/03/27/mantra-mouse.html, March 2006. last accesed 9 April 2013.

[26] N. Genes. We‘re Not Lying About Galvanic Skin Response. www.medgadget.com/2006/03/were\ not\ lying.html, March 2006. Lastaccessed 9 April 2013.

[27] M. Gerven, J. Farquhar, R. Schaefer, R. Vlek, J. Geuze, A. Nijholt,N. Ramsey, P. Haselager, L. Vuurpijl, S. Gielen, and P. Desain.The brain–computer interface cycle. Journal of Neural Engineering,6:041001, 2009.

[28] K.M. Gilleade and J. Allanson. A Toolkit for Exploring Affec-tive Interface Adaptation in Videogames Developing an AffectiveVideogame. Proceedings of HCI International, 2:370—-374, 2003.

[29] J. Golseth and C.C. LeGrand. Electronic diagnostic instruments, July1955.

[30] Starke R. Hathaway. Apparatus for measuring psychogalvanicresponses, Jan 1930.



39

9

[31] R.L. Hazlett. Measuring emotional valence during interactive expe-riences boys at video game play. In SIGCHI conference on HumanFactors, pages 1023–1026, 2006.

[32] J.A. Healey. Wearable and Automotive Systems for Affect Recog-nition from Phychiology, 2000.

[33] W. Holzer and A. Marko. Arrangement for recording variations inthe electrical resistance if the human body, January 1953.

[34] Y. Homma, I. Masaoka. Breathing rhythms and emotions. Experi-mental physiology, 93:1011–1021, 2008.

[35] E. Hudlicka. Affective game engines: motivation and requirements.In Proceedings of the 4th International Conference on Foundationsof Digital Games, pages 299–306. ACM, 2009.

[36] W. James. Mind association what is an emotion ? Oxford UniversityPresson behalf of the Mind Association, 9:188–205, 1884.

[37] C. Jennett, A.L. Cox, P. Cairns, S. Dhoparee, A. Epps, and A. Tijs,T. Waltson. Measuring and defining the experience of immersion ingames. International Journal of Human-Computer Studies, 66:641–661, 2008.

[38] C. Jones and A. Deeming. Affective human-robotic interaction.Affect and Emotion in Human-Computer Interaction, pages 175–185,2008.

[39] C. Jones and J. Sutherland. Acoustic emotion recognition foraffective computer gaming. Affect and emotion in human-computerinteraction: from theory to applications, 4868:209, 2008.

[40] Z Kaplan. Wii vitality sensor never released because it was’insufficient as a commercial product’. Nintendo World Report, July2013. Last accessed 11 July 2013.

[41] A. Kappas. Smile when you read this, whether you like it ornot: conceptual challenges to affect detection. Affective Computing,IEEE, 1(1):38–41, 2010.

[42] J.M. Kivikangas, I. Ekman, and G. Chanel. Review on psychophys-iological methods in game research. Proc. of 1st Nordic DiGRA,2010.

[43] K-E. Ko, H-C. Yang, and K-B. Sim. Emotion recognition usingEEG signals with relative power values and Bayesian network.International Journal of Control, Automation and Systems, 7(5):865–870, October 2009.

[44] R. Koller. Cardio-pneumo-electrodermograph, March 1953.[45] E.I. Konstantinidis, C.A. Frantzidis, C. Papadelis, C. Pappas, and

P.D. Bamidis. Wadeda: A wearable affective device with on-chipsignal processing capabilities for measuring electrodermal activity.XII Mediterranean Conference on Medical and Biological Engineer-ing and Computing, pages 276–279, 2010.

[46] K. Kuikkaniemi, I. Kosunen, M. Turpeinen, T. Saari, T. Laitinen,and P. Lieevonen. Designing Biofeedback for Games and PlayfulApplications. In CHI 2010, 2010.

[47] W. Liao, W. Zhang, Z. Zhu, Q. Ji, and W. D. Gray. Towards adecision-theoretic framework for affect recognition and user assis-tance. International Journal of Man-Machine Studies, 64(9):847–873, 2006.

[48] C. Liu, P. Agrawal, N. Sarkar, and S. Chen. Dynamic diffi-culty adjustment in computer games through real-time anxiety-based affective feedback. International Journal of Human-ComputerInteraction, 25:506–529, 2009.

[49] R.L. Mandryk and M.S. Atkins. A fuzzy physiological approachfor continuously modeling emotion during interaction with playtechnologies. International Journal of Human-Computer Studies,65:329–347, 2007.

[50] R.L. Mandryk and K.M. Inkpen. Physiological indicators for theevaluation of co-located collaborative play. In Proceedings of the2004 ACM conference on Computer supported cooperative work -CSCW ’04, pages 102—112, 2004.

[51] V.G. Mathison. Electropsychometer or bioelectronic instrument [v1],July 1954.

[52] V.G. Mathison. Electropsychometer or bioelectronic instrument [v2],February 1956.

[53] I.B. Mauss and M.D. Robinson. Measures of emotion: A review.Cognition & Emotion, 23(2):209–237, 2009.

[54] S. McQuiggan, S. Lee, and J. Lester. Predicting user physiologicalresponse for interactive environments: An inductive approach. 2ndArtificial Intelligence for Interactive, 2006.

[55] M.D. van der Zwaag, E.L. van den Broek, and J.H. Janssen.Guidelines for biosignal-driven HCI. CHI 2010, April 2010.

[56] D.D. Milne, J.P. Fagarty, and J.B. Duxlevie. Psychometer, January1944.

[57] B. Mittelmann and H.G. Wolff. Effective states and skin temperature:experimental study of subjects with ”cold hands” and raynaud’ssyndrome. Psychosomatic Medicine, 1:271–292, 1939.

[58] B. Mittelmann and H.G. Wolff. Emotions and skin temperature;observations of patients during psychotherapeutic (psychoanalytic)interviews. Psychosomatic Medicine, 5(3):211?231, 1943.

[59] L.E. Nacke. Affective ludology: Scientific measurement of userexperience in interactive entertainment. Thesis, 2009.

[60] M. Nichol. 100 words for facial expressions. Daily Writing Tips,2012.

[61] M.A. Nicolaou, H. Gunes, and M. Pantic. Continuous predictionof spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Transactions on Affective Computing, 2(2):92– 105, 2011.

[62] A.D. Norman. Emotional Design: Why We Love (or Hate) EverydayThings. Basic Books, 2003.

[63] R.J. Pagulayan, K. Keeker, D. Wixon, R.L. Romero, and T. Fuller.User-centered design in games. The human-computer interactionhandbook: fundamentals, evolving technologies and emerging appli-cations, pages 883–906, 2003.

[64] Burlyl R. Payne. Audible psychogalvonometer, March 1972.[65] P Petrantonakis and L. Hadjileontiadis. Emotion recognition from

EEG using higher-order crossings. IEEE Transactions on Informa-tion Technology in Biomedicine, 14(2):186–197, 2010.

[66] R. Picard and J. Healey. Affective wearables. Personal Technologies,1:231–240, 1997. 10.1007/BF01682026.

[67] R.W. Picard. Affective computing. The MIT press, 2000.[68] R.W. Picard. Emotion research by the people, for the people.

Emotion Review, 2(3):250, June 2010.[69] R.W. Picard, E. Vyzas, and J. Healey. Toward machine emotional

intelligence: analysis of affective physiological state. IEEE Trans-actions on Pattern Analysis and Machine Intelligence, 23(10):1175–1191, 2001.

[70] S. Poole. Trigger happy: Videogames and the entertainment revo-lution. Web Download Edition, 2007. Last accessed 11 September2013.

[71] P.A. Pour, T. Gulrez, O. AlZoubi, G. Gargiulo, and R.A. Calvo.Brain-computer interface: Next generation thought controlled dis-tributed video game development platform. IEEE Symposium OnComputational Intelligence and Games, pages 251–257, 2008.

[72] J.L. Raesler, C.J. Bills, and C.D. Osborne. Psychogalvanometer,January 1943.

[73] P. Rani, N. Sarker, and C. Liu. Maintaining Optimal Challengein Computer Games through Real-Time Physiological FeedbackMechanical Engineering. Pulse, 2005.

[74] N. Ravaja, T. Saari, M. Salminen, J. Laarni, and K. Kallinen. PhasicEmotional Reactions to Video Game Events: A PsychophysiologicalInvestigation. Media Psychology, 8(4):343–367, 2006.

[75] N. Ravaja, M. Turpeinen, T. Saari, S. Puttonen, and L. Keltikangas-Jarvinen. The psychophysiology of James Bond: phasic emotionalresponses to violent video game events. Emotion, 8(1):114–120,2008.

[76] C.J. Reynolds. The sensing and measurement of frustration withcomputers. PhD thesis, Massachusetts Institute of Technology, 2001.

[77] J.W. Ryan. Toy lie detector, March 1964.[78] T. Saari, N. Ravaja, J. Laarni, and M. Turpeinen. Towards Emo-

tionally Adapted Games based on User Controlled Emotion Knobs.Human Factors, 2005.

[79] J. Scheirer, R. Fernandez, J. Klein, and R.W. Picard. Frustratingthe user on purpose: a step toward building an affective computer.Interacting with Computers, 14(2):93–118, 2002.

[80] E Schrawzkopf and G. Wodtke. Electrocardiograph, August 1928.[81] K. Shelley and S. Shelley. Pulse oximeter waveform: photoelectric

plethysmography. W.B. Saunders, 2001.[82] J. Sherwood and R. Derakhshani. On classifiability of wavelet fea-

tures for EEG-based brain-computer interfaces. 2009 InternationalJoint Conference on Neural Networks, pages 2895–2902, June 2009.

[83] Uk Game Shows.com. The chair. Internet, August 2002.[84] T.C. Sottek. How valve’s steam box will reinvent the game console

as you know it. TheVerge.com, February 2013. last accessed 7thFebruary 2013.

[85] R.M. Stern, W.J. Ray, and K.S. Quigley. PsychophysiologicalRecording. Oxford university press, second edition, 2001.

[86] J. Sykes and S. Brown. Affective gaming. CHI 03 extended abstractson Human factors in computing systems CHI 03, page 732, 2003.

[87] M. Takagi. Instument for locating particular cutaneous points causedby viscero-vascular reflex, September 1965.



40

10

[88] K. Takahashi. Remarks on emotion recognition from bio-potentialsignals. In The Second International Conference on AutonomousRobots and Agents, pages 186–191. Citeseer, 2004.

[89] T.J.W. Tijs, D. Brokken, and W.A. Ijsselsteijn. Dynamic game bal-ancing by recognizing affect. In Proceedings of the 2nd InternationalConference on Fun and Games., pages 88–93, 2008.

[90] S. Tognetti, M. Garbarino, A. Bonarini, and M. Matteucci. Modelingenjoyment preference from physiological responses in a car racinggame. Computational Intelligence and Games (CIG), pages 321–328, 2010.

[91] H.W. Tygart. System for fm transmission of cardiological dataover telephone lines, February 1969. transmit physiological datafor remote analysis.

[92] W.M. Van Den Hoogen, W.A. IJsselsteijn, and Y.A.W. De Kort.Effects of Sensory Immersion on Behavioural Indicators of PlayerExperience: Movement Synchrony and Controller Pressure. di-graorg, 2009.

[93] P.H. Venables and M.J. Christie. Electrodermal Activity in Psycho-logical Research, chapter Chapter 1: Mechanisms, Intrumentation,Recording Techniques, and Quantification of responses, pages 1–124. ACADEMIC PRESS New York and London, 1973.

[94] C Vendel. The atari mindlink. www.atarimuseum.com, 2013. Lastaccessed 27 August 2013.

[95] H. Weidinger, G. Volbehr, W. Limpart, and H. Steusloff. Heartmonitoring device with filter for suppressing the frequency of anambient A.C. power source, May 1967.

[96] M.A.A.G. Wiethoff and E.M. Houwing. The value of psychophys-iological measures in human-computer interaction. In Proceedingsof the Fourth International Conference on Human-Computer Inter-action, volume 18, pages 661–665, 1991.

[97] Wikipedia. The chair (game show). Internet, december 2011.[98] G.F. Wilson, J.D. Lambert, and C.A. Russell. Performance enhance-

ment with real-time physiologically controlled adaptive aiding. Pro-ceedings of the Human Factors and Ergonomics Society, 1(13):61–64, 2000.

Thomas Christy was born in Caernarfon, Wales,UK. He obtained his BSc in Computer Science,Bangor University, in 2009. He won a Euro-graphics award for Best use of graphics, throughan advanced video game, utilising a haptic inputdevice. He is working towards his PhD at BangorUniversity, specialising in the field of AffectiveVideo Games. His work continues towards ad-vancing video game immersion through intuitivetechnological methods of human computer inter-action.



41

This article is distributed under the terms of theCreative Commons Attribution License whichpermits any use, distribution, and reproductionin any medium, provided the original author(s)and the source are credited.

Documents

Technological Advancements in Affective Gaming: A ... · Affective gaming has to move away from the conﬁdes of a laboratory and be deployed in a normal environment [13], [22], [68]