Affect in gamesCIG 2012 tutorial

Kostas KarpouzisNational Technical University of Athenskkarpou@cs.ntua.gr

what Ill talk aboutConcepts of affect/emotion/behaviour(hopefully!) related to gamesEngagement/attention/flowHow to detect/make sense of themModelling/detecting player experience and sattisfactionand adapting to

what I wont talk aboutDesigning affective NPCswhich show happiness, sadness, contempt, etc.Affect and in-game behaviourother than attention/engagemente.g. fatigue and game play, how happy/sad players perform, etc.Particular affect detection techniquesBut we can discuss those later

yours trulyPost-doc in Humaine Network of ExcellenceSince:CALLAS (emotion in arts and entertainment)Feelix Growing (affect-aware robots)Currently:Siren (serious games for conflict resolution in school environments)ILearnRW (serious games for children with dyslexia and dysorthographia)Also: Humaine Games SIG (w/ G. Yannakakis), IEEE TAC SI (Affect in Games)IEEE Aff. Comp TF, IEEE CIS Games TC TF on Education

affective gamesToys provide our first human-machine interface!Games are everywhere these days!Computers, consoles/TV, mobile phonesBrowser and Facebook/Google+ do not require installationFreemium business model attractive to both gamers and developersHuge interest in industry and academiaLots of funding from EU projects

affective gamesConstrained environment easier to track peopleNovel modalities: gestures, body movement, speech (lexical and prosody)Ideally associated with what happens in the game(Possibly) novel interaction paradigms, i.e. not WIMP

affective gamers

or (if things go wrong)

affective gamers

lets make use of it

a bit of theorytheories of emotion

terminologyEmotions, mood, personalityCan be distinguished bytime (short-term vs. long-term)influence (unnoticed vs. dominant)cause (specific vs. diffuse)Affect classified by timeshort-term: emotions (dominant, specific)medium-term: moods (unnoticed, diffuse)and long-term: personality (dominant)

terminologywhat we perceive is the expressed emotion at a given timeon top of a persons current mood, which may change over time, but not drasticallyand on top of their personalityusually considered a base line levelwhich may differ from what a person feelse.g. we despise someone, but are forced to be polite

terminologyAffect is an innately structured, non-cognitive evaluative sensation that may or may not register in consciousnessFeeling is defined as affect made conscious, possessing an evaluative capacity that is not only physiologically based, but that is often also psychologically oriented.Emotion is psychosocially constructed, dramatized feeling

how it all startedCharles Darwin, 1872Ekman et al. since the 60sMayer and Salovey, papers on emotional intelligence, 90sGolemans book: Emotional Intelligence: Why It Can Matter More Than IQPicards book: Affective Computing, 1997

why emotions?Shallow improvement of subjective experienceReason about emotions of othersTo improve usabilityGet a handle on another aspect of the "human world"Affective user modelingBasis for adapting experience to users

name that emotionso, we know what were afterbut we have to assign it a namein which we all agree uponand means the same thing for all (most?) of usdifferent emotion representationsdifferent contextdifferent applicationsdifferent conditions/environments

emotion representationsmost obvious: labelspeople use them in everyday lifehappy, sad, ironic, etc.may be extended to include user states, e.g. tired, which are not emotionsCS people like themgood match for classification algorithms

labelsbutwe have to agree on a finite setif we dont, well have to change the structure of our neural nets with each new labellabels dont work well with measurementsis joy

labelsEkmans set is the most popularanger, disgust, fear, joy, sadness, and surpriseadded contempt in the processMain difference to other sets of labels:universally recognizable across cultures when confronted with a smile, all people will recognize joy

from labels to machine learningwhen reading the claim that there are six facial expressions recognized universally across culturesCS people misunderstood, causing a whole lot of issues that still dominate the field

strike #1we can only recognize these six expressionsas a result, all video databases used to contain images of sad, angry, happy or fearful peoplea while later, the same authors discussed contempt as a possible universal, but CS people werent listening

strike #2only these six expressions exist in human expressivityas a result, more sad, angry, happy or fearful people, even when data involved HCIcan you really be afraid when using your computer?

strike #3we can only recognize extreme emotionsnow, happy people grin, sad people cry or are scared to death when afraidhowever, extreme emotions are scarce in everyday lifeso, subtle emotions and additional labels were out of the picture

labels are good, butdont cover subtle emotions and natural expressivitymore emotions are available in everyday life and usually maskedhence the need for alternative emotion representationscant approach dynamicscant approach magnitudeextreme joy is not defined

other sets of labelsPlutchikAcceptance, anger, anticipation, disgust, joy, fear, sadness, surpriseRelation to adaptive biological processes FrijdaDesire, happiness, interest, surprise, wonder, sorrowForms of action readiness IzardAnger, contempt, disgust, distress, fear, guilt, interest, joy, shame, surprise

going 2Dvertical: activation (active/passive)horiz.: evaluation (negative/positive)

going 2Demotions correspond to points in 2D spaceevidence that some vector operations are valid, e.g. fear + sadness = despair

going 2Dquadrants useful in some applicationse.g. detect extreme emotions during team comms in FPSs

going 3DPlutchik adds another dimensionvertical intensity, circle degrees of similarity four pairs of opposites

going 3DMehrabian considers pleasure, arousal and dominanceAgain, emotions are points in space

what about interaction?these models describe the emotional state of the userno insight as to what happened, why the user reacted and how the user will reactaction selectionOCC (Ortony, Clore, Collins)Scherers appraisal checks

OCC (Ortony, Clore, Collins)each event, agent and object has propertiesused to predict the final outcome/expressed emotion/action

summary on emotionperceived emotions are usually short-lasting events across modalitieslabels and dimensions are used to annotate perceived emotionspros and cons for eachadditional requirements for interactive applications

a bit of theorythe quest for Flow (and fun)

a theory of funRaph Kosterlead designer of Ultima Onlinecreative director of Star Wars Galaxieshttp://www.theoryoffun.com/theoryoffun.pdfhttp://www.raphkoster.com

a theory of fun

a theory of funWe talk so much about emergent gameplay, non-linear storytelling, or about player-entered content. Theyre all ways of increasing the possibility space, making self-refreshing puzzlesSo, what is it that makes a game fun?ever-lasting challengeHow can we keep challenges coming?

the concept of Flowa state of concentration or complete absorption with the activity at hand and the situation. It is a state in which people are so involved in an activity that nothing else seems to matter (Csikszentmihalyi,1990)Being completely involved in an activity for its own sake. The ego falls away. Time flies. Every action, movement, and thought follows inevitably from the previous one, like playing jazz. Your whole being is involved, and you're using your skills to the utmost

flow revisitedthe holy grail of game designjust the right amount of challengemaking a game very hardgamers quitmaking a game very easygamers bored

flow revisitedits not about the graphicsor the controlleror the franchise (e.g. sports games)just ask Roviomakers of Angry Birds$80M/yr, 600M dls

flow revisitedsmart games adapt to player skill and engagementkeeping them coming back for moreat the end of the day

back to the drawing boardwhat can we model? and how?definition of affective computingaffective computing is computing that relates to, arises from, or deliberately influences emotion or other affective phenomena -- Roz Picard, 1995a set of observable manifestations of a subjectively experienced emotion -- Merriam-Websters dictionary

observable manifestations

observable manifestations

hypothesisshallow treatmenti.e. not as far as personality, sticking to affectidentify/track user reactionsfacial expressions and gestures, body movements and stance, hand and body expressivity (for whole-body interaction)relate those to events in the game

hypothesisideally, we could identify the players stress level (via the observable manifestations) and their skill level (via their performance)and cluster those to identify player typesfor the particular game genre!or use them to adapt the gamemake it easier for players in distressor harder for players in the verge of boredom

hypothesiswhy bother with both affect and performance?why are players standing still?is it flow (immersion) or boredom?or why do they move around?is it immersion (e.g. in a racing game) or lack of engagement?remember: Flow skill AND engagement

generating contentwe want to make games harder or easier to match player skillpredefined levels (e.g. easy / expert)still we have to define what easy meansdata-driven activity!TGFKASM

generating contentgame level as a multi-parameter function produce game content procedurallye.g. number and size of gaps, number of opponents, etc.Multimodal player affect/satisfaction challenge in forthcoming CIGs

in a nutshellgames provide an ideal medium to induce and capture affective interactionswell-designed games bring out different (and valuable!) reactions from playersgaming is a multi-faceted activitythus, player models are usually detailedplayer affect tells us a lot about the game

lessons learnedthe Siren project

conflict resolution gamesSiren aims to produce a conflict resolution serious gamefor 10-14 y.o. childrenin school environments

conflict resolution gamesThe life cycle of conflict (Swanstrom and Weissmann, 2005)

conflict resolution gamesduring escalation, negative emotions are presentcannot use neg. emotions to indicate stress adaptation

conflict resolution gamesrather, use estimated emotion to identify where players are in this figure (which phase)

conflict resolution gamesand produce content to push users towards de-escalationlearning objective of the game!

conflict resolution gamessensed affect can be used to identify player performancei.e. whether players actually move towards resolving the conflictbut which emotions are relevant?negative vs positiveis that enough for all game genres?

in a nutshellplayer affect is genre-dependentreflects many qualities from the user modelmany open research questionssingle- vs multi-playereasy to find people to play gamesyay!

thank you!

Kostas Karpouziskkarpou@cs.ntua.gr

******************************************************************