Games: Sound and music for Interactivity and Immersion...2 Today g Main modules A. Sound and music for games • Different functions of sound and music in games (recap) • Challenges

1

Games: Sound and music for Interactivity and Immersion

Anja Volk

Sound and Music Technology,Nov 21, 2017

2

Today

g Main modulesA. Sound and music for games

• Different functions of sound and music in games (recap)• Challenges of interactivity and immersion• Music and emotion

3

Recapitulation last lecture

g Functions of sound and music in film and games

g Annabel Cohen (1999) for film: e.g.• Provides continuity between shots• Induces mood• …

g additional function for games• support interactive aspects of the game play

4

Recapitulation last lecture

g Dual nature of games:g Virtual world

• Sound and music: support reality status of the game world (such as supporting sense of presence)

g User system• Sound and music: important part of usability system, needs to be

informational and communicative

g Challenge for music and sound design for games: combination of the two functions

5

Game audio typologies

g Recapitulation film music: diegetic vs. ?g What was it about?

6


g Recapitulation film music: diegetic vs. non- (extra-) diegeticg Diegetic sound: originates from a source existing within the film

or game worldg Extradiegetic: no connection to an actual source within the film

or game world

g Application to games (Jørgensen, 2009): diegetic vs. ? g difference to extradiegetic in film?

7


g Recapitulation film music: diegetic vs. non- (extra-) diegeticg Diegetic sound: originates from a source existing with the film

or game worldg Extradiegetic: no connection to an actual source within the film

or game world

g Application to games (Jørgensen, 2009): diegetic vs. transdiegeticg Extradiegetic: can provide game player with information that

changes players’ actions, which in turn can change the game world (therefore “transdiegetic”)

8

Recapitulation: IEZA model

g Huiberts & van Tol (2008)g Conceptual design tool for game and audio design students at

HKUg Applies to game audio during the interactive game play

GMG!%:3!&Q_#!;>23=!<>4!0627>!71!@0;39!

!G`!

!

(69;G=$TH$)$G=4G=>=85356:8$:D$5I=$,#M)$K:<=CB$

#DD=@5$

W:31!8:3!273@387A!9723!><!&Q_#!79!3R0;7132E!8X>!2>;0719!A01!.3!

279871@679:32M! %:3! <7498! 2>;071E! 10;32! O..$!&E! A>180719! 9>612!

>.U3A89!8:08!043!J34A37F32!09!.371@!J4>26A32!./!>4!08847.6832!8>!

9>64A39! 8:08! 3R798! X78:71! 8:3! @0;3! X>4=2M! d/! A>;;617A0871@!

Q<<3A8!8>!8:3!J=0/34E!8:3!2397@134!71A>4J>40839!9>6129!.3=>1@71@!

8>!719801A39!012!0A87>19!X78:71!8:3!X>4=2!><!8:3!@0;3M!*>6129!><!8:3!Q<<3A8!2>;071!A01!

.3! 378:34! >1L9A4331! B7M3M! F797.=3D! >4! ><<L9A4331E! ;30171@! J439318! 71! 0627>! >1=/E! 1>8!

0AA>;J01732! ./! F7960=9! Bd>42X3==! f! %:>;J9>1E! Ia]`E! JM! IaGDM! ">;;>1! 3R0;J=39! ><!

8:79!2>;071!71!A644318!@0;39!043!8:3!9>6129!><!8:3!0F0804!3M@M!<>>8983J9E!.4308:71@E!8:3!

270=>@63!><!27<<34318! A:040A8349E!X30J>1!9>6129!96A:!09!@619:>89!012!9X>429E!F3:7A=3!

9>6129E!012!A>==7271@!>.U3A89M!%:393!3R0;J=39!.3=>1@!8>!@0;39!8:08!<308643!0!47A:!F74860=!

X>4=2E!.68!9>6129!.3=>1@71@!8>! 8:3!Q<<3A8!2>;071!043!0=9>!9331! 71! =399!430=7987A!@0;39E!

<>4!3R0;J=3E! @0;39!96A:!09!M$&6-,* B)0;3d>/E!Ia]aDE!P$I* BGHHGD!012!9$=*<'%$6*Q(6-"*

26",* BGHH[DM! %:3! =08834! <3086439! >1=/! 0! <3X! 90;J=39! ><! 9J33A:! B8:08! ><! 8:3! A:040A8349!

^047>!012!b67@7D!X:7=3!8:3!4398!><!8:3!0627>!A>197989!><!9/18:397932!j.=33J9Ei!j.33J9i!012!

jJ=71@9Mi!%:393!9/18:387A! 9>6129!43<34! 8>! 8:3!0A87F78/!><! 8:3!0F0804!^047>!012!3F3189!>4!

9

IEZA model: two functionalities

g two main perspectives on the expression of in-game audio aimed at the player's experience:g Audio used for optimizing game play: helping the player to play the

game by providing necessary gameplay information; serves usabilityg Audio used for dynamizing game play: making the gameplay

experience more intense and thrilling

GMV!%X>!<61A87>10=78739!><!@0;3!0627>!012!&Q_#!

!VI!

GHH[m!">==719E!GHH\.DM!%:3!2/10;7971@!4>=3!A01!:0F3!0!J>9787F3!71<=631A3!>1!7;;3497>1!

09!78!79!07;32!08!;0S71@!8:3!@0;3!;>43!987;6=0871@M!!

W:31! 8:393! 8X>!J349J3A87F39! 043! 43=0832! 8>! &Q_#E! 78! 9:>X9! 8:08E! 6960==/E! 8:3! &1834<0A3!

2>;071! 79!;071=/! >4731832! 8>! 8:3! >J87;79087>1!><! 8:3! @0;3! <>==>X32!./!Q<<3A8! 2>;071!

012!=098=/!8:3!2>;0719!8:08!<>4;!8:3!*38871@M!&1!>8:34!X>429E!78!79!@31340==/!;>43!27<<7A6=8!

8>!A>;;617A083!<0A860=!71<>4;087>1!X78:!8:3!*38871@!9723M!%:3!2/10;79087>1!><!8:3!@0;3!

79!J471A7J0==/!2>13!./!8:3!#<<3A8E! <>==>X32!./!8:3!_>13M!%:3!#A87F78/!;>98=/!A>1847.6839!

=399!0A87F3=/!8>!8:79!<61A87>10=78/M!&1!<7@643!YE!8:393!8X>!J349J3A87F39!043!43J43931832!71!

8:3!&Q_#!;>23=M!!

! !

(69;G=$ UH$ !I=$ 5R:$ 4=G>4=@567=>$ :8$ 5I=$ D;8@56:83C65J$ :D$ 93K=$ 3;<6:V$ :456K6>356:8$ 38<$

<J83K6>356:8V$G=4G=>=85=<$68$5I=$,#M)$K:<=CB$2=8=G3CCJV$5I=$,85=GD3@=$<:K368$6>$4G6K3G6CJ$

36K=<$35$@:87=J689$68D:GK356:8B$)DD=@5$6>$4G6K3G6CJ$;>=<$D:G$5I=$>564;C356:8$:D$3$K::<V$5I;>$

K3S689$5I=$93K=$K:G=$685=8>=B$

10

Today


• Different functions of sound and music in games• Challenges of interactivity and immersion• Music and emotion

11

Important historic milestones(according to Jørgensen, 2009)

g First Nintendo consoleg storage: primary concern for computer gamesg sound: only small amount of storage spaceg only a few sounds could be played simultaneously, usability

functions became prioritizedg Nintendo’s classic Super Mario Bros. games: use of sound for

response and urgency purposes

g Stereo introduced in 1980sg new functionalities: sound and music with high precision

compared to real world sounds

g 1990s: sound cards could produce CD quality soundg auditory focus on creating believable virtual environments g new uses of game audio beside the usability function,

connected to orientation and the sense of presence.

12

g Towards modern times:g inclusion of CD-ROM allowed heavier applications with

more sound and graphics g functionality, fidelity, sense of presence, atmosphere still

important, but came to merge with the use of sound for usability purposes

g E.g. games utilized stereo speakers to create the sense of three-dimensionality in gamy by making the sound move from one speaker to the other

g E.g. true surround systems which allowed the positioning of different channel speakers around the player in real space

Important historic milestones(according to Jørgensen, 2009)

13

An important drawback

musical content

example compare image

compare text

structure

convert to above

convert to below

Digital audio (MP3, Wav)

level 1: primitive features

speech none - hard

Time-stamped events (MIDI)

level 2: objects

text little easy fairly hard (OK job)

Music notation (Finale, Sibelius, MusicXML)

level 2:compound objects

text + markup

much easy(OK job)

-

14

An important drawback

g Greater level of interactivity requires more flexibility of music; a more granular (note) level control over the music necessary

g Composers work with MIDI sequencers and a sampling engine; by rendering out to a wav file, we are essentially creating the “one-track-orchestra” (every possible variation requires to render out a separate file).

Stevens and Reybold (2013): Game audio tutorial

15

Challenges for Interactivity

g In a surprising discussion, Sony's Jason Page admitted that the future of video game music may be based in technology that's considered antiquated by most: MIDI. Many developers have been moving away from MIDI to pre-recorded, orchestrated background music, but in the process, the interactivity of game music has been sacrificed.

g The incredible processing power of the next-generation consoles has changed the rules of MIDI … MIDI samples used by consoles can be just as good, if not better, than the samples used on dedicated synthesizers. Because MIDI loads in real-time, it retains the interactivity that composers like Koji Kondo would need, and it would allow games to load more quickly.

http://www.engadget.com/2007/03/08/midi-is-the-future-of-game-audio/#comments

16

Challenges for Interactivity

g adaptive music (Brandon 2004, Whitmore 2003)g technique to make music change according to the

player’s behavior and actions in game.

17

Forms of Interactivity(Michael Liebe, 2013)

g Linear musicg firmly coupled to certain game-immanent elementsg cannot be influenced by playersg Examples: soundtracks

g Reactive musicg music is directly connected to actions of the playersg triggered by specific actions (e.g. location change, begin a

fight …)g Often in role-playingg Example of entirely driven by player: Elektroplankton

https://www.youtube.com/watch?v=ttFoK8BTXM4

g Proactive musicg prompts the players to undertake a specific action when it is

playedg Dance Dance Revolution: players follow rhythmic patternsg https://www.youtube.com/watch?v=NqD-B8t_6us#t=100s

18

How to achieve interactive music?

g Michael Sweet (Composer): Typical techniques

g Method 1: Vertical Remixing (Layering)g Method 2: Horizontal Re-sequencing

based on pre-composed stems of music to create interactivity

19




20




Time

Height (Pitch

21




g Cross-fadingg Phrase-branchingg Musical demarcation branchingg Bridge transitiong Stinger-based sequencing

22


Michael Sweet, Composer

g Method 1: Vertical Remixing (Layering)

g break a music cue into two or more musical layers (e.g. instruments)

g control-inputs: which game events trigger the layers to enter and exit?

g the more layers, the greater the diversity

g often just two layers used

g length, tempo, and harmonic framework the same

g Example: Fall out New Vegas https://www.youtube.com/watch?v=JR38Yn9qxkQ

23



g Method 1: Vertical Remixing (Layering)g Advantages:

g Immediate changes to music based on a game eventg Less impactful than switching to an entirely new music cue,

(change is more subtle)g Disadvantages:

g Musical phrases easily interrupted (e.g. melody is faded in or out in the middle of a phrase)

g fading in or out layers can sound non-musicalg No change in tempo or harmonic structure possible

24



g Method 1: Vertical Remixing (Layering)g Advantages:

g Immediate changes to music based on a game eventg Less impactful than switching to an entirely new music cue,

(change is more subtle)g Disadvantages:

g Musical phrases easily interrupted (e.g. melody is faded in or out in the middle of a phrase)

g fading in or out layers can sound non-musicalg No change in tempo or harmonic structure possible

g When used:g when state changes from one state to another shorter than 30

secondsg puzzles within game: Completion of each phase of the puzzle

might bring in a new layer of the music indicating to the player that he’s progressing

25



g Method 2: Horizontal Re-sequencingg Cross-fading

• one music cue fades out while another music cue fades up• Example: World of Warcraft https://www.youtube.com/watch?v=9-

JKJMnH8wM• Advantages

– Easy to compose and implement into a game.– Immediate changes to the music based on a game event.– Ability to completely change the tempo, harmony, instrumentation, or

melody instantly based on a game event• Disadvantages

– The least musical of all adaptive techniques (changes abrupt)– Musical phrases are often interrupted in the middle of a phrase.– No accounting for the tempo or key changes when switching from one

musical cue to another.• When used

– Very often (is simple)– bad method to use if the cues switch more often than 30 seconds

because of the constant interruption to the player

26



g Method 2: Horizontal Re-sequencingg Phrase-branching

• waits for the current musical phrase to end before playing the next musical cue; https://www.youtube.com/watch?v=C58TuhQPHNc

g Advantages• Most musical of all the horizontal re-sequencing techniques, never

interrupts a musical phrase.• Ability to change tempo, harmony, instrumentation or melody in

the next phrase based on a game event.g Disadvantages

• Non-immediate musical change because the music change will waituntil the end of the current phrase which is dependent on the length of the phrases.

• Can be more disruptive to the player in terms of musical changes than vertical remixing.

g When used• suited for styles/genres whith shorter phrase lengths such as rock

and techno; longer phrase lengths will delay the entrance of the next music cue.

27



g Method 2: Horizontal Re-sequencingg Musical demarcation branching

• Allows music cue to switch at a musical demarcation point such as a beat, or measure

28



g Method 2: Horizontal Re-sequencingg Musical demarcation branching

• Allows music cue to switch at a musical demarcation point such as a beat, or measure

• Advantages– More musical than cross-fading.– Faster changes than phrase branching.– Ability to change tempo, harmony, instrumentation or melody in the

musical demarcation point based on a game event.• Disadvantages

– Non-immediate musical change because the music change will wait untilthe next demarcation point.

– Musical phrases can be interrupted.– Can be more disruptive to the player in terms of musical changes than

vertical remixing.

g Example: https://www.youtube.com/watch?v=oMVedBq6H24

29



g Method 2: Horizontal Re-sequencingg Bridge transition

• short musical cues used to connect one musical cue with anotherfor more seamless transitions

Musical Bridge:http://www.youtube.com/watch?v=hE_qOY5GkH0&t=1m22s

30



g Method 2: Horizontal Re-sequencingg Bridge transition

• short musical cues used to connect one musical cue with anotherfor more seamless transitions

• Advantages– Ability to link to disparate music cues in terms of tempo, harmony, and

instrumentation.– Ability to change tempo, harmony, instrumentation or melody based on

a game event.• Disadvantages

– Musical phrases can be interrupted.– The length of the bridge transition pushes the beginning of the next cue

later making it more difficult to do another change until after the next cue begins.

– Can be more disruptive to the player in terms of musical changes thanvertical remixing.

– If the same bridge transition is heard frequently, it can be repetitive tothe player.

g Example: https://www.youtube.com/watch?v=jZqaEZjxqLo

31


Michael Sweet, Composerg Method 2: Horizontal Re-sequencing

g Stinger-based sequencing• a series of stingers which are played back based on game events• player triggers these stingers individually based on game events• Stingers may overlap• do not have a connecting rhythmic framework, are composed

primarily of crescendos and accents with silence in-between• Advantages

– Stingers are usually separated by silence, so they tend to work well musically together.

– Immediate punctuation of a game event.– Ability to link to disparate music cues in terms of tempo, harmony, and

instrumentation.• Disadvantages

– No tempo map or rhythmic framework linking the music together, e.g. can feel like disparate elements.

– Can feel close to the film scoring cliche ‘Mickey-Mousing’.– Phrase lengths are heavily dependent on how the game is dramatically

scripted. Example:

g Example: https://www.youtube.com/watch?v=djI3FFM6oVM

32



g Writing interactive musicg http://interactivemusicbook.com/wp/

Other example

Stevens & Reybold (2013)The Game Audio Tutorial: A Practical Guide to Sound and Music for Interactive Games.

33

Another example for interactivity

g adaptive music (Brandon 2004, Whitmore 2003)g technique to make music change according to the player’s

behavior and actions in game.

g One technique to achieve that: leitmotifs

34

Leitmotifs

g Music is extremely repetitive

g Famous example

http://philomel.com/asa156th/mp3/Sound_Demo_1.mp3

http://philomel.com/asa156th/mp3/Sound_Demo_2.mp3

35

Leitmotifs


g Famous example: Diana Deutsch’s “Speech to Song Elusion”; for more information check out:

http://deutsch.ucsd.edu/psychology/pages.php?i=212

36

Leitmotifs


g Convenient for automatic composition, example

g Used for loops in games, but no interactivity

http://tonematrix.audiotool.com

37

Leitmotifs

g Music makes use of non-literal repetition, called variation

http://www.youtube.com/watch?v=QYrZZ68zhSs

http://www.youtube.com/watch?v=SKMVLKmgxAw

38

Leitmotifs

g Leitmotif:

g attached to a person, object, place, idea, state of mind, supernatural force or any other ingredient in a dramatic work

g a theme, or other coherent musical idea, clearly defined so as to retain its identity if modified on subsequent appearances

Grove Music Online

39

Leitmotifs

40

Leitmotifs

The inventor: Richard Wagner

41

Leitmotifs

http://ringcycle.metoperafamily.org/leitmotifs

42

Leitmotifs

g widely used in film music, e.g. LOTRg Leitmotif of “The Fellowship” in different variations:

https://www.youtube.com/watch=qxc0Z6zZcto&list=PLHmmG49wjXiCFYH5lmCtmBHDwB6REejl9&index=3

43

Leitmotifs

g Example Game: Final Fantasy 7

http://videogamemusicnerd.blogspot.nl/2012/11/analysis-final-fantasy-vii-leitmotifs.html

45

Leitmotifs

g Leitmotif challenge: how to do this automatically such that music adapts to player’s actions?g Experimentation projectg Automatic leitmotif generationg Challenging: not every motif is suitable for each algorithmg Challenging: aesthetic value

46

Today


• Different functions of sound and music in games (recap)• Challenges of interactivity and immersion• Music and Emotion

47

Immersion

g no generally accepted definition of term immersion, instead different descriptions in literature, e.g.

g the experience of losing a sense of embodiment in the present whilst concentrating on a mediated environment

g losing track of immediate physical surroundingsg being transported into the game worldg being absorbed in the activity g being identified with the situation or a character of the game

Huiberts (2010), p. 36

48

Immersion:some systematic aspects by Sander Huiberts

g Sander Huiberts, PhD

g Three basic aspects of game immersiong 1- Being transported into the game worldg 2- Absorption in the activityg 3- Identification with the situation or a character of the game

49



g Three basic aspects of game immersiong 1- Being transported into the game world

• e.g. sensory features stimulate the feeling of being there

g 2- Absorption in the activity• e.g. challenge-based immersion

g 3- Identification with the situation or a character of the game• e.g. imaginative immersion• e.g. emotional responses: increasing the empathy through specific

mood

50



g Three basic aspects of game immersiong 1- Being transported into the game world

• e.g. sensory features stimulate the feeling of being there

g 2- Absorption in the activity• e.g. challenge-based immersion

g 3- Identification with the situation or a character of the game• e.g. imaginative immersion• e.g. emotional responses: increasing the empathy through specific

mood

51

Strong experiences related to music

Strong experiences related to music: A descriptive systemALFGABRlELSSON AND SlY LINDSTROM WIK

But it was not until the second half of the performance mat me mystery and thepower (4.7) really gripped me (4.3). I was filled by an enormous warmth and heat (2.1).I really swallowed all me notes (2.3) mat were streaming out in me air, nor a single note,effect or sequence missed my hungry ears (3.7). The music became so distinct (3.7). I wascaptivated by each of me instruments and what they had to offer me (3.7). Nothing elseexisted (4.1)!I was dancing, whirling (2.2) and really gave myself up to the music and the

rhythms (4.4), overjoyed (5.2) -laughing (2.2). Tears came into my eyes (2.1) -however suange that may seem-and it was as a further sign, some kind of liberation. Themusic set me free from my sober everyday life (7.1). Now I could let my body partsdance as freely as theywanted (2.2) - just let them follow me rhythms and totally losecontrol (4.3).The music danced around like a whirlwind (4.7) in me narrow room and all the

dramatic sentiment mat it conveyed (4.7) reflected my own situation in life (7.3) - butin a new way chat I liked (5.2). I went hand in hand with me music (4.4). Afterwards Iremained standingflushed with joy (5.2), as if intoxicated (5.2), and it was a real kickfor me (7.1). I felt religious (6.3)-and me music was my god.Before I was in a very bad state. Depressed. It was during me most critical time ever in

my life. I found it hard to get on with people and had to really exert myself to be ablero get to grips with things. Afterwards I was bouncy, giggling, lively and filled with deepjoy (5.2). It was really about a life-kick (7.1). There was also a kind of religious feeling. Itwas so bewildering mat it almost felt as a salvation (6.3). It was as if I was selected to takepart in such an experience when I needed it me most (7.3). I was filled by zest for life andsuddenly had quite anomer view of me grey tomorrow (6.1, 7.1).The period after me experience was very important. It helped to loosen all knots mat I

had within me so chat instead of going even further inro my depression I started to climbout of it.

RqJort B.Late summer day, evening. Some friends and I were going to listen to me band.called Pink Floyd. The huge arena was soon filled with thousands of people (3.2), merewas room for 30000. Now begins this concert mat has been so eagerly awaited and I getgoose pimples all over me body (2.1). They have fantastic equipment, you can perceiveevery little detail in the music (3.1, 3.8). Our place in me middle meant mat we heardthe music and the effects from every side, frombehind, to the right, to the left (3.1).They played the good old pieces mixed with new ones (3.8). I recognised them (4.5) andcould stand there rocking and joining in singing (2.2). I felt one with me music (4.4). Ishivered (2.1).When they began playing again after the intermission darkness had fallen over me arena

and theymade great use of light effects, laser beams that 'went out into me universe' (3.2).Yes, it actually felt so, everything became so infinite and I was one with it (6.2).This is me first time mat I am so involved in me music (4.4). I feel how me base comes

in from me ground via me sales of my feet, continues up through my calves, thighs, me

'67

Someone listening to a band playing Finnish tango in a pub

g Empirical study by Gabrielsson & Wik (2003)g 900 people reporting on strong experiences of music

52

Someone listening to Pink Floyd in a concert

Strong experiences related to music: A descriptive systemALFGABRlELSSON AND SlY LINDSTROM WIK

But it was not until the second half of the performance mat me mystery and thepower (4.7) really gripped me (4.3). I was filled by an enormous warmth and heat (2.1).I really swallowed all me notes (2.3) mat were streaming out in me air, nor a single note,effect or sequence missed my hungry ears (3.7). The music became so distinct (3.7). I wascaptivated by each of me instruments and what they had to offer me (3.7). Nothing elseexisted (4.1)!I was dancing, whirling (2.2) and really gave myself up to the music and the

rhythms (4.4), overjoyed (5.2) -laughing (2.2). Tears came into my eyes (2.1) -however suange that may seem-and it was as a further sign, some kind of liberation. Themusic set me free from my sober everyday life (7.1). Now I could let my body partsdance as freely as theywanted (2.2) - just let them follow me rhythms and totally losecontrol (4.3).The music danced around like a whirlwind (4.7) in me narrow room and all the

dramatic sentiment mat it conveyed (4.7) reflected my own situation in life (7.3) - butin a new way chat I liked (5.2). I went hand in hand with me music (4.4). Afterwards Iremained standingflushed with joy (5.2), as if intoxicated (5.2), and it was a real kickfor me (7.1). I felt religious (6.3)-and me music was my god.Before I was in a very bad state. Depressed. It was during me most critical time ever in

my life. I found it hard to get on with people and had to really exert myself to be ablero get to grips with things. Afterwards I was bouncy, giggling, lively and filled with deepjoy (5.2). It was really about a life-kick (7.1). There was also a kind of religious feeling. Itwas so bewildering mat it almost felt as a salvation (6.3). It was as if I was selected to takepart in such an experience when I needed it me most (7.3). I was filled by zest for life andsuddenly had quite anomer view of me grey tomorrow (6.1, 7.1).The period after me experience was very important. It helped to loosen all knots mat I

had within me so chat instead of going even further inro my depression I started to climbout of it.

RqJort B.Late summer day, evening. Some friends and I were going to listen to me band.called Pink Floyd. The huge arena was soon filled with thousands of people (3.2), merewas room for 30000. Now begins this concert mat has been so eagerly awaited and I getgoose pimples all over me body (2.1). They have fantastic equipment, you can perceiveevery little detail in the music (3.1, 3.8). Our place in me middle meant mat we heardthe music and the effects from every side, frombehind, to the right, to the left (3.1).They played the good old pieces mixed with new ones (3.8). I recognised them (4.5) andcould stand there rocking and joining in singing (2.2). I felt one with me music (4.4). Ishivered (2.1).When they began playing again after the intermission darkness had fallen over me arena

and theymade great use of light effects, laser beams that 'went out into me universe' (3.2).Yes, it actually felt so, everything became so infinite and I was one with it (6.2).This is me first time mat I am so involved in me music (4.4). I feel how me base comes

in from me ground via me sales of my feet, continues up through my calves, thighs, me

'67spine (3.3) and I am filled by the music (2.3). Boundaries are wiped out. I am one withrhe universe (6.2). The music dissolves all boundaries, just like what I have understood itmeans to be psychotic (4.5, 6.2). When they play my favourite tune "Learning to fly" tearsbegin trickling (2.1). I have always thought that it is ridiculous when girls howled at theBeatles (4.5). Now I am in the same situation myself, albeit not so hysterical- No, I amstanding there very calm (5.2), rocking in time with the music (2.2), feeling whole (4.2,7.1) and just letting tears trickle down my cheeks (2.1).The concert approaches the end, and I feel dislike (5.3) as I alwaysdo when everything

is well. I want to keep it (7.1). The end became the climax when they let off one of thebiggest fireworks that I have ever seen (3.2). Like a child on Christmas Eve I stood therewith open mouth, laughed and clapped my hands (2.2). What an experience (Ll)!

Repor: C One piece of music which has given me a suong experience is the Adagimo inGustav Mahler's Fifth Symphony. The day this experience occurred I was alone at home,after school in the afternoon. I was gloomy and broody (5.3) as I usually am. Mahler'sFifth Symphony, and especially the Adagimo, is one of my favourite pieces ofmusic. Themusic fills oneself with a kind of intoxication (5.2) or makes you shiver (2.1) as everychord goes through you like a wave (2.3). You lose all feeling of time and space, the wholeroom in which I am lying begins to whirl (4.2). In the last measures' fortissimo a lightseems to pass over my closed eyes (2.2, 3.6) which goes out more and more in thefollowing diminuendo.Afterwards I remain lying for some minutes (2.2) and let the music sink down and back.

This period afterwards is the only one when I feel calm and satisfied with myself and theworld around (5.2). My own problems and the burdens of personality disappear (7.1.) atthe same time as it feels as if I've found myself (7.1).When I listen to music I always try to translate my experiences or the composer's

message into words or ideas - small words on which I meditate (4.5). In the Ariagimomy thoughts and feelings are about longing. death, God, eternal bliss (4.5, 6.1, 6.3). Themusic and my feelingsmake me understand (at least emotionally) the ununderstandablebut at the same time the elementary (7.1). Things which I usually doubt reveal themselvesas truths through the power ofmusic (7.1). The music becomes a confirmation of my ownlonging (7.3) and my doubt or faith (6.3). The music also reveals another life, anotherexistence (6.2) which I hope will be mine; an existence where you are satisfied withyourself, live in harmony with the world around, where it is peaceful and where love is thesource of life (6.2).

Repor:D (Performer). For severalyears I have played rogether with two musicians, XX andYY. We playa type of music that is sometimes designated as improvisation but is rather akind of "instant composition" that gets its sound and form directly at the same moment.All three of us have a background as both musicians and composers and have played ina kind of workshop in which we have elaborated structures, conceptions of form,emotional states etc which have been stored in the memory and are released and further

168


53

g Empirical study by Gabrielsson & Wik (2003)g 900 people reporting on strong experiences of music

g complete absorptiong one did not think about where one was or for how long this

would lastg I was totally caught in the experience g everything around me did not exist g time and space ceased to existg I dreamed myself away


54

How can we create music for games…

g … that induces these strong experiences and contributes to immersive experience of games?

55

Challenge

g How is music inducing these experiences and emotions?

g What do we know from the scientific perspective?• Juslin & Vjästfäll (2008); Juslin, Harmat, Eerola (2013)

56

How is emotion induced?g (1) brain stem reflex

g (2) rhythmic entrainment

g (3) evaluative conditioning

g (4) contagion

g (5) visual imagery

g (6) episodic memory

g (7) musical expectancy

g (8) aesthetic judgment

BRECVEMA

57

BRECVEMA

g (1) brain stem reflexg hard-wired attention responseg acoustic features (e.g. extrem loudness or speed)g Prepares body to reactg quick and automatic

58

BRECVEMA

g (2) rhythmic entrainmentg Adjustment of internal body rhythm to external rhythm in

music

59

BRECVEMA

g (3) evaluative conditioningg Certain music associated with certain eventsg Memory plays important roleg conditioned association

60

BRECVEMA

g (4) Contagiong listener perceives the emotional expression of the music, and

then “mimics” this expression internallyg Induction of the same emotion

61

BRECVEMA

g (6) Episodic memoryg an emotion is induced in a listener because the music evokes

a memory of a particular event in the listener’s life g “darling, they are playing our tune”

62

BRECVEMA

g (7) Musical expectancyg emotion is induced because a specific feature of the music

violates, delays, or confirms the listener’s expectations aboutthe continuation of the music

63

BRECVEMA

g (8) Aesthetic judgmentg subjective evaluation of the aesthetic value of the music

based on an individual set of weighted criteria

64

Examples in game musicg (1) brain stem reflex

g (2) rhythmic entrainment

g (3) evaluative conditioning

g (4) contagion

g (5) visual imagery

g (6) episodic memory

g (7) musical expectancy

g (8) aesthetic judgment

BRECVEMA

73

Summary of important points

g Challenges for interactivity and immersiong How to achieve interactive music?

g Leitmotifs

g Emotion and immersion

g BRECVEMA-model on emotion

74

Literature

g Gabrielsson & Wik (2003), Strong experiences related to music, Musicae Scientiae, Vol. 7, No. 2.

g Juslin, P. N., Harmat, L. & Eerola, T. (2013). What Makes Music Emotionally Significant? Exploring the Underlying Mechanisms. Psychology of Music 42(4): 599-623.

g New Grove Online: Leitmotif

g Michael Sweet: Writing interactive musicg http://interactivemusicbook.com/wp/

g Michael Liebe (2013): Interactivity and Music in Computer Games. In: Music and Game, (ed. P. Mohrmann), Springer.

Documents

Games: Sound and music for Interactivity and Immersion...2 Today g Main modules A. Sound and music for games • Different functions of sound and music in games (recap) • Challenges