Slide 1 (of 45)

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance

Steven R. Livingstone(BSc. BInfTech)

Slide 2 (of 45)

Problem Statement

There exists no automated method to detect and influence the emotions of music

An audience’s response to computer music has previously been inaccessible to computers and thus lost

Slide 3 (of 45)

Hypothesis

Perceived emotional content of music can be influenced by controlling both the structural and performative aspects

Audience feedback can be captured by computers to tailor the musical experience

Slide 4 (of 45)

Methodology

Score – Manipulate the structure and mark up with emotional performance metadata

Audience – Determine emotional state and attitudes using affective computing tech.

Architecture – Bring together for an awareness of score and audience

Slide 5 (of 45)

Talk Overview

1) Introduction - In

2) Research and Contribution - Re

3) System & Testing - Sy

4) Future Work - Fu

5) Summary - Su

Slide 6 (of 45)

Why Emotions?

Principal target is the computer game

Emotional Narrative is the key to game enjoyment

In 1

Slide 7 (of 45)

Why Music?

Music is a universal human trait and found everywhere [a]: cinema, television, radio, commercials, ballet,

shopping centres, transport, waiting rooms, restaurants …

Within any waking 2 hour period, a person has a 44% chance of experiencing a musical event [b]

In 2

Slide 8 (of 45)

Why Gaming Music?

Gaming music is important for: emotion, interest and information

Game music static within scenes, unlike cinema music

In 3

Slide 9 (of 45)

How are we going to do it?

Cross-disciplinary approach to research Study of emotion Music psychology Empirical Analysis

Bring this knowledge together into a computing framework

In 4

Slide 10 (of 45)

Emotion - History

1600s – Primary Emotions (Descartes)

1800s - Biological reaction mechanisms (Darwin)

1800/1900s – Perception Physiological Response Emotion (James-Lange)

1960s – Cognition and Appraisal Theory (Arnold and Lazarus)

1990s – Somatic Markers (Damásio)

R 1

Slide 11 (of 45)

Emotion – Perceived VS Induced

Perceived Emotion – The emotion observer believes the source or stimuli is experiencing or expressing

Induced Emotion – The emotion felt by the observer as a result of the stimuli (very hard to capture/use)

Fearful Speech

R 2

Slide 12 (of 45)

Emotion - Representation and Capture

Required for Empirical analysis Computational Implementation

Requirements Continuous capture over time Continuous representation of emotion (numerical)

Data Consistency

R 3

Slide 13 (of 45)

Emotion - Representation and Capture

Existing Methods

Type Continuous Capture

Continuous Representation

Consistent

Open-Ended Yes and No Partial No

Checklist Yes and No No Yes

Rank & Match Yes and No No No

Rating Scale Partial Yes Partial

R 4

Slide 14 (of 45)

Emotion – Dimensional Approach

Originally proposed by Wilhelm Wundt in the 1800s

We choose 2D with Arousal & Valence Arousal: Active Passive Valence: Positive Negative

2 dimensions offer a balance between ease of reporting and data richness [c]

R 5

Slide 15 (of 45)

Emotion - Representation and Capture

2 Dimensional Emotion Space [d]

R 6

Slide 16 (of 45)

Music Emotion Rules

Need to influence the emotions somehow … Over a century of empirical music psychology has

investigated the link between music emotion [e]

Two types Structural – Modifying the score Performative – Those applied by the performer when

converting the score to audio

R 7

Slide 17 (of 45)

Music Emotion Rules – Structural

Structural Music Rules

An understanding of the musical structure Simplified emotional grouping (octants) and testing Varying degrees of musical theory required

A New Approach [1]

R 8

Slide 18 (of 45)

Octant Structural Music Emotion Rules [2]

1 (happy)

Mode Major(19), Tempo Fast(16), Harmony Simple(8), Loudness Loud(7), Articulation Staccato(5), Pitch High(3), Rhythm Flowing(3), Pitch Range High(2), Pitch Variation Large(2), Pitch Contour Up(2), Note Onset Rapid(2), Rhythm Smooth(2), Rhythm Activity(2), Loudness Medium(1), Loudness Soft(1), Loudness Variation Small(1), Loudness Variation Rapid(1), Loudness Variation Few(1), Pitch Low(1), Pitch Range Low(1), Pitch Contour Down(1), Timbre Few(1), Rhythm Rough(1)

3 (angry)

Mode Minor(14), Loudness Loud(9), Tempo Fast(9), Harmony Complex(8), Note Onset Rapid(5), Pitch Contour Up(5), Pitch High(4), Pitch Range High(3), Pitch Variation Large(3), Loudness Soft(2), Rhythm Complex(2), Loudness Variation Large(2), Timbre Sharp(2), Articulation Non-legato(2), Pitch Variation Small(2), Articulation Staccato(2), Note Onset Slow(2), Timbre Many(1), Vibrato Fast(1), Rhythm Rough(1), Metre Triple(1), Tonality Tonal(1), Tonality Atonal(1), Tonality Chromatic(1), Loudness Variation Rapid(1), Pitch Low(1)

4

Mode Minor(12), Harmony Complex(6), Articulation Legato(3), Pitch Variation Small(3), Tempo Fast(3) , Loudness Loud(2), Loudness Soft(2), Loudness Variation Large(2), Note Onset Rapid(2), Note Onset Sharp(2), Note Onset Slow(2), Timbre Sharp(2), Loudness Variation Rapid(1), Pitch High(1), Pitch Low(1), Pitch Range High(1), Pitch Variation Large(1), Pitch Contour Up(1), Pitch Contour Down(1), Timbre Many(1), Harmony Melodic(1), Tempo Slow(1), Articulation Staccato(1), Rhythm Complex(1), Tonality Atonal(1), Tonality Chromatic(1)

6 (dreamy)

Loudness Soft(5), Tempo Slow(5), Pitch Variation Small(3), Articulation Legato(3), Note Onset Slow(3), Pitch Low(3), Pitch Range Low(2), Loudness Variation Rapid(1), Pitch High(1), Pitch Contour Down(1), Mode Minor(1), Timbre Few(1), Harmony Complex(1), Vibrato Deep(1), Metre Duple(1), Tonality Tonal(1)

7

Tempo Slow(10), Loudness Soft(9), Articulation Legato(5), Note Onset Slow(3), Pitch Low(2), Pitch Range Low(2), Pitch Variation Small(2), Timbre Soft(2), Harmony Simple(2), Mode Minor(1), Loudness Variation Rapid(1), Loudness Variation Few(1), Pitch High(1), Note Onset Rapid(1), Vibrato Intense(1), Rhythm Smooth(1), Rhythm Flowing(1), Rhythm Firm(1), Metre Duple(1)

Slide 19 (of 45)

Music Emotion Rules – Structural

Primary Music Emotion Structural Rules [2]

Can you hear them?Quad 1 (happy)

Quad 2 (angry)

Quad 3 (sad)

Quad 4 (dreamy->bliss)

R 10

Slide 20 (of 45)

Music Emotion Rules – Performative

Performative Music Rules

Requires fine-grained, continuous capture of emotion for testing

Waveform modification (very complex stuff)

Already been done (partially)

R 11

Slide 21 (of 45)

Music Emotion Rules – Performative

Performance rules to accentuate emotion … Chord asynchrony (melody lead or lag) Rubato (especially at phrase boundaries) Melody notes louder Increase dynamic range (gradient) Increase vibrato amplitude

Structural

Structural + Performative

R 12

Slide 22 (of 45)

Music Emotion Rules – Vocal

What does this table mean?

Octant Performance Vocal Structural

2

(Happy)

Fast mean tempo Fast speech rate/tempo

Tempo Fast

Fast tone attacks Fast voice tone attacks Note Onset Rapid

High sound level Medium–high voice sound level

Loudness Loud

Much pitch variability Pitch Variation Large

High pitch level Pitch High

Rising pitch contour Pitch Contour Up

Staccato articulation Articulation Staccato

R 13

Slide 23 (of 45)

Music Tension and Induced Emotion

Is musical emotion really just 2 dimensional? Perceived maybe, induced definitely not

Meyer believed that musical emotion is the inhibition or completion of musical expectations [f]

“Tense Sad”, breaks the rules … important!

R 14

Slide 24 (of 45)

Audience Consideration

Awareness of Audience

Plays an important role in performances Affective Computing Attitudes User state

R 15

Slide 25 (of 45)

Audience Consideration

Attitudes are a cognitive powerful tool: Quickly categorise data Influence decision making Relatively static

R 16

Slide 26 (of 45)

Audience Consideration

User State A listeners response to the stimulus Guides the performer Continuous feedback

Affective Computing Research from MIT Various mechanisms to detect AND affect

R 17

Slide 27 (of 45)

Research Something Real

Phew! A lot of research

Many topics not examined today

… What were we trying to do again?

R 18

Slide 28 (of 45)

Hypothesis

Perceived emotional content of music can be influenced by controlling both the structural and performative aspects

Audience feedback can be captured by computers to tailor the musical experience

Slide 29 (of 45)

The Rule System – Influence

Influencing perceived emotions E.g. make “happier”

How? Apply octant-grouped structural rules

E.g. “Influence to be upbeat and positive” Apply octant 2 rules (tempo [fast], loudness

[loud] …) to music structure

Sy 1

Slide 30 (of 45)

The Rule System – Detection

Why?

Good influencing needs emotional context

Requires Model of Music Tension Advanced pattern matching Advanced knowledge of music theory and composition

Very tricky … Not attempted before

Sy 2

Slide 31 (of 45)

The ArchitectureSy 3

Slide 32 (of 45)

The Architecture – Application Intent

Emotive Information The [Arousal, Valence] vector [3]

timeEndVtimeStartVtimeEndAtimeStartAbaseVbaseAbaseVbaseAy,RxR ,,,,2,2,1,1Scene-Inter

Sy 4

Slide 33 (of 45)

The Architecture – Audience Sensing

Audience response provides a wealth of feedback data to performer

Attitudes and audience response [A, V] incorporated E.g. Cap the fearfulness of a room’s music

Affective computing Keystroke/mouse movement: Arousal and Tension Gaze tracking/Skin conductivity: Arousal and Interest Same problems as measuring induced emotion though

Sy 5

Slide 34 (of 45)

The Architecture – Emotive Algorithm

Equalising unit for [A, V] coming from game and audience

Player Cap: Room Value: Game Event:

Resulting [A, V]:

25,402 Quadrant

18,32

ssss 5,3,0,0,15,10

ssss 9,5,0,0,7,8

Sy 6

Slide 35 (of 45)

Testing Progress [2]

Aims Influence the perceived emotions of music

with primary music emotion structural rules Rules can apply to both Western classical and

standard computer game music Testing

Listener played original work, followed by altered work (e.g. apply octant 2 rules)

How did emotion baseline change? 11 participants, played 6 altered versions

Sy 7

Slide 36 (of 45)

Testing Progress

Overall Results

Looks OK but why the A, V discrepancy?

Accuracy Quadrant Arousal ValenceUser Response 57% 90% 62%

Guess 25% 50% 50%

Weighted Improvement

130% 80% 24%

Sy 8

Slide 37 (of 45)

Testing Progress

Quadrant Breakdown

Quadrant Accuracy Selection Skew Selection Rate %

1 81% Over 56%

2 26% Under 56%

3 71% Over 3%

4 50% Correct -

Sy 9

Slide 38 (of 45)

Testing Progress

Not Angrier, why? Music selection (something deeper going on ..?) Incomplete rule implementation for quadrant 2

Sy 10

Slide 39 (of 45)

Methodology Recap

Score – Manipulate the structure and mark up with emotional performance metadata

How are we going? Structure: Implemented and progressing Performative: Identified, future implementation

Slide 40 (of 45)

Methodology Recap

Audience – Determine emotional state and attitudes using affective computing tech.

How are we going? Identified and developed a theoretical

implementation

Slide 41 (of 45)

Methodology Recap

Architecture – Bring together for an awareness of score and audience

How are we going? Theoretical Implementation at present

Slide 42 (of 45)

Future Work

Implement more structural music rules Expanded testing regime Implement performative rules Begin testing of performative rules Detection component Incorporate Affective Computing elements

Fu

Slide 43 (of 45)

Summary

There exists no automated method to detect and influence the emotions of music We’re getting there

An audience’s response to computer music has previously been inaccessible to computers and thus lost Theoretical, still Future work

Su 1

Slide 44 (of 45)

Questions?

Contact srl@itee.uq.edu.au http://itee.uq.edu.au/~srl

Papers [1] "Playing with Affect: Music Performance with Awareness

of Score and Audience", 2005, Australasian Computer Music Conference

[2] "Dynamic Response: Real-Time Adaptation for Music Emotion", 2005, Australasian Conference on Interactive Entertainment

[3] “Influencing the Perceived Emotions of Music with Intent”, 2005, Third International Conference on Generative Systems (in review)

Su 2

Slide 45 (of 45)

References

[a] Brown, S., B. Merker, and N.L. Wallin, An Introduction to Evolutionary Musicology, in The Origins of Music, S. Brown, B. Merker, and N.L. Wallin, Editors. 2000, MIT Press.

[b] Sloboda, J.A. and S.A. O'Neill, Emotions in Everyday Listening to Music, in Music and Emotion, theory and research.

2001, Oxford Press. p. 415-429. [c] Russell, J.A., Measures of emotion., in Emotion: Theory

research and experience., R.P.H. Kellerman, Editor. 1989, New York: Academic Press. p. 81-111.

[d] Schubert, E., Measurement and Time Series Analysis of Emotion in Music. 1999, University of New South Wales.

[e] Meyer, L.B., Emotion and Meaning in Music. 1956: The University of Chicago Press.

Su 3