TheRoleofFacialExpressionsinVocalEmotionalCommunication

BillThompsonMacquarieUniversity

Sydney,Australia

Categoriesofnon‐verbalbehaviour

•  Eckman&Friesen(1981)–  Illustrators–  Affectdisplays–  Emblems

–  Regulators

lowpitchhighpitch

illustratingthebeat

expressingjoy

followingtheheart

thewinningfist

Theextended‐emotionsperspective

1.Emotionsinvolvechangesinmultiplesubsystemsbeyondfeeling:cognition,motivation,physiologicalreaction&motorexpression.

2.Subsystemsaresynchronizedonmultiplelevelsandextend(viafeedback&feedforward)tosocial&physicallevels

3.Facialexpressionsareintegratedwiththesonicdimensionofmusic&functionasbothsignalsystem&foremotionalsynchronization

SeeScherer,1984,2001,2004,Grandjean&Scherer,2008,others;SeealsoworkbyAndyClark

Theextended‐emotionsperspectiveWheredofacialexpressionsfit?

1.  Introspectiveethnography:whatemotionsareinvolved?

2.  Perceptualstudies:expression/decoding3.  Productionstudies:analysisoffacialexpressions

Introspectiveethnography

Subjectiveexperience

Perceptualstudies

•  Affectsperceiveddissonance•  Affectsperceivedvalence•  Affectsperceivedstructure•  Occursautomaticallyandpreattentively

Reportedin:Thompson,2009Thompson&Balkwill,2009Thompson,Graham&Russo,2005Thompson,Russo&Quinto,2008Thompson&Russo,2007Livingstone,Thompson,Russo,2009Ceaser,Thompson,&Russo,2009

Facial expressions of emotional valence

•  Participants shown videos of sung major or minor intervals & rated emotional valence of sounded music (positive to negative).

•  In some conditions, the dynamic facial expressions were “switched” to make “incongruent” conditions (e.g., major sounded interval with facial expressions that accompanied the minor sung interval).

Thompson, Russo & Quinto (2008)

•Isconsciousattentioninvolved?Toassessthisquestion,viewersalsocountedthenumberofzerosthatappearedonthescreen.

•By“occupying”attentionalresourceswecoulddeterminetheinvolvementofconsciousattentioninaudio‐visualintegrationofemotionalinformation.

Visualsignalsofemotionareintegratedwithauditorysignalsautomaticallyandpre‐attentively

Facial expressions of melodic interval

•  Participants shown silent videos of sung melodic intervals and asked to rate the size of the “imagined” intervals

•  Several intervals presented, from zero to 12 semitones

Thompson & Russo (2007)

1

2

3

4

5

6

7

0 1 2 3 4 5 6 7 8 9 10 11 12

Interval Size (semitones)

Mean

In

terv

al

Siz

e R

ati

ng

Singer 1Singer 2Singer 3

Acrosssingers:r=0.94(perceived&actualintervalsize)

r=0.90

r=0.88

r=0.95

Facialexpressionsoftonality

•  Jazzsingerssangthelastphraseofsilentnight,substitutingthelasttoneforeachof13probetones

•  Performersinstructedtosingexpressively

•  Participantssawvideos(nosound)andjudgedthegoodness‐of‐fitofthefinalprobeexpression

•Spotsusedformotiontracking

•Last(probe)tonesungtwicetominimizeconstraintsofintervalsize

•PresentedWITHOUTsound

Presentations

r=0.88tonalhierarchy(individualr’s=0.88&0.81)r=0.03proximity

r=0.51(tonalhierarchy)r=0.86(proximity)MultipleR=0.93(bothpredictorssignificant)

Productionstudies

Motioncapture

•  Design:7participantswatcheda“model”andthenimitatedthesungphraseandemotionalintentinprecisetimewiththemusic.

•  Participants:differenttrainedsingersusedforsungmodel(target)andimitation.

FromLivingstone,Thompson&Russo,2009,MusicPerception

Methods•  Numberofmarkersused=13•  Cameras:4xMX‐F20(2megapixels)•  Conditions:

–  Emotionexpressed:3(happy,sad,neutral)–  Verbalaccompaniment:6–  Repetition:2

•  Totalnumberoftrials(captures)=36•  Tempo:80bpm•  Melody:tonic‐supertonic‐leadingtone‐tonic•  Practicetrials:about10

Modelstimuli•Bar1isthemodel

•Bar2isthecountdown•Bar3istheimitation(greenlight)

Timeframesofinterest

1.  Duringmodel Emotionalsynchronisation

2.  Beforesinging Emotionalplanning3.  Singingphase Implementationofimitation

4.  Post‐singing Lingeringemotionalsignal

Model:Grassisgreeninsummertime

PERCEPTION PLANNING IMITATION LINGERING

Imitation:Grassisgreeninsummertime? ? ?

Model

Subject

Markerplacementonface

Eyebrows•Inner/outer,leftandright

Lips•Upperandlowercentre•Leftandrightcorners

Stablepoints•Tipandbridgeofnose•Leftandrightsideofhead•Centreofforehead

Referencepointformeasuredmovement

Positiveemotion:capturesession

Timecourseoffeaturedisplacementrelativetonosetipinthehappycondition,averagedacrosscaptures/participants

Averagedisplacementofeyebrow

‐7

‐6

‐5

‐4

‐3

‐2

‐1

0

1

Perception Planning Production Post-Production

Dis

pla

ce

me

nt

(mm

)

PanelA:OuterEyebrow

Happy

Sad

Neutral

Sad

Averagedisplacementoflipcorner

‐7

‐6

‐5

‐4

‐3

‐2

‐1

0

1

Perception Planning Production Post-Production

Dis

pla

ce

me

nt

(mm

)

PanelB:LipCorner

Happy

Sad

NeutralHappy

Empiricalevidence:Electromyography(EMG)

Zygomaticandcorrugatormusclesinhappyandsadconditions

Fourparticipantswentthroughthesameprocedurebutfacial

expressionsmeasuredbyEMG

CompositeEMGfunctionsforemotionalandneutralconditions

“Emotional”isthesumofzygomaticforhappy&corrugatorforsad:Measuresoverallfacialresponsivenesstoemotionalstimuli

Bycomparingthetwocompositemeasures,wecandissociatemovementsrelatedtoproduction(neutralcomposite)frommovementsrelatedtoemotionality

Summaryofproductionstudies

•  Performersareawareoftheemotionalimpactofstructuralfeaturesinmusicandtakeanactiveroleininteractingwiththosefeaturestoshapeexperience

•  Facialmovementsreflectbothemotionalandstructuralcontentofmusic

•  Suchmovementsinfluenceperceptionsofmusic

•  Whenwatchinganemotionallysungphrase,performerssynchronisefacialexpressionswiththemodelexpression

•  Emotionalsingingisprecededbyemotionalplanning

•  Emotionalsingingisfollowedbyemotionallingering

Conclusions

1.  Personal(introspective)accountsindicatethatemotionalresponsestomusicreflectepisodicmemories,emotionalgoals,communicativeintentionsandphysiologicalresponsestomusicalattributes.

2.  Facialexpressionsandmovementsareoftenintegratedwithaffectiveexperience.Visualsignalsaffectinterpretationsofemotionalvalence,dissonance,melodicstructureandtonalstability.Audio‐visualintegrationoccursautomatically,withoutconsciouscontrol.

3.  Emotionalresponsesbyperformersarereflectedinfacialexpressions,whichreflectsynchronizationwithothers,emotionalplanning,musicproduction,andemotionallingeringandclosure