August 16, 1999 SMPC Conference '99 1

Cross-modal integration:Alignment of auditory and visual

accent structures in motion pictures

Dr. Scott D. Lipscomb

Institute for Music Research

University of Texas at San Antonio

August 16, 1999 SMPC Conference '99 2

Acknowledgments• University of California, Los Angeles

– Dr. Roger A. Kendall & Dr. Ed Carterette

• University of Texas at San Antonio– Institute for Music Research

• Dr. Don Hodges, Director

• Fellow Music Researchers

August 16, 1999 SMPC Conference '99 3

Film Music Literature

• past studies deal almost exclusively with the referential aspect of musical sound– “cognitive congruency” (Marshall & Cohen, 1988)

– selected others: Tannenbaum (1956), Thayer & Levenson (1984)

– special issue of Psychomusicology (vol. 13, 1994)

• vs. accent structure alignment– i.e., how often important events in the music coincide

with important events in the visual scene

August 16, 1999 SMPC Conference '99 4

Film Music Perception Paradigm(Lipscomb & Kendall, 1995)

AssociationJudgment

Accent StructureRelationship

Perception

Aural stimulus Visual stimulus

YES

YES

NO

NO

IMPLICITPROCESSES

No Shift ofAttentional

Focus

Shift ofAttentional

Focus

AuralStimulus

VisualStimulus

August 16, 1999 SMPC Conference '99 5

Alignment Conditionsafter Yeston (1975)

b)

a)

c)

Consonant

Out-of-phase

Dissonant

August 16, 1999 SMPC Conference '99 6

Subject Responses …two VAME ratings

• synchronization: “… how often important events in the music coincide with important events in the visual scene”

• effectiveness: “… simply concerns [the observer’s] subjective evaluation of how well the two go together”

August 16, 1999 SMPC Conference '99 7

Results of Previous Study (1997)presented at Penn State ASA Conference

Synchronization Ratings--All Experiments

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

Consonant Out-of-Phase Dissonant

Alignment Condition

Mea

n S

core

s

Exp 1

Exp 2

Exp 3

Effectiveness Ratings--All Experiments

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

Consonant Out-of-Phase Dissonant

Alignment Condition

Mea

n S

core

s

Exp 1

Exp 2

Exp 3--UntrainedExp 3-Mod &Trained

August 16, 1999 SMPC Conference '99 8

Problems with Dissonant Stimuli

Solution Found

August 16, 1999 SMPC Conference '99 9

Revised Stimulus Preparation

• A-V alignment conditions were created using Media 100 software on a Macintosh G3 computer

• sound files were manipulated in Sonic Foundry’s Sound Forge 4.0, using the Time Compress/Expand Sonic Foundry plug-in

• completed A-V composites were recorded directly from Media 100 onto VHS tape

August 16, 1999 SMPC Conference '99 10

Consonant Alignment Condition

aligned as intended by the composer

(visual & music only)

August 16, 1999 SMPC Conference '99 11

Out-of-phase Alignment Conditionalignment conditions based on Preliminary Study

A-V Scene misalignment of audio

"Dots" -100 ms

"Canon" 532 ms

"Synchromy" 532 ms

"Portrait of Elizabeth" 672 ms

"Flashback" -890 ms

"Reunion" 425 ms

August 16, 1999 SMPC Conference '99 12

Dissonant Alignment Conditions

• audio tracks for the McLaren animations were “time expanded” by 115%

• audio tracks for the “Obsession” excerpts were “time expanded” by 110%– since these excerpts were longer, the gradual

misalignment could occur at a slower pace

August 16, 1999 SMPC Conference '99 13

Subject Info & Method• N = 135 UTSA students taking music classes

– musical training n (high = 17, mod = 31, low = 87)– gender n (female = 82, male = 53)– stimuli presented to groups of subjects

• stimuli were presented to subjects using a Samsung VR 5855 video cassette recorder and a ProScan PS80690 80” big screen television

– VAME ratings were provided on a continuous line response anchored by either “not synchronized - synchronized” or “ineffective - effective”

– each subject was assigned to one of three random stimulus presentation orders

– response forms were generated so that order of VAME responses was also randomized into three versions

not synchronized synchronized

August 16, 1999 SMPC Conference '99 14

Video excerpts

August 16, 1999 SMPC Conference '99 15

Statistical Analysis - Between(SPSS’s “General Linear Model - Repeated Measures”)

• repeated measures, fully-factorial ANOVA– three repeated measures: 2 VAME ratings x 6 AV

combinations x 3 alignment conditions

• Tests of Between-Subjects Effects– no significant between-groups variation

• musical training: p = 0.090; f(2,129) = 2.457

• gender: p = 0.549; f(1, 129) = 0.361

– interaction between AV combination & musical training is not statistically significant

• p = 0.622; f(2,129) = 0.477

August 16, 1999 SMPC Conference '99 16

Statistical Analysis - Within(SPSS’s “General Linear Model - Repeated Measures”)

• Tests of Within-Subjects Effects– within-groups, ratings were significantly different depending upon

the AV combinations• p < .0005; f(5) = 20.5

– within-groups, ratings were significantly different depending upon the alignment condition

• p < .0005; f(2) = 173.036

• Within-Subjects Interaction Effects– alignment condition x VAME

• p < .0005; f(2) = 3.906

– AV combination x alignment condition• p < .0005; f(10) = 48.720

August 16, 1999 SMPC Conference '99 17

Statistical Analysis - Within(SPSS’s “General Linear Model - Repeated Measures”)

• Within-Subjects Effects - complex interactions– AV combination x VAME x musical training

• p = .006; f(10) = 2.517

– AV combination x alignment condition x VAME• p < .0005; f(10) = 3.885

August 16, 1999 SMPC Conference '99 18

Experimental Results(r = .997)

McLaren Animations

20.0030.0040.0050.0060.0070.0080.0090.00

100.00

Dots

Dots_p

Dots_x

Canon

Canon_

p

Canon_

x

Synch

rom

y

Synch

rom

y_p

Synch

rom

y_x

A-V Stimuli

Mea

n R

atin

g

Synchronization

Effectiveness

August 16, 1999 SMPC Conference '99 19

Experimental Results(r = .977)

"Obsession" Excerpts

20.0030.0040.0050.0060.0070.0080.0090.00

100.00

Portra

it

Portra

it_p

Portra

it_x

Flashb

ack

Flashb

ack_

p

Flashb

ack_

x

Reunion

Reunion

_p

Reunion

_x

A-V Stimuli

Mea

n R

atin

g

Synchronization

Effectiveness

August 16, 1999 SMPC Conference '99 20

Experimental ResultsSubject Ratings Collapsed (n=3)

McLaren AnimationsSubject Ratings Collapsed

0.00

20.00

40.00

60.00

80.00

100.00

Consonant Out-of-phase Dissonant

Alignment Condition

Mea

n R

atin

g

"Obsession"Subject Ratings Collapsed

0.00

20.00

40.00

60.00

80.00

100.00

Consonant Out-of-phase Dissonant

Alignment Condition

Mea

n R

atin

g

Results of Previous Study

August 16, 1999 SMPC Conference '99 21

Conclusions• accent structure alignment does, in fact, play an

important role in subject ratings of both “synchronization” and “effectiveness”– ANOVA & Pearson correlation coefficient

• based on the results of the present & previous investigations, there appears to be a dynamic relationship between the “association judgment” and “accent structure alignment” aspects of the model of Film Music Perception

Revised Model

August 16, 1999 SMPC Conference '99 22

Contact Info

Dr. Scott D. LipscombUTSA Division of Music6900 N. Loop 1604 WestSan Antonio, TX 78249

(210) 458-4354(210) 458-4381 FAXlipscomb@utsa.edu

http://music.utsa.edu/~lipscomb

August 16, 1999 SMPC Conference '99 23

Film Music Paradigm(revised)

Perception

Audio-Visual Congruence

AssociationJudgment

Accent StructureAlignment

AV combinationperceived

asEffective

AV combinationperceived

asIneffective

AuralStimulus

VisualStimulus

ImplicitProcesses No

Yes