10/13/98Acoustical Society of America1 Synchronization of musical sound and visual images: Issues of empirical and practical significance in multimedia

10/13/98 Acoustical Society of America 1

Synchronization of musical sound and visual images:

Issues of empirical and practical significance in multimedia

development

Scott D. Lipscomb, Ph.D.

Institute for Music Research,

University of Texas at San Antonio







Model of Film Music PerceptionLipscomb & Kendall (1994)

AssociationJudgment

Accent StructureRelationship

Perception

Aural stimulus Visual stimulus

YES

YES

NO

NO

IMPLICITPROCESSES

No Shift ofAttentional

Focus

Shift ofAttentional

Focus


Association Judgment

Referential Meaning


Influential Musical Parameters: clarity of tonal center harmonic complexity dynamic variation tempo

• absolute rate & fluctuation phrase structure melodic activity


Accent Structure Alignment

Syntactical Meaning


Sources of Perceived Accent

Musical• pitch height, loudness, timbre

Visual• spatial location, shape, color


Alignment Possibilities

b)

a)

c)

Consonant

Out-of-Phase

Dissonant


Method - AV Stimuli Experiment One

• simple single-object animations and pitch sequences (5-sec); created by author

Experiment Two• excerpts from experimental animations by

Norman McLaren (8-sec); “Dots” (1940), “Canon” (1964), & “Synchromy” (1971)

Experiment Three• excerpts from “Obsession” (25-sec) with

musical score by Bernard Herrmann


Experiment 1--VAME(reduced set)

0

10

20

30

40

50

60

70

80

90

100

V5A

5_C

1

V5A

5_O

1

V5A

8_D

1

V10

A10

_C1

V10

A10

_O1

V10

A8_

D1

V8A

8_C

1

V8A

8_O

1

V8A

10_D

1

V5A

5_C

2

V5A

5_O

2

V5A

8_D

2

V10

A10

_C2

V10

A10

_O2

V10

A8_

D2

V8A

8_C

2

V8A

8_O

2

V8A

10_D

2

V5A

5_C

3

V5A

5_O

3

V5A

8_D

3

V10

A10

_C3

V10

A10

_O3

V10

A8_

D3

V8A

8_C

3

V8A

8_O

3

V8A

10_D

3

V5A

5_C

4

V5A

5_O

4

V5A

8_D

4

V10

A10

_C4

V10

A10

_O4

V10

A8_

D4

V8A

8_C

4

V8A

8_O

4

V8A

10_D

4

AV Composite

Mea

n S

core

s

Synchronization

Effectiveness


Experiment One--VAME ResponsesAveraged Across All Levels of Musical Training

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

Consonant Out-of-Phase Dissonant

Alignment Condition

Mea

n Sc

ores Synchronization

Effectiveness


Control & Reliability vs.Ecological Validity

Experiments Two & Three


Experiment Two“Dots” by Norman McLaren


Experiment Two“Synchromy” by Norman McLaren


Experiment Two - VAMECollapsed Across Alignment Condition

Experiment Two--VAME ResponsesAveraged Across All Levels of Musical Training

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

Consonant Out-of-Phase

Dissonant

Alignment Condition

Mea

n S

core

s

Synchronization

Effectiveness


Experiment Three“Obsession”--music by B. Herrmann


Experiment Three - SynchronizationCollapsed Across Alignment Condition

Experiment Three--Synchronization RatingsAveraged Across All Levels of Musical Training

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00


Alignment Condition

Mea

n S

core

s


Experiment ThreeEffectiveness - Combined Groups

Experiment Three--EffectivenessCombined Groups

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00


Alignment Condition

Mea

n S

core

s

Untrained

Moderate-Trained


Comparison of VAME Results Across All Three Experiments


VAME - SynchronizationAcross All Experiments

Synchronization Ratings--All Experiments

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00


Alignment Condition

Mea

n S

core

s

Exp 1

Exp 2

Exp 3


VAME - EffectivenessAcross All Experiments

Effectiveness Ratings--All Experiments

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00


Alignment Condition

Mea

n S

core

s Exp 1

Exp 2

Exp 3--Untrained

Exp 3-Mod & Trained


Film Music Perception Paradigm(revised)

Perception

Audio-Visual Congruence

AssociationJudgment

Accent StructureAlignment

No Shiftof

Attentional Focus

Shiftof

Attentional Focus

AuralStimulus

VisualStimulus

ImplicitProcesses No

Yes


Enhancing Instructional Materials with Multimedia

CD-ROM Companion to the

Handbook of Music Psychology(D. Hodges, Ed.; IMR Press)


Hyperlinks to Sound Files“If a pianist plays a low A (110 Hz) on the piano, the

resulting vibration does not consist of only a single, periodic vibration at the rate of 110 times per second. Rather, the string vibrates as a whole (110 Hz), in halves (220 Hz), in thirds (330 Hz), in fourths (440 Hz), and so on. Therefore, there is vibrational energy not only at the fundamental frequency of 110 Hz but also at each of these integer multiples [audio examples: a) complex tone made up of 8 partials & b) complex tone built one partial at a time …”

Lipscomb & Hodges, 1996, p. 97


Sounds to Attract Attention“If a pianist plays a low A (110 Hz) on the piano, the

resulting vibration does not consist of only a single, periodic vibration at the rate of 110 times per second. Rather, the string vibrates as a whole (110 Hz), in halves (220 Hz), in thirds (330 Hz), in fourths (440 Hz), and so on. Therefore, there is vibrational energy not only at the fundamental frequency of 110 Hz but also at each of these integer multiples [audio examples: a) complex tone made up of 8 partials & b) complex tone built one partial at a time …”

Lipscomb & Hodges, 1996, p. 97


Exploratory Environments Beating—experimentation with constructive &

destructive interference and the critical bandwidth WaveMix—combining partials to create a

complex tone Signal—allows student to create a complex signal

by manipulating partial amplitudes & phase relationships, hear the result, and view an animation approximating molecular motion


Sound as a Cue

Computer Guts• instructional package designed by Dr. David

Sebald (UTSA Division of Music)– more examples available online at:

http://www.aim-ed.com/


Conclusions: Use of Multimedia to Enhance the Learning Experience

provide additional information through illustration• audio examples or animations

focus user attention explicit association judgments create exploratory environments

• allowing students the opportunity to incorporate the highest levels of knowledge (Bloom, 1956)

a work in progress


Contact Info:

Dr. Scott D. LipscombInstitute for Music Research

Division of Music The University of Texas at San Antonio

6900 N. Loop 1604 WestSan Antonio, TX 78249phone: (210) 458-5334FAX: (210) [email protected]

http://music.utsa.edu/~lipscomb

Documents

10/13/98Acoustical Society of America1 Synchronization of musical sound and visual images: Issues of empirical and practical significance in multimedia