Recent Research in Musical Timbre Perception

Recent Research in Musical Timbre Perception

James W. BeauchampUniversity of Illinois at Urbana-

Champaign

Andrew B. HornerHong University of Science and

Technology

Michael D. HallJames Madison University, Harrisonburg,

VA

Starting Point• Timbre experiments are based on musical

instrument sounds.


instrument sounds.• Perform short-time spectral analysis.


instrument sounds.• Perform short-time spectral analysis.• Identify parameters of ST spectrum:



Partial (harmonic) amplitudes- Time variation- Spectral envelope (centroid, irregularity, etc.)



Partial (harmonic) amplitudes- Time variation- Spectral envelope (centroid, irregularity, etc.)

Partial (harmonic) frequencies- Time variation- Inharmonicity

Methods for Studying TimbreStimuli Preparation

In Freq. Domain– Simplification– Perturbation– Normalization

Methods for Studying Timbre Listener Experiments

–Discrimination (pairs)–Timbral Distance Estimation–Classification–Identification

Stimuli PreparationIn Freq. Domain

– Simplification– Perturbation– Normalization

Methods for Studying Timbre Listener Experiments

–Discrimination (pairs)–Timbral Distance Estimation–Classification–Identification

Stimuli PreparationIn Freq. Domain

– Simplification– Perturbation– Normalization

Data Processing/Presentation–Discrimination (sensitivity) scores/plots–Multidimensional Scaling–Correspondence (R2) Measurements

Studies Reviewed

• 1999 Discrimination Study

• 2006 Discrimination Study

• 2006 Multidimensional Scaling (MDS) Study

• 2009 Discrimination/Classification Study

1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA)

• Seven reference sounds– clarinet, flute, oboe, trumpet, violin, harpsichord,

marimba



marimba

• Equalize F0, loudness, and duration.



marimba

• Equalize F0, loudness, and duration.

• Test sounds: Apply six spectrotemporal

simplifications.



marimba

• Equalize F0, loudness, and duration..

• Test sounds: Apply six spectrotemporal

simplifications.

• Subjects discriminate between original and

simplified sounds.

1999 Discrimination StudyResults

• Spectral envelope smoothing 96%

• Spectral flux elimination 91%

• Amplitude envelopes smoothing 66%

• Frequency envelopes smoothing 70%

• Freq. envs. harmonic locking 69%

• Frequency variations elimination 71%

DiscrimScore

2006 Discrimination StudyHorner, Beauchamp, and So JAES

• Eight sustained musical instrument tones– bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin



• Modified by fixed random transfer function 1−2ε < H( f) <1+ 2ε, ε = εrror lεvεl



• Modified by fixed random transfer function–

• F0, loudness, duration, centroid preserved 1−2ε < H( f) <1+ 2ε, ε = εrror lεvεl



• Modified by fixed random transfer function–

• F0, loudness, duration, centroid preserved 1−2ε < H( f) <1+ 2ε, ε = εrror lεvεl

Typical spectral envelopes:original

0

500

1000

1500

2000

300 900 1500 2100 2700 3300

frequency (Hz)

20% error

0

500

1000

1500

2000

2500

300 900 1500 2100 2700 3300

frequency (Hz)

40% error

0500

10001500

200025003000

300 900 1500 2100 2700 3300

frequency (Hz)


• Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.


• Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data.

• Best results: obtained by relative-amplitude (harmonic) spectral error:

ε rase =1N

Ak tn( ) − A'k tn( )

a

k =1

K

∑

Aka tn( )

k =1

K

∑a

n=1

N

∑ , 0 < a < 3.0

Usually, a =1 or 2


R2=0.81

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5error level

discrimination

Discrimination vs. error level (ε):

2006 Discrimination Study

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5relative-amplitude spectral error

discrimination

Discrimination vs. rel-amp spec error:

R2=0.90for

a=1.0

Horner, Beauchamp, and So JAES

2006 MDS Study

• Ten sustained musical instrument tones– bassoon, cello, clarinet, flute, horn, oboe, recorder, alto

sax, trumpet, violin

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

2006 MDS Study



• F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.


2006 MDS Study




• Two types of tones: static (flux removed) and dynamic (flux retained).


2006 MDS Study





• Subjects estimate timbral dissimilarity between instruments.


2006 MDS Study





• Subjects estimate timbral dissimilarity between instruments.

• Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab).


2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

Acoustical Correlates to Test:• Even/Odd: Ratio of even and odd harmonic rms

amplitudes.



amplitudes

• Spectral IRregularity: Degree of jaggedness of a spectrum.



amplitudes


• Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value.

For Dynamic Tones Only:



amplitudes


• Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value.

• Spectral INcoherence: Degree of spectral change relative to the average spectrum (same as flux).

For Dynamic Tones Only:


2D MDSResults:

Static ToneCase

SPSSalgorithm

Correlations: E/O: R=0.78 SIR: R=0.69

Stress=0.12


2D MDSResults:

Static ToneCase

Matlabalgorithm

Correlations: E/O: R=0.79 SIR: R=0.75

Stress=0.12


2D MDSResults:

Dynamic ToneCase

SPSSalgorithm

Correlations: E/O: R=0.71 SCV:R=0.68 SIN: R=0.56 SIR: R=0.39

Stress=0.17


2D MDSResults:

Dynamic ToneCase

Matlabalgorithm

Correlations: E/O: R=0.69 SCV:R=0.68 SIN: R=0.53 SIR: R=0.40

Stress=0.15

2009 Study Hall and Beauchamp (Canadian Acoustics)

Goals/Purpose• Exp. 1. Relative importance of spectral vs. temporal

cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.


Goals/Purpose• Exp. 1. Relative importance of spectral vs. temporal

cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.

• Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid: Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.


Experiment 1 Method• Reference stimuli: Impoverished (static) violin and trombone

sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)




• Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres.

Vn I01 I02 I03

I10 I11 I12 I13

I20 I21 I22 I23

I30 I31 I32 Tr

Temporal

Spe

ctra

l





• Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).





• Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).

• Subjects’ tasks: - 1) to discriminate tone pairs. - 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.


Experiment 1 Results

Violin

-1.00

0.001.00

2.00

3.00

4.005.00

6.00

1 2 3

Step Size

Sensitivity (d')

Amplitude Envelope

Spectral Envelope

Both

Trombone

-1.00

0.00

1.00

2.003.00

4.00

5.00

6.00

1 2 3

Step Size

Amplitude Envelope

Spectral Envelope

Both

Discrimination:

Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

reference stimuli:


Experiment 1 ResultsClassification:

Violin Responses

0.000.100.200.300.400.500.600.700.800.901.00

Vn

VnHybridTrHybrid

Tr

Spectral Envelope

Response Probability

Vn

VnHybrid

TrHybrid

Tr

Trombone Responses

0.000.100.200.300.400.500.600.700.800.901.00

Vn

VnHybrid TrHybrid

Tr

Spectral Envelope

Vn

VnHybrid

TrHybrid

Tr

Amplitude Envelope

Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.


Experiment 2 Method• Reference stimulus: Original impoverished violin.



• Test stimuli: - 1) 3 violin tones interpolated with respect to spectral

envelope (steps 1-3) (original violin amp env kept).




envelope (steps 1-3) (original violin amp env kept).- 2) 3 violin tones low-pass filtered in steps to match the

spectral centroids of the 3 interpolated tones of 1).




envelope (steps 1-3) (original violin amp env kept).- 2) 3 violin tones low-pass filtered in steps to match the

spectral centroids of the 3 interpolated tones of 1).

• Subjects’ tasks: Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)


Experiment 2 Results

0 . 0 0

0 . 5 0

1 . 0 0

1 . 5 0

2 . 0 0

2 . 5 0

3 . 0 0

3 . 5 0

4 . 0 0

4 . 5 0

5 . 0 0

1 2 3

S t e p S i z e

Sensitivity (d')

F o r m a n t S t r u c t u r e

S p e c t r a l C e n t r o i d

Discrimination:

Classification:Trombone Responses

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Formant Structure Spectral Centroid

Stimulus Type

Identification Percentage

Vn

VnHybrid

TrHybrid

Tr

Violin Responses

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Formant Structure Spectral Centroid

Stimulus Type

Response Probability

Conclusion Summary• 1999 discrimination study:

– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.

Conclusion Summary• 1999 discrimination study:

– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.

• 2006 discrimination study:– The ability to hear differences between

dynamic tones with matched spectral centroids and randomly altered spectra correlates strongly with relative spectral-amplitude differences.

Conclusions• 2006 MDS study:

– Using centroid and attack/decay normalized tones, there is strong evidence that even/odd ratio and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.

Conclusions• 2006 MDS study:

– Using centroid and attack/decay normalized tones, there is strong evidence that even/odd ratio and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.

• 2009 discrimination/classification study:– Using spectral interpolation with respect to both

spectral and temporal dimensions on impoverished violin and trombone tones:

1) Spectral differences were found to be more important than temporal differences.

2) Detailed spectral differences were much more important than mere spectral centroid differences.

Documents

Recent Research in Musical Timbre Perception