ITU-T Workshop on From Speech to Audio: bandwidth ...symphony.arch.rpi.edu/~braasj/.inside/images/... · after Dubrovski & Cherniak 1966 Lannion, France, 10-12 September 2008 International

1

Lannion, France, 10-12 September 2008

InternationalTelecommunicationUnion

Models of Binaural Interaction

Jens BlauertEmeritus Professor of Acoustics

Ruhr-Universität Bochum, Germany

Jonas BraaschAss. Professor of Communication AcousticsRensselaer Polytechn. Inst., Troy NY, USA

ITU-T Workshop on"From Speech to Audio: bandwidth extension,

binaural perception"Lannion, France, 10-12 September 2008

Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 2

Prominent Features of Binaural Hearing

• Localization– Formation of Positions of the Auditory Events

i.e., Azimuth, Elevation & Distance– Spatial Extent of Auditory Events

• Suppression of– Directional Information Coming from Reflections

e.g., Precedence Effect, Localization Dominance, Fusion – Reverberation, Coloration and Noise

• Identification & Segregation of– Auditory Streams

e.g., Concurrent Talkers (Cocktail-Party-Effect),Warning Signals


– Architecture of a Model of Binaural Hearing– The Jeffress Processor– The Lindemann/Gaik Extensions– Interpreting Binaural Activity– The Effect of Interaural Incoherence– Binaural Speech Enhancement


Part I: Basic Concepts


binaural-activity display

Architecture for a Model of Binaural Hearing


Ear-Adequate Band-Pass-Filter Bank


A Simplified Functional Model of the Hair Cells

Probabilistic

2


ΨY(τ)= 1/(t1-t0) yl(t) yr(t +τ)Σt=t0

t1

Cherry 1959

The Interaural Cross-Correlation Functionusually estimated for the physiologic range of interaural

arrival-time differences, τ < + 1 ms


The Binaural-Coincidence Processor after Jeffress 1948

There are many of these channels in parallelFM … firing modelBP … band-pass filter


Output of Jeffress’ Coincidence Processorfor Different Angles of Sound Incidence

videos provided by Braasch 2003

estimates of the interaural cross-correlation function

about 0.7 kHz about 6 kHz


Sample Output of the Binaural Modelone frontal sound source, sending out a musical chord

taken at an instant t = tofrom a running

correlogram


The Lindemann Processorplease note the monaural modules, m

Lindemann 1986 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 12

Schematic View of the Lindemann Model

3


Output of Lindemann’s Coincidence Processor with Lateral Inhibition

video provided by Braasch 2005

estimate of the inhibited interaural cross-correlation function

about 0.7 kHz


Output of the Lindemann Processorwith regard to interaural arrival-time difference (ITDs)

and interaural level differences (ILDs)

600-Hz sinusoidal sounds

ITDs ILDs

d


own HRTFs other animal‘s HRTFs

Guinea-pig experiments, after Sterbing & Hartung 1998

Single-Cell Responses from theCentral Nucleus of the Inferior Colliculus


„Natural Combinations“ of ITDs and ILDs

Gaik 1988


Weighted Contralateral Inhibition

after Gaik 1988

Gaik’sExtension


cente

rfr

equen

cyof

critic

alband

left < lateral deviation > right

Output of the Jeffress-Lindemann-Gaik Model frontal broad-band sound source

4


A „Figurative“ Plot of the Output of the Gaik Processor Block Diagram of theBinaural-Analysis Systemof IKA Bochum

binaural-activity map

pure bottom-upprocessing,signal driven !

interaural-level-difference analysis

interaural-time-difference analysis200620062006

20


Spatial Extent of the Auditory Event as a Function of Interaural Correlation

after Dubrovski & Cherniak 1966 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 22

Model Output with Incoherent Ear-Input Signalsparameter: degree of interaural coherence, k

left < center > right

prediction:


Area Covered by Auditory Events as a Function of the Degree of Interaural Coherence, k

pink noise, 12 subjects

after Blauert & Lindemann, 1985(plot enhanced for contrast)


Impulse Response and Running Interaural Cross Correlation

in a Room with Reflecting Walls

5


The Binaural Intelligibility-Level Difference, BILD

Cherry’s Experiment (1959) Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 26

Instantaneous Binaural Activity of Two Concurrent Talkers

talker # 1

left < lateral deviation > right

talker 1

talker 2

Ampl

itude

runnin

gtim

eLannion, France, 10-12 September 2008

InternationalTelecommunicationUnion 27

Architecture of a Cocktail-Party Processor, Based on Binaural Modelling

direction finder weight assigment

BinauralModel

ControlUnit

spectraldecomposition

speechcleansing

WienerFilter

source selection


Spatial Selectivity of a Model of Binaural Hearing

Bodden 1993


one noise source two concurrent talkers

two moving talkers talker plus wall reflection

Source Tracking with the Binaural Model

Bodden 1993 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 30

Binaural Modeling for Acoustically-Adverse Conditions

(a) unprocessed

(b) processed

influence of reverberation on binaural activity (model output)

6

Cross-Modal Influenceand Cognitive Influenceon the Formationof Auditory Events

top-down processinghypothesis driven !

binaural-activity map

31 Blauert 1988

Potential Applications for Binaural Algorithms

32


Communication Acoustics

Communication Acoustics represents those areas of acoustics that relate to the modern communication and information

sciences & technologies



Part II: The Precedence EffectContents

- Phenomenology- Bottom-Up Modelling - Auditory Scenes- Build-Up and Break-Down- Top-Down Modelling


Standard Stereo-Listening Arrangement


lead

lag

Auditory Effects with Two Coherent Sound Sources

summing localization precedence effect echo threshold

7


Precedence Effect, Haas Effect & Backward Inhibition

signal: running speech of 50 syllables/s

delay of the reflection


Auditory-Event Trajectoriesfor Broad-Band and Narrow-Band Signals

narrow-band signal broad-band signal


ms

Trajectories of Narrow-Band Signalstwo loudspeakers in standard stereo position

Blauert & Cobben 1978 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 40

Widening Effect with Broad-Band Sound Sources

broad-bandsounds

primaryauditoryevent

echo


Binaural Model for StudyingInter-Aural Cross-Correlation

includes a simple cochea-filterand hair-cell model

fc = 800 Hz fc = 800 Hz

Blauert & Cobben 1978 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 42

Contral-Lateral Inhibition after Lindemann

ICC

contra-laterally-inhibited ICC

Lindemann 1982

8


Cross-Correlation vs. Lindemann

impulsive input signals with no ITD, but with an ILD, looked at in an auditory „critical“ band about 800 Hz

schematic plot

Output of theLindemann Model for a Frontal Sound Plus One Lateral Reflection

includesdynamic contra-lateral inhibition

impulses (clicks)

left panel:cross correlation only

right panel:cross correlation pluscontra-lateral inhibition

44


Precedence Effect for Ongoing Signals

lead and lag pair:band-passed noise, 500 Hz center frequency, 200 ms100 Hz, 400 Hz or 800 Hz bandwidthlead: + 300-µs ITD, lag: - 300-µs ITD inter-stimulus intervals, ISI:

0, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.5 ms6 listenersacoustic pointer

stimulus presentation via headphones

Braasch & Blauert 2004 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 46

Time Course of the Listening Experiments

ITD 1 ITD 2ISI

time

rightchannel

leftchannel

leadlag

left ear

right ear

200-ms noise bursts, band-pass filtered around 500 HzISI … inter-stimulus intervalITD 1, ITD 2… inter-aural arrival-time differences

Blauert & Braasch 2004


Psycho-Acoustic Resultsno localizationdominance

band-pass-noise bursts200-ms duration, 500-Hz center frequency


Cross-Correlation Model x…type I+…type II

level diff


48

9


Output of the Jeffress Coincidence Processor – Lag Frontally-Fixed, Lead Moving About –

video provided by Jonas Braasch (2005)

estimate of the interaural cross-correlation function

about 0.7 kHz100-ms noiseISI ...10 ms

lag


Output of the Jeffress Coincidence Processor – Lag Frontally Fixed, Lead Moving About –


estimate of the interaural cross-correlation function

broad band100-ms noiseISI ... 10 ms


Modification of the Lindemann Algorithm

Aim: Processing of ITDs and ILDs independently of each other

(a) ITDs: compression of the signal before sending it to the Lindemann model to elimate the influence of ILDs

(b) ILDs: processed through excitation/inhibition, EI, cellsthat contain inhibitory elements

Braasch & Blauert 2004

Modified Lindemann Model x…type I+…type II

inhibited ICC level diff


52


Output of Lindemann’s Processor with Lateral Inhibition– Lag Frontally-Fixed, Lead Moving About –



about 0.7 kHz100-ms noiseISI ...10 ms

lag


Output of Lindemann’s Coincidence Processor with Lateral Inhibition



broad band100-ms noiseISI = 10 ms

10


Conclusions

At 100-Hz bandwidth, localization dominance was onlyobserved in half of the cases

The psycho-acoustical results could be simulated well using a combined ICC/EI model with elements of contra-lateral inhibition

In the model simulation, across frequency interaction was not required.

The degree of localization dominance depends on thesignals‘ bandwidth


Simulation for Ongoing Stimulidifferent settings of the contra-lateral inhibition

low for impulses high for ongoing sounds


Conclusions

The necessity to increase the inhibition factor of the Lindemannmodel to simulate experiments with ongoing sounds suggests that contra-lateral inhibition increases with signal duration

An alternative solution could employ a second onset-triggered inhibition process with a longer time constant on top of the Lindemann model, as was done by Djelani to simulate the build-up of the Precedence Effect for click trains


Lindemann‘s Algorithm (1985)

dynamicallyadjustable !

Lindemann 1985


Fast & Slow Melodies – Trumpet

time

Fo

Tsakostas & Blauert 2001 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 60

Echo Thresholds

Fast & Slow Melodies

35 4030

4253 52

65

4557

4855

8592

7582

60

0

20

40

60

80

100

L1 L2 L3 L4

ET

in m

s FFMSSM

F…fast, FM…fast in mixture, S…slow, SM…slow in mixture

Tsakostas & Blauert 2001

11


Low- & High-Frequency Noise

time

frequ

ency

LF

HF

LF

HF

time

frequ

ency

HF

timefre

quen

cytime

frequ

ency

LF

durations: 500 ms and 1000 ms, rise & fall time: 10 ms, linearlevel: 70 dB SPL, A-weighted

LF

LFM

HF

HFM

time: 1000 ms

Tsakostas & Blauert 2001 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 62

Echo Thresholds

LF & HF

10090 95

78

136151

134

82 80 8463

85 86 8664

135

020406080

100120140160

L1 L2 L3 L4

ET

in m

s LFLFMHFHFM

LF…low, LFM…low in mixture, HF…high, HFM…high in mixture

Tsakostas & Blauert 2001


Wolf‘s Experiment I

Wolf 1991 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 64

Wolf‘s Experiment IIalternative plot

Wolf 1991


The Clifton Effect

Clifton 1987 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 66

Sound Sources Alternating in Space

Blauert & Col 1989

12


Fusion Build-Up: Experimental Set-Up

build-up

build-up

no build-up

Djelani 2001 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 68

Fusion-Build-Up Results

Djelani & Blauert 2002


Virtual Test Room

Djelani & Blauert 2002 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 70

Test-Room SwitchingDjelani & Blauert 2002


Room-Switching Results

Djelani & Blauert 2002 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 72

Echo-Build-Up-Decay Results

Djelani & Blauert 2002

13


Wolf‘s Model

prior knowledge,plausibility

pure cross-correlation maps

cross-correlationmaps with dynamic inhibition

Wolf 1991 Lannion, France, 10-12 September 2008InternationalTelecommunicationUnion 74

The Franssen Effect

Franssen 1960

Modelling the Auditory System withHypotheses-DrivenProcessing Stages

– situational knowledge– domain knowledge– cross-modal influence

binaural- activity map

75Lannion, France, 10-12 September 2008


Audition and Cognition Come in Couples

What does the sound meanto me ?

Where and howis the sound ?

recognitioninterpretation

detectionperception


Communication Acoustics

Jens Blauert, ed. (2005)

Authors: Jens Blauert, Jonas Braasch, Hugo Fastl,Volkmar Hamacher,

Dorte Hammershøi, Ulrich Heute, Inga Holube, Herbert Hudde, Ute Jekosch, Georg Klump,

Armin Kohlrausch, Arild Lacroix, Henrik Møller,Sebastian Möller, John N. Mourjopoulos,

Pedro Novo, Steven van der Par

Springer Berlin–Heidelberg–New YorkISBN 3-540-22162-X

77

Communication AcousticsJens Blauert, ed. (2005)

Contents

78

14

Lannion, France, 10-12 September 2008

InternationalTelecommunicationUnion

Thank you!

[email protected]@rpi.edu

ITU-T Workshop on"From Speech to Audio: bandwidth extension,

binaural perception"Lannion, France, 10-12 September 2008

79Lannion, France, 10-12 September 2008


Copyright note:

This material is not in the public domain.The authors claim all applicable rights.However, permission to copy it is granted under the condition that proper reference is given to the authors.

Corresponding author:

_____________________________________

Jens Blauert, Emeritus Professor of AcousticsInstitute of Communication AcousticsRuhr-Universität BochumD-44780 Bochum, Germany

Tel.: +49 234 322 2496 (direct: 3480)Fax: +49 234 321 4165e-mail: [email protected]://www.rub.de/ika

____________________________________________

©

Documents

ITU-T Workshop on From Speech to Audio: bandwidth ...symphony.arch.rpi.edu/~braasj/.inside/images/... · after Dubrovski & Cherniak 1966 Lannion, France, 10-12 September 2008 International