Embed Size (px)
Citation preview
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
1
EE
E68
20: S
peec
h &
Aud
io P
roce
ssin
g &
Rec
ogni
tion
Lec
ture
4:
Au
dit
ory
Per
cep
tio
n
Mo
tiva
tio
n: W
hy &
ho
w
Au
dit
ory
phy
sio
log
y
Psy
cho
phy
sics
: Det
ecti
on
& d
iscr
imin
atio
n
Pit
ch p
erce
pti
on
Sp
eech
per
cep
tio
n
Au
dit
ory
org
aniz
atio
n &
Sce
ne
anal
ysis
Dan
Elli
s <
dpw
e@ee
.col
umbi
a.ed
u>ht
tp://
ww
w.e
e.co
lum
bia.
edu/
~dp
we/
e682
0/
Col
umbi
a U
nive
rsity
Dep
t. of
Ele
ctric
al E
ngin
eerin
gS
prin
g 20
06
1 2 3 4 5 6
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
2
Why
stu
dy
per
cep
tio
n?
•P
erce
pti
on
is m
essy
: Can
we
avo
id it
?
No
!
•A
ud
itio
n p
rovi
des
th
e ‘g
rou
nd
tru
th’ i
n a
ud
io
-w
hat i
s re
leva
nt a
nd ir
rele
vant
-su
bjec
tive
impo
rtan
ce o
f dis
tort
ion
(cod
ing
etc.
)-
(the
re c
ould
be
othe
r in
form
atio
n in
sou
nd...
)
•S
om
e so
un
ds
are
‘des
ign
ed’ f
or
aud
itio
n
-co
-evo
lutio
n of
spe
ech
and
hear
ing
•T
he
aud
ito
ry s
yste
m is
ver
y su
cces
sfu
l
-w
e w
ould
do
extr
emel
y w
ell t
o du
plic
ate
it
•W
e ar
e n
ow
ab
le t
o m
od
el c
om
ple
x sy
stem
s
-fa
ster
com
pute
rs, b
igge
r m
emor
ies
1
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
3
Ho
w t
o s
tud
y p
erce
pti
on
?
Th
ree
dif
fere
nt
app
roac
hes
:
•A
nal
yze
the
exam
ple
: phy
sio
log
y
-di
ssec
tion
& n
erve
rec
ordi
ngs
•B
lack
box
inp
ut/
ou
tpu
t: p
sych
op
hysi
cs
-fit
sim
ple
mod
els
of s
impl
e fu
nctio
ns
•In
form
atio
n p
roce
ssin
g m
od
els
-in
vest
igat
e an
d m
odel
com
plex
func
tions
-e.
g. s
cene
ana
lysi
s, s
peec
h pe
rcep
tion
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
4
Ou
tlin
e
Mo
tiva
tio
n
Phy
sio
log
y
-O
uter
, mid
dle
& in
ner
ear
-T
he A
udito
ry N
erve
and
bey
ond
-M
odel
s
Psy
cho
phy
sics
Pit
ch p
erce
pti
on
Sp
eech
per
cep
tio
n
Sce
ne
anal
ysis
1 2 3 4 5 6
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
5
Phy
sio
log
y
•P
roce
ssin
g c
hai
n f
rom
air
to
bra
in:
•S
tud
y vi
a:
-an
atom
y-
nerv
e re
cord
ings
•S
ign
als
flo
w in
bo
th d
irec
tio
ns
2
Out
erea
r
Mid
dle
ear In
ner
ear
(coc
hlea
)
Aud
itory
nerv
e
Mid
brai
n
Cor
tex
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
6
Ou
ter
& m
idd
le e
ar
•P
inn
a ‘h
orn
’
-co
mpl
ex r
eflec
tions
giv
e sp
atia
l (el
evat
ion)
cue
s
•E
ar c
anal
-ac
oust
ic tu
be
•M
idd
le e
ar
-bo
nes
prov
ide
impe
danc
e m
atch
ing
Pin
na
Ear
can
al
Ear
drum
(tym
panu
m)
Mid
dle
ear
bone
s
Coc
hlea
(inne
r ea
r)
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
7
Inn
er e
ar: C
och
lea
•M
ech
anic
al in
pu
t fr
om
mid
dle
ear
sta
rts
trav
elin
g w
ave
mov
ing
do
wn
Bas
ilar
Mem
bra
ne
•V
aryi
ng
sti
ffn
ess
and
mas
s o
f B
M g
ives
res
ult
s in
co
nti
nu
ou
s va
riat
ion
of
reso
nan
t fr
equ
ency
•A
t re
son
ance
, tra
velin
g w
ave
ener
gy
is
dis
sip
ated
in B
M v
ibra
tio
n
→
Fre
qu
ency
(F
ou
rier
) an
alys
is
Coc
hlea
Ova
l win
dow
(fro
m M
E b
ones
)B
asila
r M
embr
ane
(BM
)Tr
avel
ling
wav
e
Res
onan
tfr
eque
ncy
Pos
ition
16 k
Hz
50 H
z
035
mm
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
8
Co
chle
a h
air
cells
•E
ar c
onv
erts
so
un
d t
o B
M m
oti
on
;E
ach
po
int
on
BM
co
rres
po
nd
s to
a f
req
uen
cy
•H
air
cells
on
BM
co
nver
t m
oti
on
in
to n
erve
imp
uls
es (
firi
ng
s)
•In
ner
Hai
r C
ells
det
ect
mo
tio
n
•O
ute
r H
air
Cel
ls?
Var
iab
le d
amp
ing
?
[BM
an
imat
ion
]
Coc
hlea
Bas
ilar
mem
bran
e
Tect
oria
lm
embr
ane
Inne
r H
air
Cel
l(I
HC
)O
uter
Hai
r C
ell
(OH
C)
Aud
itory
ner
ve
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
9
Inn
er H
air
Cel
ls
•IH
Cs
conv
ert
BM
vib
rati
on
into
ner
ve fi
rin
gs
•H
um
an h
ear
has
~35
00 IH
Cs;
Eac
h IH
C h
as ~
7 co
nn
ecti
on
s to
Au
dit
ory
Ner
ve
•E
ach
ner
ve fi
res
(so
met
imes
) n
ear
pea
k d
isp
lace
men
t:
•H
isto
gra
m t
o g
et fi
rin
g p
rob
abili
ty:
Loca
l BM
disp
lace
men
t
Typi
cal n
erve
sign
al (
mV
)
time
/ ms
50
Firi
ngco
unt
Cyc
lean
gle
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
10
Au
dit
ory
ner
ve (
AN
) si
gn
als
•S
ing
le n
erve
mea
sure
men
ts:
•H
ard
to
mea
sure
: pro
be
livin
g A
Ns?
(log)
freq
uenc
y
100
Hz
1 kH
z10
kH
z
20406080
dB S
PL
Ton
e bu
rst
his
tog
ram
Fre
qu
ency
th
resh
old
Spi
keco
unt
Tim
e
100
100
ms
Tone
bur
st
Spikes/sec
Inte
nsity
/ dB
SP
L
300
200
100
200
040
6080
100
One
fibe
r:~
25
dB d
ynam
ic r
ange
Hea
ring
dyna
mic
ran
ge >
100
dB
Rat
e vs
.in
ten
sity
(ap
pro
x.co
nst
ant-
Q)
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
11
AN
po
pu
lati
on
res
po
nse
•A
ll th
e in
form
atio
n t
he
bra
in h
as a
bo
ut
sou
nd
:
-av
erag
e ra
te &
spi
ke ti
min
gs o
n 30
,000
fibe
rs
•N
ot
un
like
a (c
on
stan
t-Q
) sp
ectr
og
ram
?
time
/ ms
freq / 8ve re 100 Hz
p(
)
012345
010
2030
4050
60
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
12
Bey
on
d t
he
aud
ito
ry n
erve
•A
scen
din
g a
nd
des
cen
din
g
•To
no
top
ic
×
?
-m
odul
atio
n -
pos
ition
-
sou
rce?
?(fro
m ll
oydw
atts
.com
)
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
13
Per
iph
ery
mo
del
s
•M
od
eled
asp
ects
:
-ou
ter/
mid
dle
ear
- c
ochl
ea fi
lterin
g-
hair
cell
tran
sduc
tion
- e
ffere
nt fe
edba
ck?
•R
esu
lt: ‘
neu
rog
ram
’ / ‘c
och
leag
ram
’
Out
er/m
iddl
eea
rfil
terin
gS
ound
Coc
hlea
filte
rban
k
IHC
IHC
time
/ s
channel
Sla
neyP
atte
rson
12
chan
s/oc
t fro
m 1
80 H
z, B
BC
1tm
p (2
0010
218)
00.
10.
20.
30.
40.
5
102030405060
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
14
Ou
tlin
e
Mo
tiva
tio
n
Phy
sio
log
y
Psy
cho
phy
sics
-D
etec
tion
theo
ry m
odel
ing
-In
tens
ity p
erce
ptio
n-
Mas
king
Pit
ch p
erce
pti
on
Sp
eech
per
cep
tio
n
Sce
ne
anal
ysis
1 2 3 4 5 6
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
15
Psy
cho
phy
sics
•P
hysi
olo
gy
loo
ks a
t th
e im
ple
men
tati
on
;P
sych
olo
gy
loo
ks a
t th
e fu
nct
ion
/beh
avio
r
•A
nal
yze
aud
itio
n a
s si
gn
al d
etec
tio
n:
-ps
ycho
logi
cal t
ests
refl
ect i
nter
nal d
ecis
ions
-as
sum
e op
timal
dec
isio
n pr
oces
s-
infe
r na
ture
of i
nter
nal r
epre
sent
atio
ns, n
oise
, ...
→
low
er b
ound
s on
mor
e co
mpl
ex fu
nctio
ns
•D
iffe
ren
t as
pec
ts t
o m
easu
re
-tim
e, fr
eque
ncy,
inte
nsity
-to
nes,
com
plex
es, n
oise
-bi
naur
al-
pitc
h, d
etun
ing
3
pω
O(
)
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
16
Bas
ic p
sych
op
hysi
cs
•R
elat
e p
hysi
cal a
nd
per
cep
tual
var
iab
les
-e.
g. in
tens
ity
→
loud
ness
freq
uenc
y
→
pitc
h
•M
eth
od
olo
gy:
su
bje
ct t
ests
-ju
st n
otic
eabl
e di
ffere
nce
(jnd)
-m
agni
tude
sca
ling
e.g.
‘adj
ust t
o tw
ice
as lo
ud’
•R
esu
lts
for
Inte
nsi
ty v
s. L
ou
dn
ess:
Web
er’s
law
∆
I
α
I
→
log(
L
) =
k
·log(
I
)
-20
-10
010
1.4
1.6
1.8
2.0
2.2
2.4
2.6
Sou
nd le
vel /
dB
Log(loudness rating)
Har
tman
n(1
993)
Cla
ssro
om
lou
dn
ess
scal
ing
dat
a
Pow
er la
w fi
t:
L α
I 0
.22
Tex
tboo
k fig
ure:
L α
I 0
.3
L()
2lo
g0.
3I()
2lo
g=
0.3
I10
log
210
log
--------
------
⋅=
0.3
210
log
--------
------
dB 10-------
⋅=
dB
10Ú=
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
17
Lo
ud
nes
s as
a f
un
ctio
n o
f fr
equ
ency
•F
letc
her
-Mu
nso
n e
qu
al-l
ou
dn
ess
curv
es:
•H
eari
ng
imp
airm
ent:
exa
gg
erat
esfreq
/ H
z
Intensity / dB SPL
040 2060100 80120
1000
100
10,0
00
Inte
nsity
/ dB
Equivalent loudness @ 1kHz
400
0
40
80
80100
60 20
2060
2060
0
100
60 20 0In
tens
ity /
dB
Equivalent loudness @ 1kHz
40
40
80
80
rapi
dlo
udne
ssgr
owth
100
Hz
1 kH
z
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
18
Lo
ud
nes
s as
a f
un
ctio
n o
f b
and
wid
th
•S
ame
tota
l en
erg
y, d
iffe
ren
t d
istr
ibu
tio
n:
-e.
g. 2
cha
ns a
t -6
dB (
not -
10 d
B)
•C
riti
cal b
and
s: in
dep
end
ent
freq
. ch
ann
els
-~
25
tota
l (4-
6 / o
ctav
e)
[s
ndex
]
time
freq
freq
mag
freq
mag
Sam
eto
tal
ener
gyI·B
... b
ut w
ider
pe
rcei
ved
as lo
uder
I 0I 1
B0
B1
Ban
dwid
th B
‘Crit
ical
’ba
ndw
idth
Loudness
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
19
Sim
ult
aneo
us
mas
kin
g
•A
lou
der
to
ne
can
‘mas
k’ t
he
per
cep
tio
n o
f a
seco
nd
to
ne
nea
rby
in f
req
uen
cy:
•S
ug
ges
ts a
n ‘i
nte
rnal
no
ise’
mo
del
:
mas
ked
thre
shol
d
log
freq
abso
lute
thre
shol
d
mas
king
tone
Intensity / dB
deci
sion
var
iabl
ex
inte
rnal
noi
se
p(x
| I)
p(x
| I)
p(x
| I+∆
I)
σn
I
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
20
Seq
uen
tial
mas
kin
g
•B
ackw
ard
/forw
ard
in t
ime:
-su
gges
ts te
mpo
ral e
nvel
ope
of d
ecis
ion
var.
→T
ime-
freq
uen
cy m
aski
ng
‘ski
rt’:
time
Intensity / dB
mas
ker
enve
lope m
aske
d t
hre
sho
ld
sim
ulta
neou
s m
aski
ng~
10 d
B
back
war
d m
aski
ng~
5 m
sfo
rwar
d m
aski
ng~
100
ms
time
freq
intensity
Mas
king
tone
Mas
ked
thre
shol
d
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
21
Wh
at w
e d
o a
nd
do
n’t
hea
r
•T
imin
g: 2
ms
atta
ck r
eso
luti
on
, 20m
s d
iscr
im-
but:
spec
tral
spl
atte
r
•Tu
nin
g: ~
1%
dis
crim
inat
ion
-bu
t: be
ats
•S
pec
tru
m: p
rofi
le c
han
ges
, fo
rman
ts-
varia
ble
time-
freq
uenc
y re
solu
tion
•H
arm
on
ic p
has
e?
•N
ois
y si
gn
als
& t
extu
re
•(T
race
vs.
cat
ego
rica
l mem
ory
)
AB
XX
= A
or
B?
“tw
o-in
terv
alfo
rced
-cho
ice”
:
time
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
22
Ou
tlin
e
Mo
tiva
tio
n
Phy
sio
log
y
Psy
cho
phy
sics
Pit
ch p
erce
pti
on
-‘P
lace
’ mod
els
-‘T
ime’
mod
els
-M
ultip
le c
ues
& c
ompe
titio
n
Sp
eech
per
cep
tio
n
Sce
ne
anal
ysis
1 2 3 4 5 6
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
23
Pit
ch p
erce
pti
on
:A
cla
ssic
arg
um
ent
in p
sych
op
hysi
cs
•H
arm
on
ic c
om
ple
xes
are
a p
atte
rn o
n A
N
-..
but g
ive
a fu
sed
perc
ept (
ecol
ogic
al)
•W
hat
det
erm
ines
th
e p
itch
per
cep
t?-
not t
he fu
ndam
enta
l
•H
ow
is it
co
mp
ute
d?
Two
co
mp
etin
g m
od
els:
pla
ce a
nd
tim
e
4
10203040506070
00.
050.
1tim
e/s
freq. chan.
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
24
Pla
ce m
od
el o
f p
itch
•A
N e
xcit
atio
n p
atte
rn s
ho
ws
ind
ivid
ual
pea
ks
•‘P
atte
rn m
atch
ing
’ met
ho
d t
o fi
nd
pit
ch:
•S
up
po
rt:
Lo
w h
arm
on
ics
are
very
imp
ort
ant
•B
ut:
Fla
t-sp
ectr
um
no
ise
can
car
ry p
itch
freq
uenc
y ch
anne
l
freq
uenc
y ch
anne
l
AN excitation
Pit
ch s
tren
gth
reso
lved
harm
onic
s
broa
der
HF
cha
nnel
sca
nnot
res
olve
harm
onic
s
Cor
rela
te w
ith h
arm
onic
‘sie
ve’:
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
25
Tim
e m
od
el o
f p
itch
•T
imin
g in
form
atio
n is
pre
serv
ed in
AN
d
ow
n t
o ~
1m
s sc
ale
•E
xtra
ct p
erio
dic
ity
by e
.g. a
uto
corr
elat
ion
& c
om
bin
e ac
ross
fre
qu
ency
ch
ans:
•B
ut:
HF
giv
es w
eak
pit
ch (
in p
ract
ice)
lag
/ ms
time
freqpe
r-ch
anne
lau
toco
rrel
atio
n
auto
corr
elat
ion
Su
mm
ary
auto
corr
elat
ion
010
2030
com
mon
per
iod
(pitc
h)
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
26
Alt
ern
ate
& c
om
pet
ing
cu
es
•P
itch
per
cep
tio
n c
ou
ld r
ely
on
var
iou
s cu
es-
aver
age
exci
tatio
n pa
ttern
-su
mm
ary
auto
corr
elat
ion
-m
ore
com
plex
pat
tern
mat
chin
g
•R
elyi
ng
on
just
on
e cu
e is
bri
ttle
-e.
g. m
issi
ng fu
ndam
enta
l
→P
erce
ptu
al s
yste
m a
pp
ears
to
use
a fl
exib
le,
op
po
rtu
nis
tic
com
bin
atio
n
•O
pti
mal
det
ecto
r ju
stifi
cati
on
?
if o 1
and
o2
are
cond
ition
ally
inde
pend
ent
pω
o(
)ω
argm
ax
poω
()
pω(
)⋅
po(
)----
--------
--------
--------
-------
ωar
gmax
=
po 1
ω(
)p
o 2ω
()
pω(
)⋅
⋅ω
argm
ax
=
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
27
Ou
tlin
e
Mo
tiva
tio
n
Phy
sio
log
y
Psy
cho
phy
sics
Pit
ch p
erce
pti
on
Sp
eech
per
cep
tio
n-
The
sou
nds
of s
peec
h-
Pho
nem
e pe
rcep
tion
-C
onte
xt a
nd to
p-do
wn
influ
ence
s
Sce
ne
An
alys
is
1 2 3 4 5 6
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
28
Sp
eech
per
cep
tio
n
•H
igh
ly s
pec
ializ
ed f
un
ctio
n-
subs
eque
nt to
sou
rce
orga
niza
tion?
-..
but a
lso
can
inte
ract
•K
ind
s o
f sp
eech
so
un
ds:
-vo
wel
s-
glid
es-
nasa
ls-
stop
s-
fric
ativ
es...
5
2030405060
1.4
1.6
1.8
22.
22.
42.
6tim
e/s
leve
l/dB
freq / Hz
0
1000
2000
3000
4000
wat
chth
inas
adi
me
aha
s
stop
bur
stfr
icat
ive
vow
elna
sal
glid
e
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
29
Cu
es t
o p
ho
nem
e p
erce
pti
on
•L
ing
uis
ts d
escr
ibe
spee
ch w
ith
ph
on
emes
:
-ph
onem
es d
efine
min
imal
wor
d co
ntra
sts
•A
cou
stic
-ph
on
etic
ian
s d
escr
ibe
ph
on
emes
by:
wat
chth
inas
adi
me
aha
s
md
nc
tcl
^
θz
wz
h
e
II
Iay
ε • f
orm
ants
&
tra
nsi
tio
ns
• b
urs
ts
&
on
set
tim
es
time
freq
vow
elfo
rman
ts
tran
sitio
nst
op b
urst vo
icin
g on
set t
ime
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
30
Cat
ego
rica
l per
cep
tio
n
•(S
om
e) s
pee
ch s
ou
nd
s p
erce
ived
cat
ego
rica
lly
rath
er t
han
an
alo
gic
ally
-e.
g. s
top-
burs
t & ti
min
g:
-to
kens
with
in c
ateg
ory
are
hard
to d
istin
guis
h-
cate
gory
bou
ndar
ies
are
very
sha
rp
•C
ateg
ori
es a
re le
arn
ed fo
r n
ativ
e to
ng
ue
-“m
erry
” / “
mar
y” /
“mar
ry”
T
PK
P
Pi
ea
c
ou
ε follo
win
g vo
wel
burst freq fb / Hz
1000
2000
3000
4000
time
freq
stop
bur
stvo
wel
form
ants
f b
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
31
Wh
ere
is t
he
info
rmat
ion
in s
pee
ch?
•‘A
rtic
ula
tio
n’ o
f h
igh
/low
-pas
s fi
lter
ed s
pee
ch:
-su
ms
to m
ore
than
1...
•S
pee
ch m
essa
ge
is h
igh
ly r
edu
nd
ant
-e.
g. c
onst
rain
ts o
f lan
guag
e, c
onte
xt→
liste
ners
can
und
erst
and
with
ver
y fe
w c
ues
Articulation / %
1000
20406080
2000
3000
4000
freq
/ H
z
high
-pas
slo
w-p
ass
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
32
Top
-do
wn
infl
uen
ces:
Ph
on
emic
res
tora
tio
n(W
arre
n 19
70)
•W
hat
if a
no
ise
burs
t o
bsc
ure
s sp
eech
?
-au
dito
ry s
yste
m ‘r
esto
res’
the
mis
sing
pho
nem
e...
bas
ed o
n se
man
tic c
onte
xt...
eve
n in
ret
rosp
ect!
•S
ub
ject
s ar
e ty
pic
ally
un
awar
e o
f w
hic
h
sou
nd
s ar
e re
sto
red
1.4
1.6
1.8
22.
22.
42.
6tim
e / s
freq / Hz
0
1000
2000
3000
4000
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
33
A p
red
isp
osi
tio
n fo
r sp
eech
:S
inew
ave
rep
licas
(Rem
ez e
t al.
1994
)
•R
epla
ce e
ach
form
ant
wit
h a
sin
gle
sin
uso
id:
-sp
eech
is (
som
ewha
t) in
telli
gibl
e-
peop
le h
ear
both
whi
stle
s an
d sp
eech
(“d
uple
x”)
-pr
oces
sed
as s
peec
h de
spite
un-
spee
ch-li
ke
•W
hat
do
es it
tak
e to
be
spee
ch?
010
0020
0030
0040
0050
00
0.5
11.
52
2.5
30
1000
2000
3000
4000
5000
time
/ s
freq / Hz freq / Hz
Sp
eech
Sin
es
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
34
Sim
ult
aneo
us
vow
els
•M
ix s
ynth
etic
vo
wel
s w
ith
dif
fere
nt
f 0s:
•P
itch
dif
fere
nce
hel
ps
(th
ou
gh
no
t n
eces
sary
):
freq
+=
dB
/iy/
@ 1
00 H
z
/ah
/ @
125
Hz
% both vowels correct
∆f 0
(se
mito
nes)
01
1 /4
1 /2
24
255075
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
35
Co
mp
uta
tio
nal
mo
del
s o
f sp
eech
per
cep
tio
n
•V
ario
us
theo
reti
cal -
pra
ctic
al m
od
els
of s
pee
ch
com
pre
hen
sio
n, e
.g. :
•O
pen
qu
esti
on
s:-
mec
hani
sm o
f pho
nem
e cl
assi
ficat
ion
-m
echa
nism
of l
exic
al r
ecal
l-
mec
hani
sm o
f gra
mm
ar c
onst
rain
ts
•A
SR
is a
pra
ctic
al im
ple
men
tati
on
(?
)
Pho
nem
ere
cogn
ition
Lexi
cal
acce
ssG
ram
mar
cons
trai
nts
Spe
ech
Wor
ds
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
36
Ou
tlin
e
Mo
tiva
tio
n
Phy
sio
log
y
Psy
cho
phy
sics
Pit
ch p
erce
pti
on
Sp
eech
per
cep
tio
n
Sce
ne
anal
ysis
-E
vent
s an
d so
urce
s-
Fus
ion
and
stre
amin
g-
Con
tinui
ty &
res
tora
tion
-S
imul
tane
ous
vow
els
1 2 3 4 5 5
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
37
Au
dit
ory
Org
aniz
atio
n
•D
etec
tio
n m
od
el is
hu
ge
sim
plifi
cati
on
•R
eal r
ole
of
hea
rin
g is
mu
ch m
ore
gen
eral
:R
ecov
er u
sefu
l in
form
atio
n f
rom
ou
tsid
e w
orl
d
→S
ou
nd
org
aniz
atio
n in
to e
ven
ts a
nd
so
urc
es:
•R
esea
rch
qu
esti
on
s:-
wha
t det
erm
ines
per
cept
ion
of s
ourc
es?
-ho
w d
o hu
man
s se
para
te m
ixtu
res?
-ho
w m
uch
can
we
tell
abou
t a s
ourc
e?
6
02
4tim
e/s
frq/
Hz 0
2000
4000
Voi
ce
Sta
b
Rum
ble
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
38
Au
dit
ory
sce
ne
anal
ysis
:S
imu
ltan
eou
s fu
sio
n
•H
arm
on
ics
are
dis
tin
ct o
n A
N,
but
per
ceiv
ed a
s o
ne
sou
nd
(“f
use
d”)
:
-de
pend
s on
com
mon
ons
et-
depe
nds
on h
arm
onic
ity (
com
mon
per
iod)
•M
eth
od
olo
gie
s:-
ask
subj
ect h
ow m
any
‘obj
ects
’-
mat
ch a
ttrib
utes
e.g
. obj
ect p
itch
-m
anip
ulat
e hi
gher
leve
l e.g
. vow
el id
entit
y
time
freq
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
39
Seq
uen
tial
gro
up
ing
: str
eam
ing
•P
atte
rn /
rhyt
hm
: pro
per
ty o
f se
t o
f o
bje
cts
-su
bseq
uent
to fu
sion
e
mpl
oys
fuse
d ev
ents
?
•M
easu
re b
y re
lati
ve t
imin
g ju
dg
men
ts-
cann
ot c
ompa
re b
etw
een
stre
ams
•S
epar
ate
‘co
her
ence
’ an
d ‘f
usi
on
’ bo
un
dar
ies
•C
an in
tera
ct a
nd
co
mp
ete
wit
h f
usi
on
[snd
ex]
∴
–2 o
ctav
es
TR
T: 6
0-15
0 m
s
time
frequency
∆f:
1 kH
z
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
40
Co
nti
nu
ity
& r
esto
rati
on
•To
ne
is in
terr
up
ted
by
no
ise
burs
t:W
hat
hap
pen
ed?
-m
aski
ng m
akes
tone
und
etec
tabl
e du
ring
nois
e
•N
eed
to
infe
r m
ost
pro
bab
le r
eal-
wo
rld
eve
nts
-ob
serv
atio
n eq
ually
like
ly fo
r ei
ther
exp
lana
tion
-pr
ior
on c
ontin
uous
tone
muc
h hi
gher
→ c
hoos
e
•To
p-d
ow
n in
flu
ence
on
per
ceiv
ed e
ven
ts...
pu
lsat
ion
th
resh
old
[sn
dex]
time
freq
+
++
?
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
41
Mo
del
s o
f au
dit
ory
org
aniz
atio
n
•P
sych
olo
gic
al a
cco
un
ts s
ug
ges
t b
ott
om
-up
:
-(B
row
n 19
91)
•C
om
plic
atio
ns
in p
ract
ice:
-fo
rmat
ion
of s
epar
ate
elem
ents
-co
ntra
dict
ory
cues
-in
fluen
ce o
f top
-dow
n co
nstr
aint
s (c
onte
xt,
expe
ctat
ions
) ...
inpu
tm
ixtu
resi
gnal
feat
ures
(map
s)
disc
rete
obje
cts
Fro
nt e
ndO
bjec
tfo
rmat
ion
Gro
upin
gru
les
Sou
rce
grou
ps
onse
t
perio
d
frq.
mod
time
freq
E68
20 S
AP
R -
Dan
Elli
sL0
4 -
Per
cept
ion
2006
-02-
09 -
42
Su
mm
ary
•A
ud
ito
ry p
erce
pti
on
pro
vid
es t
he
‘gro
un
d t
ruth
’ u
nd
erly
ing
au
dio
pro
cess
ing
•P
hysi
olo
gy
spec
ifies
info
rmat
ion
ava
ilab
le
•P
sych
op
hysi
cs m
easu
res
bas
ic s
ensi
tivi
ties
•S
ou
nd
so
urc
es r
equ
ires
fu
rth
er o
rgan
izat
ion
•S
tro
ng
co
nte
xtu
al e
ffec
ts in
sp
eech
per
cep
tio
n
Par
tin
g t
ho
ug
ht:
Is p
itch
cen
tral
to
co
mm
un
icat
ion
? W
hy?
Tra
nsdu
ceS
cene
anal
ysis
Mul
tiple
repr
esen
t'ns
Hig
h-le
vel
reco
gniti
onS
ound
