1
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
AbstractPerception of speech under adverse listening conditions may be improved by processing it to incorporate properties of clear speech. It needs automated detection of stop land-marks and enhancement of bursts and transition segments. A technique for accurate detection of stop landmarks in continuous speech based on parameters derived from Gaussian mixture modeling (GMM) of log magnitude spectrum is presented. Applying the technique on sentences from the TIMIT database resulted in burst detection rates of 98, 97, 95, 90, and 73 % at temporal accuracies of 30, 20, 15, 10, and 5 ms respectively.
2
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
1. INTRODUCTION
Acoustic LandmarksRegions with concentration of phonetic information, important for speech perception
Stop Landmarks Closure Release burst Onset of voicing
Closure ▲ Release burst ▲ ▲ Onset of voicing/apa/
3
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Problems in Stop PerceptionPerception of transient sounds with low intensityseverely affected by noise / hearing impairment
Clear Speech Style adapted by speakers under noisy conditions (~17 % more intelligible than conversational speech) Acoustic landmarks modified in duration & intensity
◄ Conversational▼Clear
‘the book tells a story’
4
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Speech Intelligibility Enhancement Using Properties of Clear Speech
Automated detection of landmarks with Good temporal accuracy High detection rate and low false detections
Modification of speech characteristics aroundthe stop landmarks
5
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Some Earlier Landmark Detection Techniques
Liu (1996): Rate-of-rise measures of parameters from a set of fixed spectral bands. Detection rate: 84 % at 20-30 ms, ~50 % at 5-10 ms.
Niyogi & Sondhi (2002): Optimal filtering approach with log energy, log energy in the band > 3 kHz & Wiener entropy. Detection rate 90 % at 20 ms.
6
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
ObjectiveDetection of stop landmarks using Gaussian mixture modeling (GMM) of speech spectrum
▪ for improving the temporal accuracy of detection and reducing insertion errors
▪ with adaptation to speech variability
▪ for enhancing burst and transition segments to improve speech intelligibility under adverse listening conditions
7
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
2. GAUSSIAN MIXTURE MODELING OF SHORT-TIME SPEECH SPECTRUM
Approximation of spectrum using a weighted sum of Gaussian functions
Means Variances Mixture weights
Good spectral approximation with 4 or 5 Gaussians (approximating the spectral resonances)
Adaptive to speech variability
1( ) ( , , )
Mn gn gn gn
gS k w G k
8
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Spectral Modeling Short-time log magnitude spectrum of speech signal (S.R. = 10 kHz)
6 ms Hanning windowed frames (for suppressing the harmonic structure)
1 frame per ms (for tracking abrupt variations) 512-point DFT
Estimation of GMM parameters using Expectation Maximization (EM) algorithm
9
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Estimation of GMM Parameters Spectrum treated as histogram with rectangular
bins placed at each frequency index
Iterative computation of parameters as maximum likelihood estimates Initialization
Means: Average formant frequencies [600, 1200, 2400, 3600 Hz]
Variances: Extreme formant bandwidths [160, 200, 300, 400 Hz]
Mixture weights: Equal for all Gaussians
Number of iterations: ≤ 12
10
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Example: Modeling for a segment of vowel /a/
Modeling of a segment of vowel /a/: (a) windowed segment of 6 ms, (b) log magnitude spectrum (in dB), (c) smoothened spectrum (in dB), (d) GMM approximated spectrum with dotted lines indicating the individual Gaussian components.
Ag(n)
g(n)g(n)
11
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
3. DETECTION OF STOP LANDMARKSDetection based on
Rate of change (ROC) of GMM parameters
Voicing onset offset detector Spectral flatness measure
^( ) ( ( ))( )( )
g n g
g
g
A n S nnn
12
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
SpectralFlatnessMeasure
VoicingOnset /Offset
Detection GMM ParameterEstimation
Landmark DetectionSFM(n)+g, -g peaks
Speech Signal
ROC(n)
Median Smoothing &ROC computation
Windowing & Log. mag.Spectrum Computation
GMM-ROC
13
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
GMM Rate of Change Ag, g, g smoothened by 30-point median filter ROC: First difference (time step = 2 ms)
ROC Peak → Possible location of burst onset
( ) ( ) ( ) ( ) /c Ar n r n r n r n R
4
1( ) ( ) ( )A g g s
gr n A n A n n
4
1( ) ( ) ( )g g s
gr n n n n
4
1( ) ( ) ( )g g s
gr n n n n
14
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Voicing Onset-Offset Detection [Liu, 1996] Energy variations E(n) in 0:400 Hz band (6 ms Hanning windowed segments, every 1 ms) Rate-of-rise re(n) with 26 ms time-step Voicing onset [+g]: re(n) +9dB Voicing offset [-g]: re(n) -9dB
Spectral Flatness Measure [Skowronski & Harris, 2006]
(20 ms Hanning windowed segments, every 1 ms) Fricative segments: SFM 1 Voiced segments: SFM 0
2// 2 / 22 2
1 1SFM( ) | ( ) | (2 / ) | ( ) |
NN Nn n
k kn X k N X k
15
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
Stop Landmark Detection For a voicing onset [+g] or voicing offset [-g] at t,
locate the preceding [+g] or [-g] If [-g] at t0,select GMM ROC peak at tb during (t0-50, t ms), Else select GMM ROC peak at tb during (t-50, t ms) as the burst candidate.
A burst is declared, if {SFM > 0.5 for 1 ms during (tb-15, tb+15 ms)} and {each of the norm. ampl. A2, A3, A4 < 0.5 for at least 10 ms during (t0, tb)}.
For burst at tb, closure is located at t0, and voicing onset at t.
16
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
/apa/: Waveform (a), Gaussian parameter tracks (b: 1st, c: 2nd, d: 3rd, e: 4th).
(a)
(b)
(c)
(d)
(e)
Ag(n)
g(n)
g(n)
17
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
/apa/: Waveform (a), Spectrogram (b), GMM spectrogram (c), Gaussian ROC (d)
(a)
(b)
(c)
(d)
18
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
50 100 150 200 250 300 350 ms-101
50 100 150 200 250 300 350-20
020
50 100 150 200 250 300 3500
0.51
50 100 150 200 250 300 3500
0.51
50 100 150 200 250 300 3500
0.51
Time(ms)
-g+g
(a)
(b)
(c)
(d)
(e)A2
A4
A3
/apa/: Waveform (a), -g, +g peaks (b), SFM (c), GMM ROC (d), Normalized Gaussian amplitudes for Gaussian 2, 3, 4 (e)
tb
tt0
ROC peak
SFM
19
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
4. TEST RESULTSComparison with manually labeled landmarks VCV utterances
▪ Stops /b/, /d/, /g/, /p/, /t/, /k/ & vowels /a/, /i/, /u/ ▪ 10 speakers (5 F, 5 M)
TIMIT sentences▪ 50 sentences▪ 5 speakers (3 F, 2 M)
20
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
0
20
40
60
80
100
5 10 15 20 30Temporal Accuracy (ms)
Dete
ctio
n (%
)
Closure Burst Voicing Onset
36
9076
64
92 93
73
93 9883
96 98 999893
Det. Rates for VCV Utterances
21
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
0
20
40
60
80
100
5 10 15 20 30Temporal Accuracy (ms)
Dete
ctio
n (%
)Closure Burst Voicing Onset
Det. Rates for TIMIT Sentences
Insertions : 13 % (Clicks, glottal stops : 8 %, Vowel-semivowel : 4 %, Stop to /l/, /r/ : 1 %)
19
73
45 40
90
7163
9580
919790
98 9682
22
A. R
. Jay
an, P
. C. P
and e
y, E
E De
p t. ,
IIT
Bom
b ay
5. CONCLUSIONDetection rate obtained using GMM based technique: comparable to other methods at 20-30 ms temporal accuracy, better at 10-15 ms.