75
Introduction NSR DSR ESR Applications Speech Recognition on Mobile Devices: Distributed and Embedded Solutions Zheng-Hua Tan 1 Miroslav Novak 2 1 Department of Electronic Systems Aalborg University [email protected] 2 T. J. Watson Research Center IBM [email protected] Interspeech 2008, Brisbane, Australia, 22-09-2008 1

Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Embed Size (px)

Citation preview

Page 1: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

Speech Recognition on Mobile Devices:

Distributed and Embedded Solutions

Zheng-Hua Tan1 Miroslav Novak2

1Department of Electronic SystemsAalborg University

[email protected]

2T. J. Watson Research CenterIBM

[email protected]

Interspeech 2008, Brisbane, Australia, 22-09-2008

1

Page 2: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

About this tutorial

Provide an overview of speech recognition on mobiledevices

Cover network speech recognition, distributed speechrecognition and embedded speech recognition

Presume familiarity with speech recognition fundamentals

2

Page 3: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

Outline

1 Introduction

2 Network Speech Recognition

3 Distributed Speech Recognition

4 Embedded Speech Recognition

5 Applications

3

Page 4: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

4

Page 5: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Mobile technology

The prevalence of mobile devices: being used as digitalassistants, for communication or simply for fun.

Mobile phones: 3.5 billion by 2010

PDAs, MP3 players, GPS devices, digital cameras

The proliferation of wireless networks: being accessibleanywhere, anytime and from any devices.

3G, WLAN, Bluetooth, and IP networks

Free wireless connection for the public

5

Page 6: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Mobile technology

”To the same extent that TV transformed entertainment inthe 1960s and the PC transformed work during the 1980s,mobile technology is transforming the way that we willinterrelate in the next decade.”

- Michael Gold, SRI Consulting.

When will speech technology transform the way we interactwith mobile devices, and what shall be done to make ithappen?

6

Page 7: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Speech interfaces for mobile devices

Plus - opportunities:

Advances in mobile technology: powerful embeddedplatforms and pervasive networking

The course of miniaturisation

Used while on the move

Hands-free requirement in cars

Navigation in complex menu structures, inevitable butbeyond manageable

Minus - challenges:

Competing with existing, well-accepted UI methods liketyping on a keypad or pushing buttons

Disturbing in public places (Remember the history of mobile phone!)

Technical challenges

7

Page 8: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Technical challenges

Difficulty in porting state-of-the-art ASR systems onto mobiledevices

Computational constraints and power limitations

Diverse operating systems and hardware configurations

Imperfection of networks

Data compression

Transmission impairments

8

Page 9: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Resources and constraints of devices

Embedded platform vs. desktop PC

CPU Arithmetic RAM CacheHP iPAQ 624 MHz Fixed-point 64 MB 16 KBHP PC 3000 MHz Floating-point 8000 MB 6000 KB

Battery lifetime (around 3-5 h in a mobile phone whentalking)

In a consumer product, these resources are chosenaccording to requirements of the main functionality of thedevice.

ASR is considered but no driving forces

The targeted speech recognition application shall match theavailable resources, and optimization is necessary.

9

Page 10: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Resources and constraints of networks

Network availability: ’always-on’ networking

Networking facility is becoming a standard component onmobile devices

Network service is gradually moving towards a flat-ratesubscription-based business model

Network types: circuit-switched vs. packet-switched

Circuit-switched networks

A dedicated circuit (or channel) btw the two partiesA constant delay and a constant throughputIdeal for real-time communications

Packet-switched networks

Routing packets through shared nodes and data linksBeing more efficient and robust if delay is tolerableTo be the dominating network form (flexibility and costs)

10

Page 11: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Resources and constraints of networks

Transmission impairments:

Landline WirelessCircuit-switched reliable bit errorPacket-switched packet loss packet loss

Both bit error and packet loss tend to be burst-like, difficultto recover from

Network capacity is expanding, so are new applications. As aresult, data compression is always welcomed.

Low-bit rate compression in NSR is a source ofperformance degradation

The effect of data compression on DSR is often negligible

11

Page 12: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

12

Page 13: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Automatic speech recognition

Modern ASR systems are firmly based on the principles ofstatistical pattern recognition, in particular the use of hiddenMarkov models (HMMs).

The most likely sequence of words W ′ is found throughBayesian decision rule:

W′ = arg maxW

P(W|O) = arg maxW

P(O|W)P(W)

P(W) is the a priori probability of observing specifiedword sequence W and is given by a language model

P(O|W) is the probability of observing speech data Ogiven word sequence W and is determined by an acousticmodel.

13

Page 14: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Architecture of an ASR System

NSR CLIENT SERVER DSR CLIENT SERVER ESR CLIENT

Fig. 1.1 Architecture of an ASR system.

Feature Extraction

ASR Decoder

Acoustic Model

Lexicon LanguageModel

Language Model Generation

Grammar Text Corpora

Application

Speech Signal

Front-End Back-End

O 'W

)(ty

LM

After [Tan and Varga, 2008].

14

Page 15: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

ASR components

Feature extraction

Mel-frequency cepstral coefficients (MFCC)Signal processing for robustness

ASR decoding

Calculation of observation likelihood (based on acousticmodels with millions of parameters)Search (in an HMM network formed by language model,lexicon and sub-phonetic units)

15

Page 16: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Architectural solutions for ASR on devices

Rule of thumb for data-intensive computing is to placecomputation where the data is, instead of moving the data tothe point of computation [Bryant, 2007].

A remote ASR may be preferable when

The ASR requires more data from the network than fromthe microphone

The ASR computation is a big burden for the device

A quick implementation is required

Humans assist the ASR in the background to providesemi-automatic speech transcription service

An embedded ASR may be preferable when ...

16

Page 17: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Devices and networks Automatic speech recognition

Architecture of an ASR System

NSR CLIENT SERVER DSR CLIENT SERVER ESR CLIENT

Fig. 1.1 Architecture of an ASR system.

Feature Extraction

ASR Decoder

Acoustic Model

Lexicon LanguageModel

Language Model Generation

Grammar Text Corpora

Application

Speech Signal

Front-End Back-End

O 'W

)(ty

LM

The decision on where to place the ASR componentsdistinguishes three approaches: NSR, DSR and ESR.It is driven by factors including device and network resources,ASR components complexity and application.

17

Page 18: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

18

Page 19: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

Network speech recognition

Network speech recognition

Remote speech recognition that uses conventional speechcoders for the transmission of speech from a client device to arecognition server where feature extraction and recognitiondecoding take place.

Pros:Ubiquitous presence of codec on mobile devices

Plug and play, without touching the massive clientsThe only possibility for devices like a telephone

Cons:Network dependency and error-prone channels

Inter-frame dependency in coding

Distortion introduced by low bit-rate codingLinear prediction coding (LPC) vs. MFCCs

19

Page 20: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

Network speech recognition

Two ways to extract ASR features from the bitstream

(a) Reconstruction and feature extraction:NSR = a CODEC system + an ASR system

(b) Feature estimation without reconstruction -bitstream-based front-end [Kim and Cox, 2001]

ASR decoder

Channel Speech encoder Bitstream-based feature extraction

Feature extraction ASR decoder

Channel Speech encoder Speech decoder

(a)

(b)

20

Page 21: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

21

Page 22: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

Speech coding standards

ITU-T:

G.711 PCM 64 kbps (u-law, A-law)G.722.1 24 kbps, 32 kbps, 16k samples/s widebandG.723.1 ACELP 5.3 kbps, 6.3 kbps (mostly in VoIP)G.728 LD-CELP 16 kbpsG.729 CS-ACELP 8 kbps (mostly in VoIP)

GSM:

GSM-FR (Full Rate) (RPE-LTP) 13 kbpsGSM-EFR (Enhanced Full Rate) (ACELP) 12.2 kbps

3GPP:

AMR-NB 4.75-12.2 kbpsAMR-WB 6.6-23.85 kbps

IS-136 TDMA

IS-641 ACELP 7.4 kbps

22

Page 23: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

Effect of speech coding on ASR performance

Tourist info task (5kw vocab) [Besacier et al., 2001]WER%

MPEG Lay2 64 kbps 7.5None 7.7MPEG Lay3 64 kbps 7.8G.711 64 kbps 8.1G.723.1 5.3 kbps 8.8MPEG Lay1 32 kbps 27.0MPEG Lay3 8 kbps 66.2MPEG Lay2 8 kbps 93.8

Connected digit recognition [Kim and Cox, 2001]WER%

Wireline ASR 3.83IS-641 5.25Bitstream-based 3.76

23

Page 24: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

Effect of speech coding on ASR performance

Aurora 2 database is the TI digit database artificially distortedby adding noise and using a simulated channel distortion[Hirsch and Pearce, 2000].

WER% for Aurora 2 when training and testing recognizer inthe same coding mode [Hirsch, 2002]

PCM 26.77GSM-EFR 28.56AMR475 29.84ALAW 29.85AMR102 31.62GSM-FR 31.69GSM-HR 33.56

GSM-EFR performs the best among the codecs.

24

Page 25: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

25

Page 26: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications Speech coding Transmission errors

Effect of transmission error on ASR performance

Aurora 2 database (clean speech only, baseline WER% being1.77) [Kiss, 2000]

Error-free EP1 EP2 EP3GSM-EFR 2.53 3.02 4.35 12.87DSR 2.01 2.01 2.06 8.98

Network speech recognition

Supports a wide range of devices in a plug and playfashion

Has low requirement for the client devices

Suffers from coding distortion, especially when it iscoupled with transmission errors

Suffers from transcoding distortion in heterogeneousnetworks

26

Page 27: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

27

Page 28: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Distributed speech recognition

Distributed speech recognition

Remote speech recognition that adopts the client-serverarchitecture by placing the feature extraction in the client andthe computation-intensive recognition decoding in the server.

Pros:The absence of coding and transcoding problemsRobustness against comm. channel & acoustic noiseThin client, easy to update, no limits in ASR complexitySever-side playback, semi-automatic transcriptionSpeech data collection for AM/LM adaptation (like search engines)

Cons:Front-end must be implemented in the device(Not an issue if the application requires a client-side installation anyway.)

Network dependency and transmission errors28

Page 29: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Architecture of a DSR System

Fig. 1.2 Diagram of a DSR system.

Speech signal Source Coding

Channel Coding

Channel Decoding

Error-Prone Channel

Server-based error concealment

Client-based error recovery

ASR-DecoderEC

Feature Reconstruction

Error Resistance

Feature Extraction

ASR Decoder

Acoustic Model

Lexicon Language Model

Applications Error Concealment

Source Decoding

)(ty O

'W

Error Protection

From [Tan and Varga, 2008].

29

Page 30: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

ETSI STQ-Aurora DSR front-end

Mel-cepstrum front-end and compression [ES 201 108]:

13 Mel Cepstra Framing Pre-

emphasis Hamming Window

FFT Mel-scale filterbank

| | DCT

Log energy logE

Log

Ot

Feature compression

Zheng-Hua Tan

Tttttttt Eccccc ] log, , , ..., , , [O 0121121=

TTtTtTt ] ]S[ , ]S[ ..., , ]S[ [ 650=Feature-pair

Subvector28 bits

Subvector16 bits

44 bits

30

Page 31: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

ETSI STQ-Aurora DSR front-end

Frame-pair architecture

Frame 1 Frame 2 CRC 1-2 ... CRC 23-24<44 bits > <44 bits> <4 bits> ... <4 bits>< 138 octets / 1104 bits > for 12 frame-pairs

Multiframe

Sync Seq Header Frame packet2 octets 4 octets 138 octets< 144 octets / 1152 bits > for 240 ms

Bitrate

4.8 kbps with a payload of 4.4 kbps

31

Page 32: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

DSR processing

The objectives of DSR processing are to achieve

Low bandwidth requirement

High error-robustness

Low complexity and delay

DSR processing is all about redundancy:

Source coding: reduce redundancy

Channel coding: add redundancy

Error concealment: exploit redundancy

32

Page 33: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

33

Page 34: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Redundancy in speech features

July 1, 2008Acoustics 2008, Paris 1

Time (ms)

13 Mel Cepstra Framing Pre-

emphasis Hamming Window

FFT Mel-scale filterbank

| | DCT

Log energy logE

Log

Ot

November 1, 2005IEEE MMSP 2005, Shanghai, China 5

ms 00 4535252010

10 ms frame shift15 ms overlap

25 ms frame length34

Page 35: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Correlation within and across MFCC Vectors

Temporal correlation (redundancy) in feature stream due to

The overlapping in feature extraction processingThe speech production process itself

Correlation within and across MFCC vectors (from[So and Paliwal, 2008]):

Quantization of Speech Features: Source Coding

the Aurora-2 database (Hirsch and Pearce 2000). The MFCCs consist of 13 cepstral coefficients, 12

0}{ iic . The log energy coefficient log E, which is often concatenated with the MFCC feature set in ASR, has not been included. Rather than presenting a 13 13 matrix of coefficients, we have plotted the absolute value of the covariance coefficients in Fig. 7.6. Because of the large difference in magnitude of the variance of c0 compared with those of the other cepstral coefficients, we have applied a square root operation to the covariance coefficients to compress the dynamic range. There-fore, the coefficients on the diagonal represent the standard deviation of each cepstral coefficient rather than the variance.

We can see that a large percentage of the energy is contained in the zeroth cep-stral coefficient, c0. Recall that the final stage of MFCC computation comprises a discrete cosine transform (DCT), which tends to compact most of the energy into the zeroth cepstral coefficient or DC component. In addition, most of the off-diagonal covariance coefficients have low magnitude, which indicates that the cepstral coeffi-cients are weakly correlated with each other—apart from c0, where the cross-variance with the other cepstral coefficients appears to be higher. This suggests that the other cepstral coefficients 12

1}{ iic contain some information of the zeroth cepstral coefficient. Hence, in most speech recognition systems, c0 is not included in the feature set.

12

34

56

78

910

1112

13

12

34

56

78

910

1112

13

0

10

20

30

40

50

60

Column number of covariance matrixRow number of covariance matrix

Squ

are

root

of c

ovar

ianc

e co

effic

ient

Standard deviation of c0

Standard deviation of c1

Standard deviation of c2

Fig. 7.6 Graphical representation showing the absolute value of the covariance coefficients of MFCCs within a single vector with compressed dynamic range (log energy is not included)

143

Stephen So and Kuldip K. Paliwal Because the efficiency of scalar quantization is generally optimal when the vector

components are not correlated (which is the basis of block quantization), the covari-ance statistics of MFCCs (shown in Fig. 7.6) suggest that directly scalar quantizing the MFCCs may not be optimal. In which case, a further transform (such as the KLT) may be required to remove the remaining correlation and henceforth improve the rate-distortion performance.

This improvement will be become apparent when comparing the results between the scalar quantizer and the block quantizer.

Correlation across Successive MFCC Vectors (Interframe Dependencies)

In order to examine the correlation across successive MFCC vectors, we concatenate these vectors to form higher dimensional vectors and compute the covariance matrix of this new vector set. Any linear dependencies between MFCCs in successive vec-tors will be shown by large off-diagonal coefficients in the corresponding rows and columns of the covariance matrix. Figure 7.7 is similar to Fig. 7.6, where the covari-ance matrix is graphically represented in a three dimensional representation. We also present the graphical covariance matrix representation for two, three, four, and five

Fig. 7.7 Graphical representation showing the coefficients of the covariance matrix of MFCCs within a multiple successive vectors with compressed dynamic range: a two vectors, b three vectors, c four vectors, and d five vectors

144

60

(a) b)

(c) (d)

40

20

0

60

40

20

0

10

1020

3040

502030

4050

60

40

20

0

1020

3040

5060 10

2030

40 5060

60

40

20

0

10

5 10 15 20 25 30 3520

30

510

1520

25 510

1520

25

35

Page 36: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

36

Page 37: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Source coding

Source coding is to compress information for transmission overbandwidth-limited channels.

Transmission of uncoded feature vectors requires a bitrate of41.6 kbps

13 MFCCs, 100 Hz frame rate and 32 bit floating pointvalue

State-of-the-art DSR quantization techniques can achieve abitrate of 300 bps [So and Paliwal, 2008].

Quantization is a process of lossy coding with the challengebeing the rate-distortion trade-off.

37

Page 38: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Quantization

Scalar quantization (SQ): input samples are quantizedindividually

Vector quantization (VQ): input samples are quantized asvectors [Digalakis et al., 1999]

Split VQ: each vector is partitioned into subvectors which arethen independently quantized, as done in the DSR front-end:Ot = [[St

0]T , ...[St6]T ]T

Lower storage and computational requirement than fullVQSignificantly better performance than SQ at any bit-rate

Block quantization (transform coding)

38

Page 39: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Quantization

Scalar quantization (SQ)

Vector quantization (VQ)

Block quantization (transform coding): the componentsof a block of samples are decorrelated by using a lineartransformation (eg DCT, PCA) before SQ

2D-DCT [Zhu and Alwan, 2001]GMM-based block quantization [So and Paliwal, 2006]Efficient but with drawbacks:Inter-frame coding exploits correlation across consecutiveMFCC vectors, so error in one frame has considerableimpact on the quality of the following frames.

39

Page 40: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Histogram-based quantization

Acoustic noise may move feature vectors to a differentquantization cell in a fixed VQ codebook, introducing extradistortion!

From [Wan and Lee, 2008].

HQ: The partition cells are dynamically defined by thehistogram of a segment of the most recent past values of theparameter to be quantized.

40

Page 41: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Histogram-based quantization

A dynamic quantization, based on signal local statistics, not onany distance measure, nor related to any pretrained codebook.

Aurora2 (SetA,B,C) (WER%) From [Wan and Lee, 2008].MFCC SVQ 4.4k 2DDCT 1.45k HVQ1.9k HQ3.9k38.92 43.49 40.11 22.76 18.74

HQ is also better than methods like MVA, PCA and HEQ.

41

Page 42: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Source coding & error-resistance

A low bit-rate source coding method is highly sensitive totransmission errors.

There is a trade-off between the error-resistance and the lowbit-rate achieved by the removal of redundancy.

No free lunch theorem

Coding efficiency multiplied by robustness is constant.[Ho, 1999]

So, error recovery and concealment has a role to play ...

42

Page 43: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

43

Page 44: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error-robustness techniques

Server-based error concealment

ARQ InterleavingFEC

PassiveActive

Feature-reconstruction ASR-decoder EC

Error detection

Joint codingClient-based Server-based

Parity check CRC Checksum Block Convolutional Statistical InterpolationInsertion

Client-based error recovery

Soft-feature decoding

Error-robustness techniques

From [Tan et al., 2005].

44

Page 45: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error detection

Error detection methods

CRC (cyclic redundancy check), linear block codesconsistancy test

Data block size

17

0

5

10

15

20

25

30

35

EP1 EP2 EP3 GSM EPs

Erro

r Rat

e (%

)

Frame-pairOne-frameSub-vector1Sub-vector2

45

Page 46: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error recovery - client based techniques

Channel coding:Forward error correction (FEC) [Borgstrom et al., 2008]

media-specific FECmedia-independent FEC: e.g. (n, k) block encoding(Reed-Solomon, BCH, Golay)

Multiple description coding (MDC): encoding a sourceinto 2+ substreams to be delivered on separate channelsJoint source and channel coding: UEP (unequal errorprotection)

Packetization

Interleaving: to counteract burst errors at the cost ofdelay [Milner and James, 2006]

46

Page 47: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error recovery - client based techniques

A common attribute is the participation of the client aimed atexploiting the characteristics of channels and signals.

It is always a trade-off btw the achieved performance and therequired resources:

FEC trades bandwidth for redundancy

MDC trades multiple channels for uncorrelatedtransmission errors among descriptions

Interleaving trades delay for randomizing errordistribution.

One disadvantage is their weak compatibility.

47

Page 48: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error concealment - server based techniques

EC generally deploys the strong temporal correlation residingin speech features and uses the statistical info about speech.

EC techniques

Feature-reconstruction EC: create a substitution as closeto the original as possible.

ASR-decoder EC: modify ASR decoder to handledegradations introduced by transmission errors - uniqueto DSR

48

Page 49: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error concealment - server based techniques

Feature-reconstruction EC:

Insertion-based techniques: splicing, mean valuesubstitution, repetition

Interpolation-based techniques: linear, cubic

Soft-feature decoding based techniques[Peinado et al., 2003]

Statistical-based techniques: use a priori info aboutspeech features [Gomez et al., 2003]

ASR-decoder EC:

Weighted Viterbi decoding [Cardenal-Lopez et al., 2004],[Tan et al., 2007]

Uncertainty decoding [Ion and Haeb-Umbach, 2006]

49

Page 50: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Repetition EC at subvector level

EC generally operates at vector level, yet error rates forsubvector are significantly lower [Tan et al., 2007].

Advanced Speech Processing, MM7, Zheng-Hua Tan, 2007 1

Buffering matrix

Consistency test

))1())1()1((( ))0())0()0((( 11j

tj

tjj

tj

tj TSSdORTSSd >−>− ++

Subvector concealment (cont.)

B2NA1-2NA2A1A V V V . V V V ++++A

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢⎢⎢⎢

+−+++

+−+++

+−+++

+−+++

+−+++

+−+++

+−+++

BNNA

BNNA

BNNA

BNNA

BNNA

BNNA

BNNA

62

612

62

61

66

52

512

52

51

55

42

412

42

41

44

32

312

32

31

33

22

212

22

21

22

12

112

12

11

11

02

012

02

01

00

.

.

.

.

.

.

.

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

Advanced Speech Processing, MM7, Zheng-Hua Tan, 2007 2

Consistency matrix and subvector concealment

B8A7A6A5A4A3A2A1A V V V V V V V V V V ++++++++A

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢⎢⎢⎢

1110011001111111111110011111111111111001111111111111100111111110000111

0 for inconsistent 1 for consistentC =

6

5

4

3

2

1

0

SSSSSSS

Subvector concealment (cont.)

50

Page 51: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Weighted Viterbi decoding

Weighted Viterbi decoding

δt(j) = maxi

[δt−1(i)aij ][bj(Ot)]γ(t)

γ(t) =

{αn, n = 1...N/2

1− αN−n+1, n = N/2 + 1...N

Feature-based weighted Viterbi decoding

δt(j) = maxi

[δt−1(i)aij ]K∏

k=1

[bj(ot(k))]γk (t)

γk(t) =

{αd(ot(k),ot+1(k))/Tk , S t

j consistentγk(t + p).β|p|, ot(k) substituted by ot+p(k)

51

Page 52: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Uncertainty decoding

In standard form, state emission prob. (modelled by GMM) is

bj(Ot) = p(Ot |sj) =K−1∑k=0

wjkN(Ot ; µjk ,Σjk)

where Ot is the observing vector, and sj is the state.

In uncertainty decoding, Ot is considered corrupted and theuncorrupted, unobservable vector X is a random variable witha distribution p(X|Ot).

Integration over the feature uncertainty:

bj(Ot) =

∫p(X|Ot)bj(X)dX =

K−1∑k=0

wjkN(µX|Ot ; µjk ,Σjk+ΣX|Ot )

The standard HMM decoding remains, but the variance ofeach Gaussian is increased.

52

Page 53: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error concealment - server based techniques

Remarks:

No requirement for modifying the client-side of DSR,compatible with the ETSI-DSR standards

Repetition EC works pretty well with short burst length

Statistical based techniques benefit from a prioriknowledge of speech and is useful in particular when burstlength is long

ASR-decoder based techniques are unique for DSR andcan be applied in combination with other EC

Computational cost is of concern

53

Page 54: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

A frame-rate perspective

Strong temporal correlation in speech features

ASR performance is intact with a frame loss rate (shortburst-length) of 50% (From [James and Milner, 2004])

So why not deliberately drop some speech frames (e.g.applying HFR, VFR), and then conducting repetition based”error concealment”?

54

Page 55: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Half frame-rate front-end

Aurora 2 database, WER% [Tan et al., 2007]16-state HMM 8-state HMM

Full frame rate 1.00 6.3HFR-Duplication 1.02 5.84HFR-NoDuplication 10.63 1.40

This motives a number of coding schemes (e.g. MDC,interleaving), which exploit temporal correlation of speech forerror-robust and bandwidth-flexible DSR.

55

Page 56: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

HFR motivated coding schemes

(a) ETSI-DSR front-end frame-pair scheme

(b) One-frame scheme

(c) HFR scheme

(d) Interleaving12 scheme

(e) Interleaving24 scheme

24

1 2 3 4 5 6 7 8 9 10 11 12

20 21 22 2313 14 15 16 17 18 19

24

1 2 3 4 5 6 7 8 9 10 11 12

20 21 22 2313 14 15 16 17 18 19

1 3 5 7 9 11 13 15 17 19 21 23

24

1 3 5 7 9 11 13 15 17 19 21 23

16 18 20 222 4 6 8 10 12 14

24

1 3 5 7 9 11 2 4 6 8 10 12

16 18 20 2213 15 17 19 21 23 14

56

Page 57: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Variable frame-rate front-end

A posteriori SNR weighted energy based variable frame rateanalysis [Tan and Lindberg, 2008]

Frame selection based on the a posteriori SNR weightedenergy distance of two consecutive frames:

D(t) = | log E (t)− log E (t − 1)| · SNRpost(t)

Frame selection example

Beneficial for source coding and noise robustness: at 1.5kbps, WERs are 1.2% and 32.8% for clean and noisyspeech (vs no compression: 1.0% and 38.7%).

57

Page 58: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Error-robustness performance on Aurora 2, EP3

WER (%) Bit-rate (bps) Complexity Compatibility with

ETSI-DSR standards

Splicing 24.00 4 800 Low Yes

No CRC 8.88 4 600 Low No

Linear interpolation 7.35 4 800 Low Yes

Repetition (Aurora) 6.70 4 800 Low Yes

Weighted Viterbi 4.78 4 800 Low Yes

RS(32, 16) 3.45 9 600 High No

One-frame 3.41 5 000 Low No

Uncertainty decoding 3.20 4800 Medium Yes

Subvector 2.65 4 800 Low Yes

Interleaving12 2.43 4 800 Low No

Subvector + WVD 2.01 4 800 Low Yes

Uncertainty decoding

(inter-frame correlation)

1.98 4800 Medium Yes

H-MAP 1.91 4 800 High Yes

Interleaving24 1.74 4 800 Low No

H-FBMMSE 1.34 4 800 High Yes

MDC 1.04 5 200 Low No

Error-free 0.95 4 800 - -

58

Page 59: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

59

Page 60: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Overview of DSR standards

Mel-cepstrum DSR front-end (FE) [ES 201 108]

ETSI STQ-Aurora, 2000

Advanced DSR front-end (AFE) [ES 202 050]

ETSI STQ-Aurora, 200253% error rate reduction in acoustic noise

Extension for speech construction and tonal languages(XFE & XAFE) [ES 202 211], [ES 202 212]

ETSI STQ-Aurora, 2003

Fixed point specifications for AFE and XAFE[3GPP TS 26.243]

3GPP, 2004

60

Page 61: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Advanced front-end

From [ES 202 050]

Significant improvement over the basic front-end in noiserobustness

61

Page 62: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Extended front-ends

Objectives of the extended front-ends

Support speech construction and tonal languages.

Development trend of DSR and speech codecs:

A convergence, though with different optimizationobjectives [Kim, 2008], [Milner and Shao, 2007].

62

Page 63: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

AMR vs. DSR

Aurora databases (WER%) using AFE [Kelleher et al., 2002]DSR 4.4kbps AMR 12.2kbps AMR 4.75kbps

Aurora 2 12.6 15.3 18.7Aurora 3 9.6 11.6 14.5

Aurora 2 database (WER%) [Kiss, 2000]EP1 EP2 EP3

GSM-EFR 3.02 4.35 12.87DSR 2.01 2.06 8.98

Extensive comparison organised by 3GPP and conducted byindustry [3GPP TR 26.943].

63

Page 64: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

1 IntroductionDevices and networksAutomatic speech recognition

2 Network Speech RecognitionSpeech codingTransmission errors

3 Distributed Speech RecognitionProperties of MFCCsQuantizationError recovery and concealmentStandardsSystems

4 Embedded Speech Recognition

5 Applications

64

Page 65: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Remote speech recognition system

Microsoft R©Response PointTMis an innovative phone systemsoftware (VoIP enabled).

”Response Point is an example of using the righttechnology for the right context and application. Theblue button/voice recognition makes it easier for peopleto take the advantage of todays speech technology.”- X.D. Huang

65

Page 66: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Remote speech recognition system

PromptuTMprovides multimodal solutions for mobile devicesusing client-server speech recognition technology.

66

Page 67: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Remote speech recognition system

vlingo systems allow you to say anything to your mobile phoneand still be recognized properly.

Hierarchical Language Model Based Speech Recognition

Adaptation

67

Page 68: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

A configurable DSR system

4 Chapter 6

concealment is conducted for feature reconstruction. Secondly, the error-corrected speech packages are decoded into a set of cepstral features and VAD information. Subsequently, the cepstral features are processed by the SPHINX speech recogniser. The recogniser presents its result (either the best or N-best results) at the utterance end – detected by the VAD information - and transmits back to the Result Listener of the client. To increase system usability and flexibility, three typical recognition modes are represented, namely: Isolated word recognition, Grammar based recognition and Large vocabulary recognition. Each is defined by a set of prototype files at the server side. The choice is done at system initialisation, and specific settings can be changed at any time. The setting may be different across a group of end-users.

Figure 6-2. The system architecture

A Command Processor is implemented at both the client and server side to support the interchange of configuration commands. Potential commands include control commands to start or stop recognition, choice of recognition

From [Xu et al., 2006].68

Page 69: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

A configurable DSR system

Real-time efficiency in using different realisations of theAFE (Advance Front-End) and an H5550 IPAQ with a400 MHz XScale CPU and 128 MB memory

Algorithm FloatingP FixedP FixedP + FFT Optim.X Real time 3.98 0.82 0.69

69

Page 70: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications MFCC Quantization ER.EC Standards Systems

Distributed multimodal services

From [Pearce et al., 2005]

70

Page 71: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

References and further reading I

Part I. Network and Distributed Speech Recognition

3GPP TS 26.243

“ANSI C Code for the fixed-point distributed speech recognition extended advanced front-end.” 2004.

3GPP TR 26.943

“Recognition performance evaluations of codecs for Speech Enabled Services(SES).” 2004.

Besacier et al.

“The effect of speech and audio compression on speech recognition performance.”in IEEE Multimedia Signal Processing Workshop, Cannes, France, October 2001.

Borgstrom, B.J., Bernard, A. and Alwan, A.

“Error recovery: channel coding and packetization.”in Z.-H. Tan, and B. Lindberg (eds.), Automatic speech recognition on mobile devices and overcommunication networks, Springer, 2008.

Bryant, R.

“Data-intensive supercomputing: The case for DISC.”CMU Technical Report CMU-CS-07-128, May 2007.

Cardenal-Lopez et al.

“Soft decoding strategies for distributed speech recognition over IP newtworks.”in Proc. ICASSP, Montreal, Canada, 2004.

Digalakis, V., Neumeyer, L. and Perakakis, M.

“Quantization of cepstral parameters for speech recognition over the World Wide Web.”IEEE J. Select. Areas Communications, vol. 17, no. 1, pp. 82-90, 1999.

169

Page 72: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

References and further reading II

ETSI Standard ES 201 108

“Distributed Speech Recognition; Front-end Feature Extraction Algorithm; Compression Algorithm, v1.1.2.”2000.

ETSI Standard ES 202 050

“Distributed speech recognition; advanced front-end feature extraction algorithm; compression algorithm.”2002.

ETSI Standard ES 202 211

“Distributed speech recognition; extended front-end feature extraction algorithm; compression algorithm,back-end speech reconstruction algorithm.” 2003.

ETSI Standard ES 202 212

“Distributed speech recognition; extended advanced front-end feature extraction algorithm; compressionalgorithm, back-end speech reconstruction algorithm.” 2003.

Gomez, A.M., Peinado, A.M., Sanchez, V., and Rubio, A.J.

“A source model mitigation technique for distributed speech recognition over lossy packet channels.”in Proc. Eurospeech, Geneva, Switzerland, 2003.

Hirsch, H.G.

“The influence of speech coding on recognition performance in telecommunication networks.”in Proc. ICSLP, Denver, USA, September 2002.

Hirsch, H.G. and Pearce D.

“The Aurora experimental framework for the performance evaluation of speech recognition systems undernoisy conditions.”in Proc. ISCA ITRW ASR, Paris, France, 2000.

170

Page 73: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

References and further reading III

Ho, Y.-C.

“The no free lunch theorem and the human-machine interface.”IEEE Control Syst., 8-10, June 1999.

Ion, V. and Haeb-Umbach, R.

“Uncertainty decoding for distributed speech recognition over error-prone networks,”Speech Communication, vol. 48, pp. 1435-1446, 2006.

James, A.B. and Milner, B.P.

“Towards improving the robustness of distributed speech recognition in packet loss.”in Proc. COST278 & ISCA Research Workshop on Robustness Issues in Conversational Interaction,Norwich, UK, 2004.

Kelleher, H, Pearce, D., Ealey, D. and Mauuary, L.

“Speech recognition performance comparison between DSR and AMR transcoded speech.”in Proc. ICSLP, Denver, USA, September 2002.

Kim, H.K.

“Speech recognition over IP networks”in Z.-H. Tan, and B. Lindberg (eds.), Automatic speech recognition on mobile devices and overcommunication networks, Springer, 2008.

Kim, H.K. and Cox, R.V.

“A bitstream-based front-end for wireless speech recognition on IS-136 communications system.”IEEE Trans. Speech and Audio Processing, vol. 9, no. 5, pp. 558-568, 2001.

Kiss, I.

“A comparison of distributed and network speech recognition for mobile communication systems.”in Proc. ICSLP, Beijing, China, October 2000.

171

Page 74: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

References and further reading IV

Milner, B. P. and James, A. B.

“Robust speech recognition over mobile and IP networks in burst-like packet loss.”IEEE Trans. Audio, Speech and Language Processing, vol. 14, no. 1, pp. 223-231, 2006.

Milner, B. and Shao, X.

“Prediction of fundamental frequency and voicing from Mel-frequency cepstral coefficients forunconstrained speech reconstruction.”IEEE Transactions on Audio, Speech and Language Processing, vol. 15, no. 1, pp. 24-33, 2007.

Pearce, D., Engelsma, J., Ferrans, J. and Johnson, J.

“An architecture for seamless access to distributed multimodal services.”in Proc.INTERSPEECH, Lisbon, Portugal, September 2005.

Peinado, A., Sanchez, V., Perez-Cordoba, J., and de la Torre, A.

“HMM-based channel error mitigation and its application to distributed speech recognition.”Speech Communication, vol. 41, pp. 549-561, 2003.

S.So, and K.K. Paliwal,

“Scalable distributed speech recognition using Gaussian mixture model-based block quantization,”Speech Communication, vol. 48, pp. 746–758, 2006.

S.So, and K.K. Paliwal,

“Quantization of speech features: Source coding.”in Z.-H. Tan, and B. Lindberg (eds.), Automatic speech recognition on mobile devices and overcommunication networks, Springer, 2008.

Tan, Z.-H., Dalsgaard, P. and Lindberg, B.

“Automatic speech recognition over error-prone wireless networks.”Speech Communication, vol. 47, no. 1-2, pp. 220-242, 2005.

172

Page 75: Speech Recognition on Mobile Devices: Distributed and ...kom.aau.dk/~zt/doc/MobileASR_Tan.pdf · Speech Recognition on Mobile Devices: Distributed and Embedded Solutions ... circuit-switched

Introduction NSR DSR ESR Applications

References and further reading V

Tan, Z.-H., Dalsgaard, P. and Lindberg, B.

“Exploiting temporal correlation of speech for error-robust and bandwidth-flexible distributed speechrecognition.”IEEE Trans. on Audio, Speech and Language Processing, vol. 15, no. 4, pp. 1391-1403, 2007.

Tan, Z.-H. and Lindberg, B.

“A Posteriori SNR Weighted Energy Based Variable Frame Rate Analysis for Speech Recognition.”in Proc. Interspeech, Brisbane, Australia, 2008.

Z.-H..Tan, and I. Varga,

“Network, distributed and embedded speech recognition: an overview,”in Z.-H. Tan, and B. Lindberg (eds.), Automatic speech recognition on mobile devices and overcommunication networks, Springer, 2008.

Wan, C.-Y. and Lee, L.-S.

“Histogram-based quantization for robust and/or distributed Speech Recognition.”IEEE Trans. on Audio, Speech and Language Processing, vol. 16, no. 4, pp. 859-873, 2008.

Xu, H., Tan, Z.-H., Dalsgaard, P., Mattethat, R. and Lindberg, B.

“A configurable distributed speech recognition system.”in H. Abut, J.H.L. Hansen, K. Takeda (eds.), Digital Signal Processing for In-Vehicle and Mobile Systems 2,Springer Science, New York, 2006.

Zhu, Q. and Alwan, A.

“An efficient and scalable 2D DCT-based feature coding scheme for remote speech recognition.”in Proc. ICASSP, Salt Lake City, USA, 2001.

173