VOICE RECOGNITION

AMRITA MORE – 416AASHNA PARIKH - 417

INTRODUCTION

• A user gives a predefined voice instruction to the system through microphone, the system understand this command and execute the required function.

• It facilitates the user to run windows through your voice without use of keyboard or mouse.

KEY TERMS

• Speaking Modeso Isolated Wordso Continuous Speech

• Vocabulary sizes• Language Model• Acoustic Model • Dictionary

REALIZATION OF MANDARIN SPEECH RECOGNITION SYSTEM USING

SPHINX

• Mandarin: It is the main language of China spoken by 855 million native speakers.

• Mandarin Continuous Digit Recognition System

• It is a small vocabulary speech recognition system which has only ten identity objects 0-9.

This technique builds speech recognition system using

Sphinx. It also includes Pocket sphinx, Sphinx Train, CMUCLMTK.

SPHINX

• Sphinx is a set of Java classes used in background to recognize the voice.

• It is open source and is provided by Java,

• Sphinx is built on JSAPI.

• It uses HMM algorithm and BNF grammar.

OVERALL PROCESSING

FEATURE EXTRACTION

• It generates a set of 51 dimension feature vectors which represent important characteristics of speech signals.

• It is used to convert the speech waveform to some type of parametric representation.

• A wide range of possibilities exist for parametrically representing

the speech signal. Such as LPC (Linear Prediction Coding) and MFCC (Mel Frequency Cepstral Coefficients).

Speech Data

Feature Extraction Text Data

Acoustic Model

Recognition Engine

Training

Language Model

output

Improved Acoustic Model Training

• Sphinx Train is the acoustic model training tool.

Speech Data

Feature Extraction Text Data

Acoustic Model

Recognition Engine

Training

Language Model

output

Language Model Training

Text

text2wfreq text2idngram Id-N-gram

Wfreq2vocab idngram2lm

lm3g2dmp

vocab

arpa.dmp arpa

binlm2arpa

POCKET SPHINX

• Decoding Engine

• It is used as a set of libraries that include core speech recognition functions.

• Input is audio file in wave format and the final output of recognition is displayed as text.

Feature Extraction

Acoustic Model

Recognition Engine

Training

output

Language Model

Text DataSpeech

Data

HIDDEN MARKOV MODEL (HMM)

• Real-world has structures and processes which have (or produce) observable outputs:

o Usually sequential (process unfolds over time)

o Cannot see the event producing the output

Example: speech signals

HMM Background

• Basic theory developed and published in 1960s and 70s

• No widespread understanding and application until late 80s

• Few Reasons:– Theory published in mathematic journals which were not

widely read by practicing engineers

– Insufficient tutorial material for readers to understand and apply concepts

HMM Overview

• Machine learning method

• Makes use of state machines

• Based on probabilistic model

• Can only observe output from states, not the states themselves– Example: speech recognition

• Observe: acoustic signals• Hidden States: phonemes

(distinctive sounds of a language)

HMM Components

• A set of states (x’s)

• A set of possible output symbols (y’s)

• A state transition matrix (a’s):probability of making transition from one state to the next

• Output emission matrix (b’s):probability of a emitting/observing a symbol at a particular state

• Initial probability vector:o probability of starting at a

particular stateo Not shown, sometimes assumed

to be 1

Observable Markov Model Example

• Weathero Once each day weather is

observedState 1: rainState 2: cloudyState 3: sunny

o What is the probability the weather for the next 7 days will be:

sun, sun, rain, rain, sun, cloudy, sun

o Each state corresponds to a physical observable event

RainyRainy CloudyCloudy SunnySunny

RainyRainy 0.40.4 0.30.3 0.30.3

CloudyCloudy 0.20.2 0.60.6 0.20.2

SunnySunny 0.10.1 0.10.1 0.80.8

Common HMM Types

• Ergodic (fully connected):o Every state of model can be reached in a

single step from every other state of the model

• Bakis (left-right):o As time increases, states proceed from left

to right

HMM Advantages

• Advantages:

o Effectiveo Can handle variations in record structure

Optional fieldsVarying field ordering

HMM Uses

• Speech recognition:Recognizing spoken words and phrases

• Text processing: Parsing raw records into structured records

• Bioinformatics: Protein sequence prediction

• Financial:o Stock market forecasts (price pattern prediction)o Comparison shopping services

THE LEXICAL ACCESS COMPONENT OF THE CMU CONTINUOUS SPEECH

RECOGNITION SYSTEM

• The CMU Lexical Access System hypothesizes words from a phonetic dictionary.

• Word hypothesis are anchored on syllabic nuclei and are generated independently for different parts of the utterance.

EXAMPLES

WORD SYLLABIC NUCLEI

Cat [kæt] [æ]

Parser

Verifier

Coarse labeler

Anchor Generator

Lexicon

Matcher

Front End

Lattice

Integrator

Word Hypothesizer System Diagram

MATCHING ENGINE

• Words are hypothesized by matching an input sequence of labels against the stored representation of the possible pronunciation.

• It uses the Beam search algorithm which is a modified best first search strategy.

• The beam search algorithm can simultaneously search paths with different lengths.

Parser

Verifier

Coarse labeler

Anchor Generat

or

Lexicon

Matcher

Front End

Lattice Integrator

THE LEXICON

• The lexicon (dictionary) is stored in the form of a phonetic network.

• The sources of pronunciations that have been used:o On-line phonetic dictionary, such as the Shop Dictionary.o Letter-to-sound compiler (The Talk System).

• The current CMU lexicon is constructed using a base over 150 rules covering several types of phenomena:o Including co-articulator phenomena.o Front-end characteristics.

Parser

Verifier

Coarse labeler

Anchor Generat

or

Lexicon

Matcher

Front End

Lattice Integrator

ANCHOR GENERATION

• To eliminate unnecessary matches, the voice recognition system uses syllable anchors to select locations in an utterance where words are to be hypothesized.

• The anchor generation algorithm is straight forward and is based on the following reasoning:o Words are composed of syllables, and all the syllable contain a vocalic center.o Word divisions cannot occur inside vocalic center.o The coarse labeler provides information about vocalic, non-vocalic and silent

regions.

• The algorithm is implemented in such a way that the “best” hypothesis will be generated.

Parser

Verifier

Coarse labeler

Anchor Generat

or

Lexicon

Matcher

Front End

Lattice Integrator

ANCHORS HAVE BEEN USED IN THE SYSTEM IN 2 MODES:

Single Anchor:

o In the single anchor mode, anchors of different lengths are generated and the matcher is invoked separately for each one. Although this procedure is simple, its inefficient too.

Multiple Anchor:

oThe multiple anchor mode, reduces the computations, and also reduces the number of hypothesis generated.

COARSE LABELER

• The coarse labeling algorithm is based on the ZAPDASH (Zero-crossing And Peak to peak amplitude of Differenced And Smoothed data) algorithm.

• The algorithm is robust and speaker independent, and operates reliably over a large dynamic range.

Parser

Verifier

Coarse labeler

Anchor Generat

or

Lexicon

Matcher

Front End

Lattice Integrator

PHONETIC LATTICE INTEGRATOR

• The phonetic labels produced by the front-end are grouped into four separate lattices: vowels, fricatives, closures and stops.

• The role of the integrator is to combine these separate streams and produce a single lattice consisting of non-overlapping segments.

• The integrator maps the label space used by the front-end into the label space used in the lexicon.

Parser

Verifier

Coarse labeler

Anchor Generat

or

Lexicon

Matcher

Front End

Lattice Integrator

JUNCTION VERIFIER

• The verifier basically examines junctures between words and determines whether these words can be connected together in sequence.

• The verifier deals with three classes of junctures:o Abutmentso Gapso Overlaps

Parser

Verifier

Coarse labeler

Anchor Generato

r

Lexicon

Matcher

Front End

Lattice Integrator

CONCLUSION

• Its not nearly enough detailed to actually write a speech recognizer, but it exposes the basic concepts.

• The basic concepts we learnt today to implement speech recognition are:

1. Sphinx2. Lexical Access System3. HMM Model

• The real life implementations of these techniques are still in the development phase while some are successfully launched.

• Example: Winvoice using Sphinx.

REFERENCES

• Alexander I. Rudnicky, Lynn k. Baumeister, Kevin H. DeGraaf, “The Lexical Access Component of The CMU Continue Speech Recognition”, pp. 376-379, 1987, IEEE.

• Yun Wang Xueying Zhang, “Realization of Mandarin Continuous Digits Speech Recognition”, pp. 378-380, 2010, IEEE.

• Todd A. Stephenson, “Speech Recognition with Auxiliary Information”, pp. 189-203, 2004, IEEE.