11

7-Speech Recognition7-Speech Recognition

Speech Recognition Concepts Speech Recognition Concepts

Speech Recognition ApproachesSpeech Recognition Approaches

Recognition TheoriesRecognition Theories

Bayse RuleBayse Rule

Simple Language ModelSimple Language Model

P(A|W) Network TypesP(A|W) Network Types

22

7-Speech Recognition (Cont’d)7-Speech Recognition (Cont’d)

HMM Calculating ApproachesHMM Calculating Approaches

Neural ComponentsNeural Components

Three Basic HMM ProblemsThree Basic HMM Problems

Viterbi AlgorithmViterbi Algorithm

State Duration ModelingState Duration Modeling

Training In HMMTraining In HMM

33

Recognition TasksRecognition Tasks

Isolated Word Recognition (IWR)Isolated Word Recognition (IWR)

Connected Word (CW) , And Continuous Connected Word (CW) , And Continuous Speech Recognition (CSR)Speech Recognition (CSR)

Speaker Dependent, Multiple Speaker, And Speaker Dependent, Multiple Speaker, And Speaker Independent Speaker Independent

Vocabulary SizeVocabulary Size– Small <20Small <20– Medium >100 , <1000Medium >100 , <1000– Large >1000, <10000Large >1000, <10000– Very Large >10000Very Large >10000

44

Speech Recognition ConceptsSpeech Recognition Concepts

NLPSpeech

Processing

Text Speech

NLPSpeech

ProcessingSpeech

Understanding

Speech Synthesis

TextPhone Sequence

Speech Recognition

Speech recognition is inverse of Speech Synthesis

55

Speech Recognition Speech Recognition ApproachesApproaches

Bottom-Up ApproachBottom-Up Approach

Top-Down ApproachTop-Down Approach

Blackboard ApproachBlackboard Approach

66

Bottom-Up ApproachBottom-Up Approach

Signal Processing

Feature Extraction

Segmentation

Signal Processing

Feature Extraction

Segmentation

Segmentation

Sound Classification Rules

Phonotactic Rules

Lexical Access

Language Model

Voiced/Unvoiced/Silence

Kno

wle

dge

Sou

rces

Recognized Utterance

77

UnitMatching

System

Top-Down ApproachTop-Down Approach

FeatureAnalysis

LexicalHypothesis

SyntacticHypothesis

SemanticHypothesis

UtteranceVerifier/Matcher

Inventory of speech

recognition units

Word Dictionary Grammar

TaskModel

Recognized Utterance

88

Blackboard ApproachBlackboard Approach

EnvironmentalProcesses

Acoustic Processes Lexical

Processes

SyntacticProcesses

SemanticProcesses

Blackboard

99

Recognition TheoriesRecognition Theories

Articulatory Based RecognitionArticulatory Based Recognition– Use from Articulatory system for recognitionUse from Articulatory system for recognition– This theory is the most successful until nowThis theory is the most successful until now

Auditory Based RecognitionAuditory Based Recognition– Use from Auditory system for recognitionUse from Auditory system for recognition

Hybrid Based RecognitionHybrid Based Recognition– Is a hybrid from the above theoriesIs a hybrid from the above theories

Motor TheoryMotor Theory– Model the intended gesture of speakerModel the intended gesture of speaker

1010

Recognition ProblemRecognition Problem

We have the sequence of acoustic We have the sequence of acoustic symbols and we want to find the words symbols and we want to find the words that expressed by speakerthat expressed by speaker

Solution : Finding the most probable of Solution : Finding the most probable of word sequence by having Acoustic word sequence by having Acoustic symbolssymbols

1111

Recognition ProblemRecognition Problem

A : Acoustic SymbolsA : Acoustic Symbols

W : Word SequenceW : Word Sequence

we should find so that we should find so that W)|(max)|ˆ( AWPAWP

W

1212

Bayse RuleBayse Rule

),()()|( yxPyPyxP

)(

)()|()|(

yP

xPxyPyxP

)(

)()|()|(

AP

WPWAPAWP

1313

Bayse Rule (Cont’d)Bayse Rule (Cont’d)

)(

)()|(max

AP

WPWAPW

)|(max)|ˆ( AWPAWPW

)()|(max

)|(maxˆ

WPWAPArg

AWPArgW

W

W

1414

Simple Language ModelSimple Language Model

nwwwww 321

),...,,,(

),...,,|(

).....,,|(

),|()|()(

)|()(

121

121

1234

123121

1211

WWWWP

WWWWP

WWWWP

WWWPWWPWP

wwwwPwP

nnn

nnn

iii

n

i

Computing this probability is very difficult and we need a very big database. So we use from Trigram and Bigram models.

1515

Simple Language Model Simple Language Model (Cont’d)(Cont’d)

)|()( 211

iii

n

iwwwPwP

)|()( 11

ii

n

iwwPwP

Trigram :

Bigram :

)()(1

i

n

iwPwP

Monogram :

1616

Simple Language Model Simple Language Model (Cont’d)(Cont’d)

)|( 123 wwwP

Computing Method :Number of happening W3 after W1W2

Total number of happening W1W2

AdHoc Method :

)()|()|()|( 332321231123 wfwwfwwwfwwwP

1717

Error Production FactorError Production Factor

Prosody (Recognition should be Prosody (Recognition should be Prosody Independent)Prosody Independent)

Noise (Noise should be prevented)Noise (Noise should be prevented)

Spontaneous SpeechSpontaneous Speech

1818

P(A|W) Computing P(A|W) Computing ApproachesApproaches

Dynamic Time Warping (DTW)Dynamic Time Warping (DTW)

Hidden Markov Model (HMM)Hidden Markov Model (HMM)

Artificial Neural Network (ANN)Artificial Neural Network (ANN)

Hybrid SystemsHybrid Systems

1919

Dynamic Time Warping Dynamic Time Warping Method (DTW)Method (DTW)

To obtain a global distance between two speech patterns a time alignment must be performed

Ex :A time alignment path between a template pattern “SPEECH” and a noisy input “SsPEEhH”

2020

Artificial Neural NetworkArtificial Neural Network

...

1x

0x

1w0w

1Nw

1Nx

y)(

1

0

i

N

iixwy

Simple Computation Element of a Neural Network

2121

Artificial Neural Network Artificial Neural Network (Cont’d)(Cont’d)

Neural Network TypesNeural Network Types– PerceptronPerceptron– Time DelayTime Delay– Time Delay Neural Network Computational Time Delay Neural Network Computational

Element (TDNN)Element (TDNN)

2222

Artificial Neural Network Artificial Neural Network (Cont’d)(Cont’d)

. . .

. . .

0x

0y 1My

1Nx

Single Layer Perceptron

2323

Artificial Neural Network Artificial Neural Network (Cont’d)(Cont’d)

. . .

. . .

Three Layer Perceptron

. . .

. . .

2424

Hybrid MethodsHybrid Methods

Hybrid Neural Network and Matched Filter For Hybrid Neural Network and Matched Filter For RecognitionRecognition

PATTERN

CLASSIFIER

SpeechAcoustic Features Delays

Output Units

2525

Neural Network PropertiesNeural Network Properties

The system is simple, But too much The system is simple, But too much iteration is needed for trainingiteration is needed for training

Doesn’t determine a specific structureDoesn’t determine a specific structure

Regardless of simplicity, the results are Regardless of simplicity, the results are goodgood

Training size is large, so training should be Training size is large, so training should be offlineoffline

Accuracy is relatively goodAccuracy is relatively good

2626

Hidden Markov ModelHidden Markov Model

Observation : O1,O2, . . . Observation : O1,O2, . . .

States in time : q1, q2, . . .States in time : q1, q2, . . .

All states : s1, s2, . . ., sNAll states : s1, s2, . . ., sN

tOOOO ,,,, 321

tqqqq ,,,, 321

Si Sjjiaija

2727

Hidden Markov Model (Cont’d)Hidden Markov Model (Cont’d)

Discrete Markov ModelDiscrete Markov Model

)|(

),,,|(

1

121

itjt

zktitjt

sqsqP

sqsqsqsqP

Degree 1 Markov Model

2828

Hidden Markov Model (Cont’d)Hidden Markov Model (Cont’d)

)|( 1 itjtij sqsqPa

ija : Transition Probability from Si to Sj ,

Nji ,1

2929

Discrete Markov Model Discrete Markov Model ExampleExample

S1 : The weather is rainyS2 : The weather is cloudyS3 : The weather is sunny

8.01.01.0

2.06.02.0

3.03.04.0

}{ ijaA

rainy cloudy sunnyrainy

cloudy

sunny

3030

Hidden Markov Model Example Hidden Markov Model Example (Cont’d)(Cont’d)

Question 1:How much is this probability:Sunny-Sunny-Sunny-Rainy-Rainy-Sunny-Cloudy-Cloudy

22311333 ssssssss

22321311313333 aaaaaaa

87654321 qqqqqqqq410536.1

3131

Hidden Markov Model Example Hidden Markov Model Example (Cont’d)(Cont’d)

Question 2:The probability of staying in state Si for d days if we are in state Si?

NisqP ii 1),( 1The probability of being in state i in time t=1

)()1()( 1 dPaassssP iiidiiijiii

d Days

3232

Discrete Density HMM Discrete Density HMM ComponentsComponents

N : Number Of StatesN : Number Of States

M : Number Of OutputsM : Number Of Outputs

A (NxN) : State Transition Probability A (NxN) : State Transition Probability MatrixMatrix

B (NxM): Output Occurrence Probability in B (NxM): Output Occurrence Probability in each stateeach state

(1xN): Initial State Probability(1xN): Initial State Probability),,( BA : Set of HMM Parameters

3333

Three Basic HMM ProblemsThree Basic HMM Problems

Given an HMM Given an HMM and a sequence of and a sequence of observations observations O,O,what is the probability what is the probability ? ?

Given a model and a sequence of Given a model and a sequence of observations observations OO, what is the most likely , what is the most likely state sequence in the model that produced state sequence in the model that produced the observations?the observations?

Given a model Given a model and a sequence of and a sequence of observationsobservations O, O, how should we adjust how should we adjust model parameters in order to maximize model parameters in order to maximize ? ?

)|( OP

)|( OP

3434

First Problem SolutionFirst Problem Solution

)(),|(),|(11

tq

T

ttt

T

tobqoPqoP

t

TT qqqqqqq aaaqP132211

)|(

)()|(),( yPyxPyxP

)|(),|()|,( zyPzyxPzyxP We Know That:

And

3535

First Problem Solution (Cont’d)First Problem Solution (Cont’d)

)|(),|()|,( qPqoPqoP

)()()(

)|,(

122111 21 Tqqqqqqqq obaobaob

qoP

TTT

T

TTTqqq

Tqqqqqqqq

q

obaobaob

qoPoP

21

122111)()()(

)|,()|(

21

Computation Order : )2( TTNO

3636

Forward Backward ApproachForward Backward Approach

)|,,,,()( 21 iqoooPi ttt

Niobi ii 1),()( 11

Computing )(it

1) Initialization

3737

Forward Backward Approach Forward Backward Approach (Cont’d)(Cont’d)

NjTt

obaij tjij

N

itt

1,11

)(])([)( 11

1 2) Induction :

3) Termination :

N

iT ioP

1

)()|(

Computation Order : )( 2TNO

3838

Backward VariableBackward Variable

),|,,,()( 21 iqoooPi tTttt

NiiT 1,1)(1) Initialization

2)Induction

NiAndTTt

jobaiN

jttjijt

11,,2,1

)()()(1

11

3939

Second Problem SolutionSecond Problem Solution

Finding the most likely state sequenceFinding the most likely state sequence

N

itt

ttN

it

t

ttt

ii

ii

iqoP

iqoP

oP

iqoPoiqPi

11

)()(

)()(

)|,(

)|,(

)|(

)|,(),|()(

Individually most likely state :Ttiq t

it 1)],([maxarg*

4040

Viterbi AlgorithmViterbi Algorithm

Define : Define :

Ni

oooiqqqqP

i

tttqqq

t

t

1

]|,,,,,,,,[max

)(

21121,,, 121

P is the most likely state sequence with this conditions : state i , time t and observation o

4141

Viterbi Algorithm (Cont’d)Viterbi Algorithm (Cont’d)

)(].)(max[)( 11 tjijti

t obaij

1) Initialization

0)(

1),()(

1

11

i

Niobi ii

)(it Is the most likely state before state i at time t-1

4242

Viterbi Algorithm (Cont’d)Viterbi Algorithm (Cont’d)

NjTt

aij

obaij

ijtNi

t

tjijtNi

t

1,2

])([maxarg)(

)(])([max)(

11

11

2) Recursion

4343

Viterbi Algorithm (Cont’d)Viterbi Algorithm (Cont’d)

)]([maxarg

)]([max

1

*

1

*

iq

ip

TNi

T

TNi

3) Termination:

4)Backtracking:

1,,2,1),( *11

* TTtqq ttt

4444

Third Problem SolutionThird Problem Solution

Parameters Estimation using Baum-Parameters Estimation using Baum-Welch Or Expectation Maximization Welch Or Expectation Maximization (EM) Approach(EM) Approach

Define:

N

i

N

jttjijt

ttjijt

tt

ttt

jobai

jobai

oP

jqiqoP

ojqiqPji

1 111

11

1

1

)()()(

)()()(

)|(

)|,,(

),|,(),(

4545

Third Problem Solution Third Problem Solution (Cont’d)(Cont’d)

N

jtt jii

1

),()(

1

1

)(T

tt i

T

tt ji

1

),(

: Expectation value of the number of jumps from state i

: Expectation value of the number of jumps from state i to state j

4646

Third Problem Solution Third Problem Solution (Cont’d)(Cont’d)

)(1 ii

T

tt

T

tt

ij

i

jia

1

1

)(

),(

T

tt

Vo

T

tt

j

j

j

kb kt

1

1

)(

)(

)(

4747

Baum Auxiliary FunctionBaum Auxiliary Function

q

qoPqoPQ )|,(log)'|,()|( '

)'|()|(

)|()|(: '

oPoP

QQif

By this approach we will reach to a local optimum

4848

Restrictions Of Restrictions Of Reestimation FormulasReestimation Formulas

11

N

ii

NiaN

jij

1,11

NjkbM

kj

1,1)(1

4949

Continuous Observation Continuous Observation DensityDensity

We have amounts of a PDF instead of We have amounts of a PDF instead of

We haveWe have

)|()( jqVoPkb tktj

1)(,),,()(1

dooboCob j

M

kjkjkjkj

Mixture Coefficients

Average Variance

5050

Continuous Observation Continuous Observation DensityDensity

Mixture in HMMMixture in HMM

),,()( jkjkjkk

j oCMaxob

M2|1M1|1

M4|1M3|1

M2|3M1|3

M4|3M3|3

M2|2M1|2

M4|2M3|2

S1 S2 S3Dominant Mixture:

5151

Continuous Observation Continuous Observation Density (Cont’d)Density (Cont’d)

Model Parameters:Model Parameters:

),,,,( CA

N×N N×M×K×KN×M×KN×M1×N

N : Number Of StatesM : Number Of Mixtures In Each StateK : Dimension Of Observation Vector

5252

Continuous Observation Continuous Observation Density (Cont’d)Density (Cont’d)

T

t

M

kt

T

tt

jk

kj

kjC

1 1

1

),(

),(

T

tt

t

T

tt

jk

kj

okj

1

1

),(

),(

5353

Continuous Observation Continuous Observation Density (Cont’d)Density (Cont’d)

T

tt

jktjkt

T

tt

jk

kj

ookj

1

1

),(

)()(),(

),( kjt Probability of event j’th state and k’th mixture at time t

5454

State Duration ModelingState Duration Modeling

Si Sj

Probability of staying d times in state i :

)1()( 1ii

diii aadP

jia

ija

5555

State Duration Modeling State Duration Modeling (Cont’d)(Cont’d)

Si Sjjia

……. …….

HMM With clear duration

ija )(dPj)(dPi

5656

State Duration Modeling State Duration Modeling (Cont’d)(Cont’d)

HMM consideration with State Duration :HMM consideration with State Duration :– Selecting using ‘sSelecting using ‘s– Selecting usingSelecting using– Selecting Observation Sequence Selecting Observation Sequence

using using in practice we assume the following in practice we assume the following

independence:independence:

– Selecting next state using transition probabilities Selecting next state using transition probabilities . We also have an additional constraint: . We also have an additional constraint:

),(),,,(1

1

11 121 tq

d

tdq OtbOOOb

iiq 1

dOOO ,,, 21 )(

1dPq1d

21qqa

),,,(11 21 dq OOOb

jq 2

011qqa

5757

Training In HMMTraining In HMM

Maximum Likelihood (ML)Maximum Likelihood (ML)

Maximum Mutual Information (MMI)Maximum Mutual Information (MMI)

Minimum Discrimination Information (MDI)Minimum Discrimination Information (MDI)

5858

Training In HMMTraining In HMM

Maximum Likelihood (ML)Maximum Likelihood (ML)

)|( 1oP

)|( 2oP)|( 3oP

)|( noP

.

.

.

)]|([*V

rOPMaximumP

ObservationSequence

5959

Training In HMM (Cont’d)Training In HMM (Cont’d)

Maximum Mutual Information (MMI)Maximum Mutual Information (MMI)

)()(

)|,(log),(

POP

OPOI

v

ww

v

wPwOP

OPOI

1

)(),|(log

)|(log),(

Mutual Information

}{, v

6060

Training In HMM (Cont’d)Training In HMM (Cont’d)

Minimum Discrimination Information Minimum Discrimination Information (MDI)(MDI)

dooP

oqoqPQI )|(

)(log)():(

),,,( 21 TOOOO

),,,( 21 tRRRR

Observation :

Auto correlation :

):(inf),( PQIPR )(RQ