1

2009 Almost-Spring Short Course on Speech Recognition

Instructors: Bhiksha Raj and Rita Singh

Welcome

2

What will the course be about

• We will cover most relevant topics of speech recognition

• The focus will be on the theory and practice– We will not discuss code for the most part– We will keep maths out of it as far as possible,

however

• We will discuss algorithms and implementation details

3

Instructors

• Bhiksha Raj: Carnegie Mellon University– Expert in speech recognition

• Rita Singh: Carnegie Mellon University– Expert in speech recognition

• Peter Wolf: Independent Consultant– Previously in Dragon Systems Inc.– Sphinx4 expert, expert in speech recogintion

application development– Brought in primarily as a resource for helping with

sphinx4 and answering applications related questions

4

Format of Course

• 3 Lectures daily– Morning: 8.00 AM, 1.00 – 1.30 ours– Late Morning / Early Afternoon: 11:00 AM– Afternoon: 2.30 PM

• The schedule is flexible – timings may vary depending on how much is covered

• Lectures expected to last 1.00 – 1.5 hours each

• Intervening times expected to be taken up by exercises

5

Instruction Format

• Lectures will be pictorially oriented

• Although we will cover general topics, the specific implementations described will be based on CMU Sphinx– Most other systems are similar

• Exercises will be based on sphinx

6

Lecture Outline: Day 1

• Lecture 1: “Speech recognition for dummies”– a quick development of speech recognition as string

matching

• Lecture 2: “Feature computation”– Explaining how features are computed for speech

recognition, including all signal processing

• Lecture 3: “Hidden Markov Models”– Describing HMMs and all associated problems

7

Lecture Outline: Day 2

• Lecture 1: “Training From Continuous Speech”– How to train models from continuous speech– Phonemes, why we need them and how to train them

• Lecture 2: “Context dependent phonemes”– What are context dependent phonemes– Various types of context dependent phonemes– Training CD phonemes

• Lecture 3: “Decision Trees and State Tying”– All about decision trees for parameter sharing in ASR systems

8

Lecture Outline: Day 3

• Lecture 1: “Training context-dependent models with tied states”– A (relatively) short lecture explaining the final overall process for

training models

• Lecture 2: “Language Modelling”– How to model “language” for speech recognition– Statistical language modelling

• Lecture 3: “Decoding: Basics”– Describing the basic ideas behind the decoding strategies for

continuous speech

9

Lecture Outline: Day 4

• Lecture 1: “Decoding: Advanced”– Explaining various more advanced approaches to decoding

• Arriving at the state of art

• Lecture 2: “Advanced Topics”– Adaptation, Normalization, Discriminative Training etc.

• Session 3: Open.– Any spillover– Question Answering

10

Exercises: Day 1

• There will be exercises following most lectures

• Lecture 1: None

• Lecture 2: Exercise on capture and feature computation from speech signals

• Lecture 3: None

11

Exercises: Day 2

• Lecture 1: “Training From Continuous Speech”– Exercise on training phoneme models and

recognizing with them

• Lecture 2: “Context dependent phonemes”– Exercise on training models for context-dependent

phonemes and recognizing with them

• Lecture 3: “Decision Trees and State Tying”– Exercise on learning decision trees

12

Exercises: Day 3

• Lecture 1: “Training context-dependent models with tied states”– Exercise on complete training of the ASR system

• Lecture 2: “Language Modelling”– Exercises on building JSGF grammars and Ngram

LMs for speech recognition

• Lecture 3: “Decoding: Basics”

13

Lecture Outline: Day 4

• Lecture 1: “Decoding: Advanced”– Decoding with various speech recognition system

variants:• Sphinx3 flat, Sphinx3 tree, Sphinx4

• Lecture 2: “Advanced Topics”– No exercises

• Session 3: Open.– No exercises

14

Software to Install

• We will be using the CMU sphinx extensively– Sphinxtrain– Sphinx3 decoder– Sphinx4 decoder– CMU LM Toolkit or SRI LM Toolkit

• We will need additional software to go with it– Java, ant, groovy for S4

15

Sphinx Downloads: http://cmusphinx.sourceforge.net

16

• Sphinxbase: – Click on the “sphinxbase” link on the left

– Click “all releases”

– Download version 0.4.1• http://downloads.sourceforge.net/cmusphinx/sphinxbase-0.4.1.tar.bz2?use_

mirror=superb-east

• Sphinx3: – Click on “sphinx3” link on left

– Click on “all releases”

– Download version 3-0.8• http://downloads.sourceforge.net/cmusphinx/sphinx3-0.8.zip?

use_mirror=internap

Sphinx Downloads: http://cmusphinx.sourceforge.net

17

• Cepview: – Click on the “cepview” link on the left

• lm3g2dmp: – Click on “lm3g2dmp” link on left

• The above two are visualization / data-structure optimization tools and are not critical– But they are small, so you might as well download them

• CMULM toolkit: You may install SRI LM toolkit instead– Better maintained – CMU toolkit is not currently maintained

Sphinx Downloads: http://cmusphinx.sourceforge.net

18

• Sphinx4: – For this workshop download a copy of sphinx that is under development

at github.com– http://github.com/juanzanos/sphinx4/tree/master

• Click on download link– Caveat: some scripts may not run; if so we will revert to release version

• Sphinx4 will also need– Java JDK 1.6 -- from http://javasoft.com– Apache ant -- from http://ant.apache.org– A useful scripting tool (some of our latest scripts are in it): Groovy– Groovy can be had from http://groovy.codehaus.org

• Bookmark this link:– http://cmusphinx.sourceforge.net/sphinx4/doc/

UsingSphinxTrainModels.html

Sphinx Downloads: http://cmusphinx.sourceforge.net

19

Operating Systems

• Sphinxbase and Sphinx3 packages have been tried and tested on linux– We are not windows people

• Suggestion: Prefer linux-based machines– You may also try to run these programs on cygwin under

windows• Sphinx* should compile under cygwin

• Install “tcsh” under cygwin

• We will provide tcsh scripts

• Sphinx4 is platform independent

20

Additional Packages

• Would be useful to have a visualization tool– Need to visualize matrices as surfaces

• Matlab would be great

• If you don’t have matlab, download octave– http://www.gnu.org/software/octave/

21

Data

• You may use any data you wish to

• For exercise we will attempt to provide a small amount of data– As much as can be dealt with on your

computers

22

Questions

• ?