Human/Computer CommunicationsHuman/Computer CommunicationsUsing SpeechUsing Speech

Ellis K. ‘Skip’ CaveEllis K. ‘Skip’ CaveInterVoice-Brite Inc.InterVoice-Brite Inc.

skip@intervoice.comskip@intervoice.com

Famous Human/Computer Communication - 1968

InterVoice-Brite

• Twenty years building speech applications

• Largest provider of VUI applications and systems in the world

• Turnkey Systems– Hardware, software, application design, managed services

• 1000’s of installations worldwide

• Banking, Travel, Stock Brokerage, Help Desk, etc.– Bank of America– American Express – E-Trade– Microsoft help-desk

Growth of Speech-Enabled Applications

• Analysts estimate that 15% of IVR ports sold in 2000 were speech enabled

• By 2004, 48.5% of IVR ports sold will be speech-enabled

– Source: Frost & Sullivan - U.S. IVR Systems Market, 2001

• IVB estimates that in 2002, 50% of IVB ports sold will be speech enabled.

Overview

• Brief History of Speech Recognition

• How ASR works

• Directed Dialog & Applications

• Standards & Trends

• Natural Language & Applications

History

• Natural Language Processing– Computational Linguistics

– Computer Science

– Text understanding

• Auto translation

• Question/Answer

• Web search

• Speech Recognition– Electrical Engineering

– Speech-to-text

• Dictation

• Control

Turing Test

• Alan M. Turing

• Paper -”Computing Machinery and Intelligence” (Mind, 1950 - Vol. 59, No. 236, pp. 433-460)

• First two sentences of the article:– I propose to consider the question, "Can machines

think?” This should begin with definitions of the meaning of the terms "machine" and "think."

• To answer this question, Turing proposed the “Imitation Game” later named the “Turing Test”– Requires an Interrogator & 2 subjects

Turing Test

Observer

Subject #1

Subject #2

Subject #2Which subject is a machine?

Turing Test

• Turing assumed communications would be written (typed)

• Assumed communications would be unrestricted as to subject

• Predicted that test would be “passed” in 50 years (2000)

• The ability to communicate is equated to “Thinking” and “intelligence”

Turing Test - 50 Years Later

• Today - NL systems still unable to fool interrogator on unrestricted subjects

• Speech Input & Output possible

• Transactional dialogs in restricted subject areas possible -

• Question/Answer queries feasible on large text databases

• May not fool the interrogator, but can provide useful functions– Travel Reservations, Stock Brokerages, Banking,

etc.

Speech Speech RecognitionRecognition

Voice Input - The New Paradigm

• Automatic Speech Recognition (ASR)

• Tremendous technical advances in the last few years

• From small to large vocabularies– 5,000 - 10,000 word vocabulary

• Stock brokerage - E-Trade - Ameritrade

• Travel - Travelocity, Delta Airlines

• From isolated word to connected words– Modern ASR recognizes connected words

• From speaker dependent to speaker independent– Modern ASR is fully speaker independent

• Natural Language

13 Parameters 13 Parameters 13 Parameters

Signal Processing Front-EndFeature

Extraction

Overlapping Sample Windows

25 ms Sample - 15ms overlap - 100 samples/sec.

Cepstrum

• Cepstrum is the inverse Fourier transform of the log spectrum

1,,1,0,)(log2

1)( LndeeSnc njj

Mel Cepstral Coefficients• Construct mel-frequency domain using a triangularly-

shaped weighting function applied to mel-transformed log-magnitude spectral samples:

Mel-Filtered Cepstral CoefficientsMost common feature set for recognizersMotivated by human auditory response characteristics

Mel Cepstrum

• After computing the DFT, and the log magnitude spectrum (to obtain the real cepstrum), we compute the filterbank outputs, and then use a discrete cosine transform to compute the mel-frequency cepstrum coefficients:

• Mel Cepstrum

– 39 Feature vectors representing on 25ms voice sample

Cepstrum as Vector Space Features

Feature Ambiguity

• After the signal processing front-end

• How to resolve overlap or ambiguity in Mel-Cepstrum features

• Need to use context information– What preceeds? What follows?

• N-phones and N-grams

• All probabalistic computations

The Speech Recognition Problem

Find the most likely word sequence Ŵ among all possible sequences given acoustic evidence A

A tractable reformulation of the problem is:

Language model

Acoustic model

Daunting search task

ASR Resolution

• Need – Mel Cepstrum features into probabilities

– Acoustic Model (tri-phone probabilities)

• Phonetic probabilities

– Language Model (bi-gram probabilities)

• Word probabilities

• Apply Dynamic Programming techniques– Find most-likely sequence of phonemes & words

– Viterbi Search

Acoustic Models

• Acoustic states represented by Hidden Markov Models (HMMs)

– Probabilistic State Machines - state sequence unknown, only feature vector outputs observed

– Each state has output symbol distribution

– Each state has transition probability distribution

s0 s1 s2

q(i|s0) q(i|s1) q(i|s2)

t(s0 |s0)

t(s1 |s0)

t(s1 |s1)

t(s2 |s1)

t(s2 |s2)

p(s0)

Subword Models

• Objective: Create a set of HMM’s representing the basic sounds (phones) of a language?

– English has about 40 distinct phonemes

– Need “lexicon” for pronunciations

– Letter to sound rules for unusual words

– Problem - co-articulation effects must be modeled

• “barter” vs “bartender”

• Solution - “tri-phones” - each phone modified by onset and trailing context phones

Language Models

• What is a language model?

– Quantitative ordering of the likelihood of word sequences

• Why use language models?

– Not all word sequences equally likely

– Search space optimization

– Improved accuracy

• Bridges the gap between acoustic ambiguities and ontology

Finite State Grammars

Allowable word sequences are explicitly specified using a structured syntax

• Creates a word network

• Words sequences not enabled do not exist!

• Application developer must construct grammar

• Excellent for directed dialog and closed prompting

• Narrow range of responses allowed

– Only word sequences coded in grammar are recognized

• Straightforward ASR engine. Follows grammar rules exactly

– Easy to add words to grammar

• Allows name lists

• “I want to fly to $CITY”

• “I want to buy $STOCK”

Finite-State Language Model

Statistical Language Models

Stochastic Context-Free Grammars

• Only specifies word transition probabilities

• N-gram language model

• Required for open ended prompts: “How may I direct your inquiry?”

• Much more difficult to analyze possible results

– Not for every interaction

• Data, Data, Data: 10,000+ transcribed responses for each input task

Statistical State Machines

Mixed Language Models

• SLM statistics are unstable (useless) unless examples of each word in each context are presented

• Consider a flight reservation tri-gram language model:I’d like to fly from Boston to Chicago on Monday

Training sentences required for 100 cities: (100*100 + 100*7) = 10,700

• A better way is to consider classes of words:I’d like to fly from $(CITY) to $(CITY) on $(DATE)

Only one transcription is needed to represent 70,000 variations

Viterbi

• How do you determine the most probable utterance?

• The Viterbi Search returns the n-best paths through the Acoustic model and the Language Model

Dynamic Programming (Viterbi)

N-Best Speech Results

• ASR converts speech to text• Use “grammar” to guide recognition • Focus on “speaker independent” ASRs• Must allow for open context

ASRSpeech

Waveform

Grammar

“Get me two movie tickets…”“I want to movie trips…”“My car’s too groovy”

N-Best Result

N=1N=2N=3

What does it all Mean?

Text output is nice, but how do we represent meaning ?

• Finite state grammars - constructs can be tagged with semantics

<item> get me the operator <tag>OPERATOR</tag> </item>

• SLM uses concept spottingItinerary:slm “flightinfo.pfsg” = FlightConcepts

FlightConcepts [

(from City:c) {<origin $c>}

(to City:c) {<dest $c>}

(on Date:d) {<date $d>}

]

• Concepts may also be trained statistically

– but that requires even more data!

Directed Directed DialogsDialogs

Directed Dialog

• Finite-State Grammars - Currently most common method to implement speech-enabled applications

• More flexible & user-friendly than key (Touch-Tone) input

• Allows Spoken List selection– System: “What City are you leaving from?”

– User: “Birmingham”

• Keywords easier to remember than numeric codes– “Account balance” instead of “two”

• Easy to skip ahead through menus– Tellme - “Sports, Basketball, Mavericks”

Issues With Directed Dialogue

• Computer asks all the questions– Usually presented as a menu

– “Do you want your account balance, cleared checks, or deposits?”

• Computer always has the initiative– User just answers questions, never gets to ask any

questions

• All possible answers must be pre-defined by the application developer (grammars)

• Will eventually get the job done, but can be tedious

• Still much better than Touch-tone menus

Issues With Directed Dialogue

• Application developer must design scripts that never have the machine ask open-ended questions– “What can I do for you?”

• Application Developer’s job - design questions where answers can be explicitly predicted.– “Do you want to buy or sell stocks”

• Developer must explicitly define all possible responses– Buy, purchase, get some, acquire

– Sell, dump, get rid of it

Examples of Directed Dialog

Southwest Airlines

Pizza Inn

Brokerage

Standards Standards & Trends& Trends

VoiceXML

• VoiceXML - A web-oriented voice-application programming language– W3C Standard - www.w3.org

– Version 1.0 released March 2000

– Version 2.0 ready to be approved

• http://www.w3.org/TR/voicexml20/

– Voice dialogues scripted using XML structures

• Other VoiceXML support – www.voicexml.org

– voicexmlreview.org

VoiceXML

• Assume telephone as user device

• Voice or key input

• Pre-recorded or Text-to-Speech output

Why VoiceXML?

• Provides environment similar to web for web developers to build speech applications

• Applications are distributed on document servers similar to web

• Leverages the investment companies have made in the development of a web presence.

• Data from Web databases can be used in the call automation system.

• Designed for distributed and/or hosted (ASP) environment.

VoiceXML Architecture

VoiceXMLBrowser/Gateway

Mobile Device

VUI

Telephone

Network

InternetVoiceXML

Browser

WebServer

WebServer

Serve VoiceXML

Document

Voice

VoiceXML Example

<?xml version="1.0"?><vxml version="1.0">

<!--Example 1 for VoiceXML Review --> <form> <block> Hello, World! </block> </form></vxml>

VoiceXML Applications

• Voice Portals– TellMe,

• 1-800-555-8355 (TELL)

• http://www.tellme.com

– BeVocal

• 1-408-850-2255 (BVOCAL)

• www.bevocal.com

The VoiceXML Plan

– Third party developers write VoiceXML scripts that they will publish on the web

– Callers to the Voice Portals will access these voice applications like browsing the web

– VoiceXML will use VUI with directed dialog

• Voice output

• Voice or key input

– hands/eyes free or privacy

Speech Application Language Tags (SALT)

• Microsoft, Cisco Systems, Comverse Inc., Intel, Philips Speech Processing, and SpeechWorks

• www.saltforum.org

• Extension of existing Web standards such as HTML, xHTML and XML

• Support multi-modal and telephone access to information, applications, and Web services, independently or concurrently.

SALT - “Multi-modal”

• Input might come from speech recognition, a keyboard or keypad, and/or a stylus or mouse

• Output to screen or speaker (speech)

• Embedded in HTML documents

• Will require SALT-enabled browsers

• Working Draft V1.9

• Public Release - March 2002

• Submit to IETF - midyear 2002

SALT Code

• <!—- Speech Application Language Tags -->

• <salt:prompt id="askOriginCity"> Where would you like to leave from? </salt:prompt>

• <salt:prompt id="askDestCity"> Where would you like to go to? </salt:prompt>

• <salt:prompt id="sayDidntUnderstand" onComplete="runAsk()">

• Sorry, I didn't understand. </salt:prompt>

• <salt:listen id="recoOriginCity"

• onReco="procOriginCity()” onNoReco="sayDidntUnderstand.Start()">

• <salt:grammar src="city.xml" />

• </salt:listen>

• <salt:listen id="recoDestCity"

• onReco="procDestCity()" onNoReco="sayDidntUnderstand.Start()">

• <salt:grammar src="city.xml" /> </salt:listen>

Evolution of the Speech Interface

• Touch-Tone Input

• Directed Dialogue

• Natural Language– Word spotting

– Phrase spotting

– Deep parsing

Natural Natural Language Language

UnderstandingUnderstanding

What is Natural Language • User can take the initiative

– Computer says “How can I help you?”

• User can state request in a single interaction– What is the price of IBM?

– “I want to sell all my IBM stock today at the market”

• User can change initiatives midstream– I want to buy some stock - How much do I have in my

account?

• Natural Language is closely related to Artificial Intelligence

• Feasible today, if scope of discussion is limited

Natural Language

• NLP, NLU, NLG – All of these mean very specific things to the

computational linguistics community

– NLP - Machine processing of human text or speech

– NLU - Machine understanding of unconstrained human-originated text (or speech)

– NLG - Machine generation of human-understandable text (or speech)

• To build a true NL dialog engine you must deal with NLU and NLG

Natural Language Engine

Natural Language Understanding

Natural LanguageGeneration

Database Manipulation

Natural Language Engine

Text in Text out

Spoken Natural Language Engine

Natural Language Understanding

Text in Text out

Speech in

Automatic SpeechRecognition

Text-to-Speech

Speech out

Text-to-Speech

• Modern Text-to Speech– More natural-sounding

– Better prosody

– Improved proper noun handling

• People Names

• Street Names

• Cities

– Abbreviation handling

• Dr. = Doctor or Drive

Overview of Dialog Systems

• Openness of dialog• Can ASR hear everything?

• Can NLP understand everything heard?

• Can DM cope with multiple strands / directions?

• Does Prompt Generator sound natural?

B/O

NL ParserVoice in

ASR

Voice out PromptGenerator

TTS

DialogManager

Dialog Sophistication

• NL Parser• Ontology + syntax

• Ontology

• Morphology Concept Spotting

• Word spotting

more

less

• Dialog Management• Complex Mixed Initiatives

• Dialog context

• Query Answering (Q & A)

• Directed dialog

less

more

Natural Language Technologies

• Improve Existing Applications– Scheduling - Airlines, Hotels

– Financial - Banks, Brokerages

• Enabling New Applications– Catalogue Order

– Complex Travel Planning

– Information Mining - Voice Web browsing

– Direct Telemarketing AI

• Many applications requires Text-to-Speech

Shallow-Parse NLU

• Nuance, Speechworks, Philips,…

• W3C Semantic Attachments for Speech Recognition Grammars (Working Draft 11 May 2001)

• Driven by an Interpretation Grammar.

– “Full” - Finite State Grammar (whole sentence must match)

– “Robust” Statistical Grammar (word or phrase-spotting - partial parses of phrase fragments)

– Associating semantic tags (ABNF { }, XML <tag/>) to the Recognition Grammar rules

– Semantic Attachments for Speech Recognition Grammars

• Constrained by the Recognition Grammar

– Need to write partial Recognition Grammar even when using SLM

• No interpretation in-context

Statistical Language Model

Phrase-

Spotting

Rules

Buy stock

Sell StockGet stock quote

Get operator

60%

20%

90%

N-Best Phrases

SLM Interpretive or Recognition Grammar

Probabilistic Word Transitions

ASRFront End

1- “I want to buy stock…”2- “I want to go by the lock…”3- “High doesn’t mock….”

SpeechWaveform

Shallow-Parse NLU

• Latest thing from ASR vendors– Phillips

• Speech Pearl - SLM

• Speech Mania - FSG

– Nuance

• Say Anything - SLM

• Requires two new constructs– Statistical Language Model

– Interpretation Grammar

– Allows more open responses

Shallow-Parse NLU

• Developer must define and code “interpretation grammar” that lists keywords and phrases that can occur– Need programming skills to define interpretive grammars.

More complex coding than fixed grammars

– wild cards, concept spotting, phrase matching

• SLM – Need large number of example utterances to get

reliable word-sequence statistics

– Usually requires recording and transcription of live conversations in application topic

– More difficult to add new words to grammar

• Must also provide usage and sequence statistics with each word added

– SLM outputs n-best utterance list

Shallow-Parse NLU

• Interpretive Grammars– Allows “wild card” descriptions: “(to $CITY)”

• “I want to fly to $CITY” - “I want to go to $CITY”

• “I need to fly to $CITY”

– Allows out-of sequence phrases

• I want to go from Chicago to Dallas today

• Today I want to go to Dallas from Chicago

Shallow-Parse NLU

• Allows more open dialog than Finite-State grammars

• Requires more development effort than Finite-State Grammars

• Will allow a smaller technology step than full NLU for less-demanding applications

Deep Parsing NLU

• Full linguistic analysis of Speech– Syntactic and Semantic analysis

• Core Language Model contains data structures required for language understanding– Lexicon

– Ontology

– Parsing Rules

• Eliminate – Scripting

– Manually-defined semantic tags

• developer doesn’t have to define concepts

Deep Parsing NLU

• Lexicon - A List of Words and their Syntactical and Semantic Attributes

• Root or stem word form

– fox, run, Boston

• Optional forms plural, tenses

– fox ,foxes

– run, ran, running

• part of speech

– fox - noun

– run - verb

– Boston - proper noun

• Link to Ontology

– fox - animal, brown, furry

– run - action, move fast

– Boston - city

Deep Parsing NLU

• Ontology - Classes & Relationships

PERSON LOCATION DATE TIME PRODUCT NUMERICAL MONEY ORGANIZATION MANNER VALUE

DEGREE DIMENSION RATE DURATION PERCENTAGE COUNT

time of day

midnightprime time

clock time

hockeyteam

team,squad

institution,establishment

financialinstitution

educationalinstitution

numerosity,multiplicity

integer,whole number

population denominatorthickness

width,breadth

distance,length

altitude wingspan

Deep Parsing NLU

Who was the first Russian astronaut to walk in space

WP VBD DT JJ NNP NN TO VB IN NN

NPNP

PPVP

VP

SVP

S

walk

space

space

walk

walk

astronaut

astronaut

astronaut

PERSON

astronaut

PERSON

first

walk

Russian

space

Parsing Rules

Natural Language Understanding

• NL Parser

– N-Best input in NLSML format

– Context free parsing based on Ontology, Lexicon and Rules

– Filtering out redundant interpretations

– Extended W3C NLSML output

• Dialog Server

– Extended W3C NLSML input

– In context interpretation

– Dialog Directives output

NL ParserVoice in ASRNL Dialog

ServerNLSML NLSML

KB

NL Interpretation is split between two components

Natural Language

• Stock Transaction

• 30 seconds

• Airline Reservation

• TTS

Human Language Technology Institute at UTD

• Opening March 2002 (open house March 7-8)

• Foster research in Human Language Technology

• Establish ties with local industry

• 6 new faculty positions

• Currently funded at about 1 million per year by state and government agencies

Speech Technology Professionals Forum

• Monthly meetings in Telecom Corridor– Third Tuesday - March 19

• Rick Tett– rick.tett@candora.com

– www.candora.com/stpf

Conclusions

• Consumer applications in Stock Brokerages, Travel Agencies, etc. are raising the standards for directed-dialogue production quality and usability

• VoiceXML and SALT will open VUI Directed Dialog application development

• Artificial Intelligence and Natural Language technology making rapid advances - enabling highly conversational applications

The Speech User Interface

• Speech will emerge as the preferred mobile interface

• Selectable voice or key input– Hands/eyes free or privacy

• Selectable voice or screen output– When small screens make sense

• Hands-free Wireless Web Access – WAP phones not required

– 3G phones not required

The Speech User Interface

• Natural Language will make VUI highly conversational– no menus

– no memorizing keywords or keystrokes

• Many so-called graphical applications can be made more efficient with a pure speech interface– Driving Maps, Bank Statements

The Speech User Interface

The computing terminal of tomorrow -

is already here!

Speech -

The optimal computing interface

Thank You!

References

• Speech & Language Processing– Jurafsky & Martin -Prentice Hall - 2000

• Statistical Methods for Speech Recognition – Jelinek - MIT Press - 1999

• Foundations of Statistical Natural Language Processing– Manning & Schutze - MIT Press - 1999

• Dr. J. Picone - Speech Website– www.isip.msstate.edu