Upload
clifton-clark
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
Speech-to-Speech Translation with
ClarificationsJulia Hirschberg, Svetlana Stoyanchev
Columbia UniversitySeptember 18, 2013
Outline Main Problem
Key Ideas
Solution Details
Impact
Issues, Gaps, and Future work
Speech Translation Speech-to-Speech translation system
3
L1 Speaker
lation
SpeechQuestion(L1)
Translated Question (L2)
Answer (L2)
Translated Answer (L1)
L2 Speaker
TranslationSystem
Speech Translation Translation may be impaired by:
Speech recognition errors
Word Error rate in English side of Transtac is 9%
Word error rate in Let’s Go bus information is 50%
A speaker may use ambiguous language
A speech recognition error may be caused by use of out-of-vocabulary words
4
TranslationSystem
Speech Translation Speech-to-Speech translation system
Introduce a clarification component
5
L1 Speaker
SpeechQuestion(L1)
Translated Question (L2)
Answer (L2)
Translated Answer (L1))
Clarification sub-dialogue
Clarification sub-dialogue
L2 SpeakerDialogue M
anager
Dialogue M
anager
Key Ideas Use targeted clarifications
Address challenges with targeted clarifications
Data collection for system evaluation
Most Common Clarification Strategies in Dialogue Systems
“Please repeat”
“Please rephrase”
System repeats the previous question
7
What Clarification Questions Do Human Speakers Ask?
Targeted reprise questions (M. Purver) o Ask a targeted question about the part of an utterance that was
misheard or misunderstood, including understood portions of the utterance
o Speaker: Do you have anything other than these XXX plans?
o Non-Reprise: What did you say?/Please repeat.
o Reprise: What kind of plans?
88% of human clarification questions are reprise
12% non-reprise
• Goal: Introduce targeted (reprise) questions into a spoken system
8
Advantages of Targeted Clarifications
More natural
User does not have to repeat the whole utterance/command
Provides grounding and implicit confirmation Speech-to-speech translation
Useful in systems that handle natural language user responses/commands/queries and a wide range of topics and vocabulary
Tutoring system
Virtual assistants (in car, in home): a user command may contain ASR error due to noise, background speech, etc.
9
Types of Clarification Questions in the TBOLT
System Rephrase part
• Used when an error is OOV and NOT a name (works on difficult non-OOV words as well)
• Asks to rephrase the error segment
• “I did not understand when you said: fiscal. Please give me another word or phrase for it.”
Spelling• Used for names
• “Please spell ‘Rockefeller’.”
Disambiguation
• Used to disambiguate between homophones
• “Did you mean plain as in extensive tract of level open land, or, plane as in an aircraft?”
Types of Questions (cont.)• Reprise (as found in human-human communication)
o Repeats part of the utterance before the error segment
o User: We will search some of the XXX to make sure everyone is safe.
o System: We will search some of the what?
• Reprise/ Rephrase-part
o Combines a targeted question with a rephrase question
o System: We will search some of the what? Please say another word or phrase for this: ‘vehicles’.
• Confirmation
o A yes/no question to confirm an utterance
o “Did you say ‘the breach is located here’?”
Requirement for a Targeted Question
Error Detection Error segment boundaries Error type
Does the error contain a proper name? Does the error contain an out-of-
vocabulary (OOV) word?
Challenges with Using Targeted Clarifications
1)A system's clarification question may not be appropriate because
An error segment may not be detected correctly An error type is not identified correctly
2)How should a system construct a targeted/reprise clarification question
3)How long should the system persist in its clarification?
Our goal is to address these challenges empirically
13
Examples of Inappropriate Clarification Questions
Rephrase-part (incorrect error type detected)• User: Captain Bell wants to ask you some questions.
• Error Detection: Captain <OOV> to ask you some questions.
• System: I did not understand when you said "Bell wants". Please give me another word or phrase for it.
Reprise/ Targeted Rephrase (error segment is identified incorrectly)• User: The insurgents are denouncing us.
• ASR/Error detection: The insurgents are doing now and <ERROR>
• System: The insurgents are doing now and what? Please say another word or phrase for "cing us".
Examples (cont.)• Spelling (error type is incorrectly detected as a
name)• User: It sounds like there is a kernel of truth to your
story. • Error Detection: It sounds like there is a <NAME>
of truth to your story. • System: I think I heard a name when you said this:
"kernel". Please spell that name.
Examples (cont.)• Disambiguation (disambiguated word is
misunderstood)• User: How often do you have problems with generators?
• ASR/Error Detection: How often do you have problems with general?
• System: Did you mean general as in broad or general as in a military officer?
Causes of Inappropriate Questions
Rephrase part Partial word is detected as an error Detected segment contains a name Detected segment is a function word (to, from, the …)
Disambiguate Neither choice for disambiguation is correct Disambiguated word is misrecognized
Spell Non-name Long segment
RepriseContains undetected recognition error
Goal
• Develop a method to automatically identify when an inappropriate question is asked
Use user’s answers to detect if a question was inappropriate
Data Collection• Simulation clarification system
• Users were asked to read a sentence and then were played a pre-recorded question
• Led to believe they were interacting with the actual system
Data Collection(cont.)• Prepared 228 questions
84 appropriate 144 inappropriate For each type of clarification questions, create
appropriate and inappropriate questions, Total 19 categories of clarification questions
• Each subject was asked 144 questions
• Recorded their initial utterances and their answers to the questions
User Responses• Subjects tended to be cooperative
• Answers varied from subject to subject
• Example: “I did not understand when you said: ‘Betirma’. Please give me another word or phrase for it.”
o “No"
o "Betirma"
o “Betirma bravo echo tango india romeo mike alpha"
User Responses (cont.)• Example 2:
User: “How often do you have problems with generators?”
System: “Did you mean general as in broad or general as in a military officer?”
o "generator as in a machine for making electricity"
o "no"
o "generators"
Method Extract lexical and prosodic features from
responses
Number of pauses, speech energy, speech tempo
Lexical and prosodic difference between initial response and an answer to clarification
Measure number of times subjects replay each question
Measure latency: length of pause before answer
• Determine whether questions are appropriate or inappropriate based on user responses
Challenge 2: Constructing Targeted Clarification Questions
Previous work: collected clarification questions using mturk (Stoyanchev et al. 2012, 2013)
Using human-generated questions manually created a set of generation rules
Evaluated generated questions with human subjects
Types of Questions R_GEN Generic: <context before error> what?
Applies if no other rules apply Sentence: The doctor will most likely prescribe XXX Question: the doctor will most likely prescribe WHAT?
R_SYN Syntactic: about <context before error> what about <context after error> ? Applies when: there is VB after error; VB and error share a parent Sentence: When was the XXX contacted? Question: When was WHAT contacted?
R_NMOD: which <parent word>? Applies when: DEP TAG error = NMOD and parent POS = NN | NNS Sentence: Do you have anything other than these XXX plans Question: Which plans?
R_START: what about <context after error>
Evaluation Questionnaire• Generated questions automatically using the rules
for a set of 84 sentences• Asked humans (mturk) to create a clarification
questions for the same sentences• Questionnaire applied to both human and computer-
generated questions
SubjectsMturk
Recruited 6 subjects from the lab
Inter-annotator Agreement
Results
Results
DiscussionR_GEN and R_SYN performance is comparable to
human-generated questions
R_NMOD (which …?) outperforms all other question types including human-generated questions
R_START rule did not work
Key Ideas Use Targeted Clarifications
Address challenges with targeted clarifications
Experiment on automatic detection of inappropriate questions
Experiment on automatic detection of when to terminate clarification
Data collection for system evaluation
Image Description and Questioning
Speaker1: A car is burning behind the girl The girl looks startled There was a massive explosion
Speaker2: A woman is standing in front of a
burning car Everything around her seems to
have been destroyed What caused this destruction?
Show user an image and ask to describe it and construct questions
Data Collection for System Evaluation
• Advantages: Do not prime users with words in a verbally
described scenario Elicits natural speech compared to reading Can be extended to a 2-way dialogue where the
interviewee is given a narrative or video information for answering interviewer's questions.
• Disadvantages: Uncontrolled vocabulary (can not force to
mispronounce words) No control across subject pairs
ImpactImpact on Speech-to-Speech Translation
Detecting when a targeted clarification question was inappropriate is an important feature for determining next dialogue move in clarification
Impact beyond Speech-to-Speech TranslationTargeted clarifications can be used in spoken
dialogue systemsEspecially useful for non-slot-filling (tutoring,
virtual assistants)
Future Work Appropriate and inappropriate questions
Analyze the data collected in responses to appropriate and inappropriate clarification questions
Use machine learning to predict if an utterance is an answer to appropriate or inappropriate clarification question
Targeted (reprise) clarification questions Which information from an initial sentence should a reprise clarification
question contain?
Using human-constructed questions, determine which information is essential to be repeated in a targeted question
Clarification length
How long should the system focus on a targeted clarification before back off?
Collect data and use machine learning to predict on each system’s turn whether a clarification should continue or stops
Conclusions• Used an error-simulation system to collect data
Data collection experiment for automatic detection of answers to 'inappropriate' system clarifications
Evaluation of automatically generated reprise clarification questions shows that they could be used in a system
Proposed an experiment for determining an optimal length of targeted clarification
• Collected audio data for system evaluation using an image description method
36
Thank youQuestions?
37
Challenge 3: Clarification Length
How long should the system focus on a targeted clarification before back off?
In a Speech-to-Speech translation: back-off= translate
In spoken dialogue systems : back-off = ask a generic question to 'please rephrase'.
• The answer depends on how patient and cooperative are users.
Evaluation of Clarification Length
• BOLT 2012 system behaviour: System asks targeted clarification at most 3 times before translating.
• Goal: Determine dynamically at each clarification turn whether the system should terminate clarification process.
• Use data to learn the dialogue strategy
Experiment Design• Simulate sequence of unsuccessful clarification questions.
• Give user an option to hit “translate” button
• Distractor cases: Simulate successful clarification
User: This computer is not operational
System: Please rephrase “not operational”
User: not working
System: thank you ( translate and show next question)
• Experimental case:
Loop asking 3 – 5 different targeted questions
Clarification dialogue continues until the user hits “translate”
• Use a combination of distractor and experimental cases
Method• Use data to determine when system should give
up on a targeted clarification
Apply machine learning Features:
Dialogue length (more likely to give up as dialogue continues to fail)
Question type Appropriateness of a clarification question Confidences of error detection and
classification components