Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Communications Engineering – Dialogue SystemsUlm University (Germany)dialogue‐systems.org
Dialogue Systems Research at Ulm University –Adaptive Speech Interfaces for Technical Companions TGMIS Istanbul | November 2014Wolfgang Minker, Maxim Sidorov and Stefan Ultes
TGMIS Istanbul | November 2014Page 2
Human‐Computer‐Interaction – State‐of‐the‐Art
Environment
User• Explicit functional requirements• Explicit system responses
Mental model
Environmental parameterEmotional parameter
Technical system
TGMIS Istanbul | November 2014Page 3
Human‐Computer‐Interaction – Beyond State‐of‐the‐Art
Environment
Technical system
User• Explicit functional requirements• Explicit system responses
Implicit functional requirements
Mental status
Environmental parameterEmotional parameter
TGMIS Istanbul | November 2014Page 4
Spoken Dialogue Systems – Towards Companions
Companionsystem
User
Mental model
Environmental parameterEmotional parameter
Know
ledge mod
elUser centered
functionality and assistance
User oriented Human‐Computer dialog
• Explicit functional requirements• Explicit system responses
• Implicit functional requirements• Implicit system responses
Capture andinterpretation of the situation
TGMIS Istanbul | November 2014Page 5
Companion Systems
• Contain beyond state‐of‐the art technical systems that are able to – autonomously perceive their environment– plan actions and pursue aims – carry out natural and unconstrained dialogues with users
Properties close to human interaction partners:– assistiveness– adaptiveness– proactiveness– individuality– availability– cooperativeness– Trustworthiness
A Companion‐Technology for Cognitive Technical Systems (2009‐16) Adaptive and TRusted Ambient eCOlogies (2008‐11) (EU‐FP7) A Knowledge‐Based Information Agent with Social Competence and Human Interaction Capabilities (2015‐18) (EU‐HORIZON2020)
TGMIS Istanbul | November 2014Page 6
Linguisticanalysis anddialogue
management
Text generation and speech synthesis
Acoustic front‐endSpeech recognition
User state and capability
management
Planning and task management
Devicemanagement
ProsodyImage data
Spoken Dialogue Systems – Towards Companions
• Assistiveness, adaptiveness and proactiveness
Enhance the linguistic analysis and dialogue management components
TGMIS Istanbul | November 2014Page 7
Current and Past PhD Theses
Speech Analysis Dialogue ManagementClassification and Optimization:• Automatic Categorization of Human‐Human
and Human‐Machine Conversation based on Hierarchical Classification
• Interaction Quality Modelling for Human‐Human Conversations
• Evolutionary Algorithms for Automated Classifier Design in Spoken Dialogue Systems
• Automatic Estimation of Verbal Intelligence
Assistiveness:• User‐ and situation‐adaptive explanations in dialogue systems• Domain‐Level Reasoning for Dialogue Systems
Adaptiveness:• Statistical Modeling for Online Monitoring of Adaptive Spoken
Dialog Systems• Statistical Modeling for User‐centered, Adaptive Spoken Dialog
Systems• Situation‐ and User‐Adaptive Dialogue Management• Model‐Driven Adaptation for Spoken Dialogues in Intelligent
Environments
Emotion Recognition:• Speech‐Emotion Recognition in Adaptive
Dialogue Systems• Emotion Recognition for Adaptive Spoken
Dialogue Systems
Adaptive Multimodal:• Interactive Anthropomorphic Interface Assistants• Intuitive Speech Interface Technology for Information Exchange
Tasks• Adapting Multimodal Interactive Systems to User Behaviour
Proactiveness:• Proactive Spoken Dialogue Interaction in Multi‐Party
Environments
TGMIS Istanbul | November 2014Page 8
Current and Past PhD Theses
Speech Analysis Dialogue ManagementClassification and Optimization:• Automatic Categorization of Human‐Human
and Human‐Machine Conversation based on Hierarchical Classification
• Interaction Quality Modelling for Human‐Human Conversations
• Evolutionary Algorithms for Automated Classifier Design in Spoken Dialogue Systems
• Automatic Estimation of Verbal Intelligence
Assistiveness:• User‐ and situation‐adaptive explanations in dialogue systems• Domain‐Level Reasoning for Dialogue Systems
Adaptiveness:• Statistical Modeling for Online Monitoring of Adaptive Spoken
Dialog Systems• User‐Adaptive Spoken Dialogue Management• Situation‐ and User‐Adaptive Dialogue Management• Model‐Driven Adaptation for Spoken Dialogues in Intelligent
Environments
Emotion Recognition:• Speech‐Emotion Recognition in Adaptive
Dialogue Systems• Emotion Recognition for Adaptive Spoken
Dialogue Systems
Adaptive Multimodality:• Interactive Anthropomorphic Interface Assistants• Intuitive Speech Interface Technology for Information Exchange
Tasks• Adapting Multimodal Interactive Systems to User Behaviour
Proactiveness:• Proactive Spoken Dialogue Interaction in Multi‐Party
Environments
TGMIS Istanbul | November 2014Page 9
Linguisticanalysis anddialogue
management
Text generation and speech synthesis
Acoustic front‐endSpeech recognition
User state
Planning and task management
Devicemanagement
User‐Adaptive Spoken Dialogue Management (Stefan Ultes)
• Provide appropriate system behavior based on perceived user state
User state recognition
AdaptiveDialogue
Management
TGMIS Istanbul | November 2014Page 10
Linguisticanalysis anddialogue
management
Text generation and speech synthesis
Acoustic front‐endSpeech recognition
User state
Planning and task management
Devicemanagement
User‐Adaptive Spoken Dialogue Management
• Provide appropriate system behavior based on perceived user state
User state recognition
AdaptiveDialogue
Management
Automatic user state recognition:• idea: usage of statistical classifiers
to recognize: intoxication emotions user Satisfaction perceived coherence
• focus on Interaction Quality (IQ) objective form of user
satisfaction analysis of multiple statistical
modeling approaches static models (SVM...) sequence models
(HMM...) evaluation of IQ in dialogues
TGMIS Istanbul | November 2014Page 11
• Provide appropriate system behavior based on perceived user state
Linguisticanalysis anddialogue
management
Text generation and speech synthesis
Acoustic front‐endSpeech recognition
User state
Planning and task management
Devicemanagement
User‐Adaptive Spoken Dialogue Management
User state recognition
AdaptiveDialogue
Management
Adaptive Dialogue Management (DM)• change system behavior / dialogue
strategy based on Interaction Quality• explicitly
rule‐based system adapt:
grounding initiative prompt design
• implicitly automatically learn best strategy
statistical DM (POMDP) reinforcement learning
IQ part of dialogue state IQ part of reward function
Experiments and results at my poster
TGMIS Istanbul | November 2014Page 12
Linguisticanalysis anddialogue
management
Text generation and speech synthesis
Acoustic front‐endSpeech recognition
Emotion recognizer
Planning and task management
Devicemanagement
Emotion Recognition for Adaptive SDS (Maxim Sidorov)
• Emotion Recognition is used to change a dialogue strategy or to redirect an end user to a human‐assistant
Speech‐ or multimodal‐based emotion recognition
Multimodal automatic emotion recognition:• machine learning algorithms for
modelling ANN SVM KNN
• usage of multimodal features audio visual text
• types of fusion feature‐based or early fusion decision‐based fusion
(weighted sum, product, meta‐modelling algorithm)
model‐based
TGMIS Istanbul | November 2014Page 13
Linguisticanalysis anddialogue
management
Text generation and speech synthesis
Acoustic front‐endSpeech recognition
Emotion recognizer
Planning and task management
Devicemanagement
Enhancement of Emotion Recognition
• Emotion Recognition is used to change a dialogue strategy or to redirect an end user to a human‐assistant
Speech‐ or multimodal‐based emotion recognition
• gender‐ or speaker‐adaptive emotion models system A: Independent models
for each speaker and gender system B: Incorporating
speaker or gender hypothesizes directly into feature vector
• multi‐objective genetic algorithm‐based feature selection to maximize emotion
recognition performance and minimize number of features
simultaneously
Experiments and results at my poster
TGMIS Istanbul | November 2014Page 14
Conclusions
• How to optimally adapt spoken language dialogue systems to user status and context of use? ( Adaptiveness)
– relevant context information captured and interpreted– information integrated into a user‐oriented human‐computer dialogue– adaptive dialogue modeling strategies
• How to reduce the cognitive burden of the user? ( Assistiveness and Proactiveness)
– more powerful back‐end and dialogue strategies– multi‐user interaction– dialogue history management
Enhanced spoken dialogue interaction plays a key role in advanced technical systems.
TGMIS Istanbul | November 2014Page 15
Acknowledgements