Upload
vuthuy
View
227
Download
0
Embed Size (px)
Citation preview
Louis-Philippe MorencyCarnegie Mellon University
Social and EmotionalIntelligence in AI and Agents(Modeling Human Communication Dynamics)
Social Intelligent Agents
Customer Service
News reporter
Project manager
Teacher
Co-writer
Confident
Natural Computer Interaction
Customer service
News reporter
Project manager
Teacher
Co-writer
Confident
▪ Rapport
▪ Empathy
▪ Persuasion
Social
Cognitive▪ Attention
▪ Distraction
▪ Engagement
Emotion▪ Content
▪ Surprise
▪ Frustration
Human Multimodal Behaviors
▪ Gestures▪ Head gestures▪ Eye gestures▪ Arm gestures
▪ Body language▪ Body posture▪ Proxemics
▪ Eye contact▪ Head gaze▪ Eye gaze
▪ Facial expressions▪ FACS action units▪ Smile, frowning
Verbal Visual
Vocal
▪ Lexicon▪ Words
▪ Syntax▪ Part-of-speech▪ Dependencies
▪ Pragmatics▪ Discourse acts
▪ Prosody▪ Intonation▪ Voice quality
▪ Vocal expressions▪ Laughter, moans
Behavioral Multimodal Interpersonal Societal
A Central Challenge:
Modeling Human Communication Dynamics
• Vocal• Visual• Verbal
50 shades of “yeah”
Automatic Sensing for Intelligent Agents
OpenFace ToolkitFreely available for research
https://github.com/TadasBaltrusaitis/OpenFace
AI Technologies for Mental Health Assessment
ClinicianReport
Patient
MultiSense
SimSensei
OR
Clinician
Sensing User’s Mental Health Behavior Markers
DAIC
0.2
0.4
0.6
0.8
Patient Reference
Distress Not-distress2 weeks1 weekToday
Not-Depressed Depressed
Smile
Tense Voice
Open Posture
Emotional Expressiveness
Not-distressed Distressed
Distress Assessment
Interview Corpus
Depressed vs Non-depressed
Smile Dynamics - Behavior Indicators
Number of smiles
1
Smile duration
Smile intensity
S. Scherer, G. Stratou, J. Boberg, J. Gratch, A. Rizzo and L.-P. Morency. Automatic Behavior Descriptors for Psychological Disorder Analysis. IEEE Conference on Automatic Face and Gesture Recognition, 2013
PTSD vs Non-PTSD
Negative Expressions - Behavior Indicators
Overall population
2
Men only
Women only
G. Stratou, S. Scherer, J. Gratch and L.-P. Morency. Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences. International Conference on Affective Computing and Intelligent Interaction, 2013
Suicidal vs Non-suicidal
Speech Patterns - Behavior Indicators
First person pronouns(e.g., me, my, mine, I)
3
Voice tenseness
Repeater vs Non-repeater
V. Venek, S. Scherer, L.-P. Morency, A. Rizzo and J. Pestian, Adolescent Suicidal Risk Assessment in Clinician-Patient Interaction, IEEE Transactions on Affective Computing, January 2016
Unusual thoughts vs No symptom
Facial Expressivity - Behavior Indicators
With clinician
4
Alone in the room
Schizophrenia
S. Vijay, T. Baltrusaitis, L.-P. Morency, L. Pennant, D. Öngür and J. Baker, Automatic prediction of psychosis symptoms from facial expressions, CHI Computing and Mental Health Workshop, 2016
Modeling Interpersonal Dynamics
▪ Interlocutors adapt:
▪ Lexicon (gestural and verbal)
▪ Nonverbal Behavior (facial
expressions, posture)
▪ Prosody and speech
▪ High entrainment
signifies:▪ Understanding
▪ Flow of the conversation
▪ Cooperation
Interpersonal
Prediction of Immediate Negotiation Outcome
Dyadic Negotiation
Respondant’sBehaviors
Proposer’sBehaviors
JointPrediction
Model
Smile
Head Nod
Gaze
Self-touch
Smile
Head Nod
Gaze
Self-touch
History
Accept?Reject?
Predicting Listener Behaviors[IVA 2008, Best paper award]
listenerSpeaker
• Nonverbal
behaviors– Eye gaze
• Prosody
• Lexical
Prediction
Rapport Dataset
• 50 dyadic interactions
• Storytelling scenario
• Greedy forward selection
Best feature/encoding set
1. Pause
2. Eye gaze
3. “and”
4. Eye gaze
Virtual
Encodin
g
dic
tionary
• Backchannel
feedback(e.g. head nods
Latent Mixture of Discriminative Experts
Speaker
• Nonverbal
behaviors– Eye gaze
• Prosody
• Lexical
Encodin
g
dic
tionary
listener
• Backchannel
feedback(e.g. head nods
Virtual
y5y4y3y2y1
h5h4h3h2h1
x5x4x3x2x1
y5y4y3y2y1
x5x4x3x2x1
y5y4y3y2y1
x5x4x3x2x1
y5y4y3y2y1Prediction
Listeners
Discriminative experts
0
0.1
0.2
0.3
0.4
F1
me
as
ure
Rapport Dataset
Wisdom analysis
Syntax• Nouns
• Modifiers
Audio• Pauses
• Low pitch
Visual• Gaze
• Eye brows
Wisdom of
crowds
[ACL 2011, AAMAS 2010 – Best paper award]
Social Agents and Natural Computer Interaction
Customer service
News reporter
Project manager
Teacher
Co-writer
Confident
▪ Rapport
▪ Empathy
▪ Persuasion
Social
Cognitive▪ Attention
▪ Distraction
▪ Engagement
Emotion▪ Content
▪ Surprise
▪ Frustration
▪ Gestures▪ Head gestures
▪ Body language▪ Body posture
▪ Eye gaze
▪ Facial expressions▪ Smile, frowning
▪ Prosody▪ Voice quality
▪ Vocal expressions▪ Laughter, moans
Verbal
Visual
Vocal