Affective User Modeling

Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..

[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..

Affective User Modeling

@MEi:CogSci

Marko Tkalčič

[email protected]

http://ldos.fe.uni-lj.si/markot

mailto:[email protected]





Overview

Traditional user modeling

in recommender systemsNeed for affective user

modeling!

Emotions & detection

HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



User modeling

Prediction of users behavior

Why?

– Product recommendation



Amazon



Netflix



Recommender systems



Recommender systems

DB

Recommender

System



Recommender systems

DB

Recommender

System

KnowledgeFeedback




In movie recommender systems – Netflix example

XXX [genre = A]

YYY [genre = B]

ZZZ [genre = C]

XYY [genre = B]

XXY [genre = C]

User profile:

A: 0

B: 0

C: 0





XXX [genre = A]

YYY [genre = B]

ZZZ [genre = C]

XYY [genre = B]

XXY [genre = C]

User profile:

A: 0

B: 0

C: 0

YYY





XXX [genre = A]

YYY [genre = B]

ZZZ [genre = C]

XYY [genre = B]

XXY [genre = C]

User profile:

A: 0

B: 5

C: 0

YYY R=5





YYY [genre = B]

XYY [genre = B]

ZZZ [genre = C]

XXX [genre = A]

XXY [genre = C]

User profile:

A: 0

B: 5

C: 0





YYY [genre = B]

XYY [genre = B]

ZZZ [genre = C]

XXX [genre = A]

XXY [genre = C]

User profile:

A: 0

B: 5

C: 0

XYY





YYY [genre = B]

XYY [genre = B]

ZZZ [genre = C]

XXX [genre = A]

XXY [genre = C]

User profile:

A: 0

B: 4

C: 0

XYY R=3





YYY [genre = B]

XYY [genre = B]

ZZZ [genre = C]

XXX [genre = A]

XXY [genre = C]

User profile:

A: 0

B: 4

C: 0





YYY [genre = B]

XYY [genre = B]

ZZZ [genre = C]

XXX [genre = A]

XXY [genre = C]

User profile:

A: 0

B: 4

C: 0

ZZZ





YYY [genre = B]

XYY [genre = B]

ZZZ [genre = C]

XXX [genre = A]

XXY [genre = C]

User profile:

A: 0

B: 4

C: 5

ZZZ R=5



Context-aware user modeling

Users have different preferences in different contexts

?????

User profile:

Context = alone

A: 0

B: 4

C: 5

User profile:

Context = friends

A: 5

B: 2

C: 3

User profile:

Context = children

A: 1

B: 5

C: 1



Context-aware user modeling

Users have different preferences in different contexts

ZZZ [genre = C]

XXY [genre = C]

YYY [genre = B]

XYY [genre = B]

XXX [genre = A]

User profile:

Context = alone

A: 0

B: 4

C: 5

User profile:

Context = friends

A: 5

B: 2

C: 3

User profile:

Context = children

A: 1

B: 5

C: 1

Context = alone



General user modeling framework

Data-centric = uses data that

– Is available (genres, actors, directors ...)

– Easy to acquire (rating, „liking“ ...)

But NOT necessarily data that carry information

USER MODEL

Controlled variables

Uncontrolled variables

Selected MM items

Prediction accuracy

Huge MM DB



General user modeling framework

Data-centric = uses data that

– Is available (genres, actors, directors ...)

– Easy to acquire (rating, „liking“ ...)

But NOT necessarily data that carry information

USER MODEL

Controlled variables


Prediction accuracy

?



Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



It is not so simple!

Bounded rationality theory [Daniel Kahnemann (nobel prize for

economics 2002)]

Decision making = rational + emotional



Need for affective user modeling!

(Tkalčič et al., 2010)

Affective + generic variables

>

Generic) variables

USER MODEL

Controlled variables = generic + affective variables




Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



Overview of emotions

Emotions are complex human experiences

Evolutionary based

Several definitions

We take with simple models, easy to incorporate in computers:

– Basic emotions

– Dimensional model

– Circumplex model



Basic emotions

Discrete classes model

Different sets

Darwin: Expression of emotions in man and animal

Ekman definition (6 + neutral):

– Happiness

– Anger

– Fear

– Sadness

– Disgust

– Surprise



Dimensional model

Three dimensions

– Valence

– Arousal

– Dominance

Each emotive state is a point in the VAD space



Circumplex model

Maps basic emotions dimensional modelArousal

Valence

high

negative positive

low

neutral

sadness

fear

disgust

surprise

joyanger



How to detect emotions?

Explicit vs. Implicit

Explicit

– Questionnaires (SAM)

Implicit:

– Work done in the affective computing community

– Different modalities (sources):

• Facial actions (video)

• Physiological signals ( GSR, EEG)

• Voice

• Posture

• ...

– ML techniques

• Classification (basic emotions)

• Regression (dimensional model)



Emotion detection from videos of facial expressions

Problem statement:

– Explicit affective labeling has drawbacks:

• Annoying

• Time consuming

• Potentially inaccurate in real applications

Proposed solution:

– Implicit affective labeling through emotion detection from facial video

– Aggregation of emotions detected from several users



Experiment

2 datasets:

– Posed (Kanade Cohn)

– Spontaneous (LDOS-PerAff-1)

Input: Video streams of facial expressions as responses to visual stimuli

Output: emotive states as distinct classes

Gabor features kNN

Emotive

state



Results and conclusions

Posed dataset: accuracy = 92 %

Spontaneous dataset: accuracy = 62%

Reasons for bad results:

– Weak learning supervision

– Non optimal video acquisition (face rotation, occlusions, changing lightning ...)

– Non extreme facial expressions



Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



The Affective User Modeling framework

Problem statement:

– Research is done in a scattered fashion

– Researchers do not benefit from each other‘s work

Goal:

– Researchers to identify their position

– To benefit from each other‘s work

– To establish affective user modeling as a (sub)field?



The proposed framework - 1

Content application

Give

content

time

Entry stage Consumption stage Exit stage

Give

recommendations

choice




Content application

Entry mood

Detect

entry

mood

Give

content

Exit mood

time


Give

recommendations

choice

• Context

• Decision making

• Influence

• Diversification

• Decision making profile




Content application

Entry mood

Detect

entry

mood

Give

content

Content-induced affective state

Observe user

time


Give

recommendations

choice

• Affective tagging

• Affective user profiles




Content application

Entry mood

Detect

entry

mood

Give

content

Content-induced affective state Exit mood

Observe user

time


Give

recommendations

choice

Detect

exit

mood

• Implicit feedback

• Evaluation metrics

(user satisfaction)




Content application

Entry mood

Detect

entry

mood

Give

content

Content-induced affective state Exit mood

Observe user

time


Give

recommendations

choice

Detect

exit

mood

• Implicit feedback

• Evaluation metrics

(user satisfaction)

• Affective tagging

• Affective user profiles

• Context

• Decision making

• Influence

• Diversification

• Decision making profile



Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



Profiling in CBR systems

Item Profile (md)

Id 1

Title Girl

Genre Erotic

Item Profile (md)

Id 2

Title Basketball

Genre Sport

Item Profile (md)

Id 3

Title Kitchen

Genre Still life

User Profile (up)

Id 1

Action 80

Erotic 60

Sport 95

Still life 35

… …



Proposed solution

We propose tu use AFFECTIVE METADATA

Multimedia content ELICITS (induces) emotions

Underlying assumption: users differ in their preferences for emotions



Affective modeling

Emotion description models

– Basic emotions (Ekman: anger, fear, joy, disgust, surprise,sadness)

– Dimensional model (VAD - valence-arousal-dominance)

We aggregated the emotive responses of many users to a single image:

– First two statistical moments of V, A and D

– Item profile

The user profile is the result of the training an ML classifierArousal

Valence

high

negative positive

low

neutral

sadness

fear

disgust

surprise

joyanger

Valence mean

Valence mean

Dominance mean

Valence mean

Class = 0

Class = 1

Class = 0

Class = 1 Class = 0

<=4.23 >4.23

<=6.71>6.71

<=5.92 >5.92

<=5.21 <=5.21



Experiment

EMOTION

INDUCTION

IAPS Image

Stimuli

Consumed

Item

Metadata

(Item Profile)

Explicit

Rating

Machine

Learning

Ground

Truth

Ratings

Predicted

Ratings

Confusion

Matrix

User Profile

generic

metadata

affective

metadata



Results

Pearson chi-square statistical significance test to compare the confusion

matrices

Scalar measures P, R, F

Generic+affective metadata > generic metadata

Avg(v) best feature (71% of users)

SVM best classifier



Latent factors

Images with affective properties:- Valence- Arousal- Dominance

Users with personality properties:- Extraversion- Agreeableness- Conscientousness- Neuroticism- openness

Matrix factorization

Users – items rating matrix

Users latent factors space

U22

U12 U11

U21

Items latent factors space

I22

I12 I11

I21

Latent factors



Latent factors - results



Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



A personality-based user similarity measure

Collaborative filtering recommender (CFR) systems:

– Similar users have similar preferences

– Rating-based similarity measures

Which content should I watch tonight?



Problem statement

Problem statement:

– New user problem: hard to assess user similarities without overlapping ratings

bad recommendations

Proposed solution (hypothesis)

– A personality based user similarity measure under cold start conditions



personality

Personality: accounts for individual differences ( = explains the variance)

– Old greeks: choleric, melancholic, phlegmatic, sanguine

– The five factor model (FFM) – Big5:

• Extraversion

• Agreeableness

• Conscientousness

• Neuroticism

• Openness

Underlying assumption:

– Users with similar personalities have similar preferences

Measuring personality:

– the IPIP questionnaire

– For each user u a five tuple b =(b1, b2, b3, b4, b5)



Experiment

Proposed USM:

Baseline USM:

Find similar usersGet recommended

itemsF measure

Rating-based USMpersonality-based

USM

Simulate cold-start

stage



Results

F measures of all users:

– At each cold start stage s we compared both USM with the t-test



Overview



modeling!


HOW?

The proposed

AUM framework

Example 1

DatasetExample 2



The LDOS-PerAff-1 dataset

Properties of the dataset

– Content items

– End users

– Generic and affective metadata (for content items)

– Personality metadata (for users)

– Video recordings of users during consumption

– Explicit ratings



Data acquisition setup

•Explanations to the user

•Personality assessment with the IPIP questionnaire

•Computer interaction:

•Emotion induction approach

•Images from the IAPS dataset

•Content

•Stimuli

•Explicit Likert ratings

•Matlab GUI

•Webcam recording



Dataset basic statistics

52 users (avg(age)=18.3 yrs, 37 females)

IPIP 50 items questionnaire

70 colour images from the IAPS dataset

3640 videoclips (320x240 @ 15 fps)



Excerpt



Future work

Looking for a robust, all-encompassing user model

Experimental work to prove parts of the model

Validation in real-world scenarios



Thank you.

Questions?

Technology

Affective User Modeling