Facial Affect Displays During Tutoring Sessions

Mattijs Ghijsen – Dirk Heylen*

Rieks op den AkkerAnton Nijholt

2

INES

3

Facial Expressions

• What do they tell about how the student is experiencing the situation?

• How might we be able to use this to improve the tutoring system?

4

Previous Work

• Based on– Student Activity– Difficulty– Errors

• Adjust– Adjust teaching strategy– Adjust dialogue move

5

Pilot

• Collect data• See what happens

6

Interpreting

• If some expression occurred• What triggered it?• What does it mean?

= How can a system make sense of what happened and react appropriately?

7

Elements to consider

• Events:– system– student

• Appraisals• Facial Expressions

8

Facial Expression Dictionary?

• Emotions• Mental state• Conversational function• Adaptor

9

More complex

• Using Stimulus Evaluation Checks:– Situation: how could it be evaluated– Face: what could have triggered it

• Why do it?• Some way of flexibility:

– if match between f.e.-sec and sit.-sec: ok– if no match than: reason

10

Component Process Model

• Scherer• Eclectic• Appraisal process

– Stimulus Evaluation Checks– 5 criteria– Performed sequentially

11

Stimulus Evaluation Checks

Novelty change in the pattern of external or internal stimulation

Intrinsic pleasantness

whether a stimulus event is pleasant, inducing approach tendencies, or unpleasant, inducing avoidance tendencies

Goal/Need significance

whether a stimulus event is relevant to important goals or needs of the organism; outcome consistent with expectations; conducive or obstructive to reaching goals

Coping potential

evaluating the causation of a stimulus and the coping potential available, particularly the degree of control over the event and relative power

Norm/Self compatibility

evaluating whether the event, particularly an action conforms to social norms, cultural conventions…

12

Component Process Model

Function Subsystem Component

Evaluation of stimulation

Information processing

Cognitive

System regulation Support Neurophysiological

Preparation and direction of action

Executive Motivational

Communication and direction of action

Action Expressive

Monitoring, attention, focus, reflection

Monitor Subjective feeling

13

SEC AU

Novelty High Low

Sudden 1 + 2 + 5 + 26/27(1/2=brow raiser; 5= eyelid; 26/27=mouth)

Familiarity 4b + 7

Predictability 4b + 7

14

INES

15

The Exercise

• Ask patient to put arm on table• Disinfect part of the arm• Insert the needle• Inject medicine

16

Movie

17

Movie

18

Movie

19

Movie

20

Movie

21

Movie

22

Movie

23

Situation – Expression

Patient asks for clarification raised brows (2), frown (1), head pulled back (1)

Patient repeatedly does not understand what student says

smile (2)

Patient pulls up her sleeve smile (4), raised brows (2), nod (1), pull head back (1)

Student is disinfecting the arm smile (7), raised brow (1)

24

Facial Expression Appraisal

Expression Novel Pleasant Goal significant

Coping Potential

Smile (22) 3 18 19 14

Raised eyebrows (11)

7 4 2 0

Mouth corners pulled down (2)

2 1 1 1

26

For our application

• Facial expressions tell us something

• that might be of use.– Students are finding it pleasant– Students are concentrating– Students are not frustrated– Students were not expecting

something

27

Discussion

• Not only emotion• Pilot: not real data• Complicated procedure• Many things indeterminate

28

Questions:

• How to classify situations in terms of SECs?– Situation: what to take into account?– Appraisal not situation– Coder consistency

• Facial expression:– how to analyse in terms of SECs– ambiguity – incompleteness

• How to make a system work with such information structures?

29

Problem

• Matching 1) potential appraisals of situation

types with2) potential significance of facial

expression

• 1) = hard to do for types• 2) = lots of ambiguity

30

So

• Why do it?• Some way of flexibility:

– if match between f.e.-sec and sit.-sec: ok

– if no match than: reason