FACIAL EMOTION RECOGNITION BY ADAPTIVE PROCESSING OF

TREE STRUCTURES

Jia-Jun Wong and Siu-Yeung Cho Forensic and Security LabSchool of Computer EngineeringNanyang Technological UniversitySingapore

presented by

Abstract

• Introduction– What are the basic emotions?– Existing systems?

• Facial Emotion Tree Structures– What are Tree Structures?– Why use Tree Structures?– How to process Tree Structures?

• Performance Evaluation– Under perfect feature location situations– With missing features

• Conclusions

Introductions

• There are six basic emotions according to psychologists• Emotions are innate and universal• Facial Emotions are revealed faster• FACS by Paul Ekman

– 100 hours to train• Computerize methods

– Surface texture analysiswith PCA

– Facial Motion through optic flow

– ICA, etc.

Facial Emotion Recognition System

Emotion Recognition

Feature Extraction

Face Detection

Face Cropping and Resizing

Image Processing

Eyes Detection

Nose Detection

Mouth Detection

Gabor Filter Feature Extraction

Facial Emotion Tree Structure

Transformation

Probabilistic Based Recursive Neural

Network

Facial EmotionTree Structure Representation

Recognised Emotion State

Feature Locations

Localised Gabor Features

Capture Image

What are Tree Structures?

• Traditionally features are stored and used in a flat vector format– Simple to implement and use– This loses feature to feature

relationship information• Flat feature vector can be transformed

into tree structures– Encodes feature to feature

relationship information– More flexibility in recognition

A

B

C

D

ca

b dfe

Scene

Sky House Ground

A B C

a b c d e f

Face Emotion Tree Structure (FEETS)

L3L3

L1L1

L2L2

L4L4

L5L5

Adaptive Processing of Tree Structures

• Step 1: Encode Data into Tree Structure

• Step 2: Feed each node into a interconnecting Neural Node

F00

F09 F18

F01 F02

Neural Node

Neural Node

Neural Node

Neural Node

Neural NodeF02

F01

F09

F18

F00

Adaptive Processing of Tree Structures• Maximum number of children for a node, which is the branch factor

is assumed for a task.

Tree Node

GMM_GGMM_1

Output, y

Input attributes, u Children’s output, y

Output layer

Hidden layer

Input layer

Probabilistic Recursive Model

• Class likelihood function

• Unsupervised Learning– Expectation Step

– Maximisation Step

• Supervised Learning– Levenberg Marquardt Algorithm

G

ggg pPp

1

,

TN

jjp log,*

TN

j

G

ggjg

kj npPEnQ

1 1

ˆ,,loglogˆ,

T

T

N

j jg

N

j jjg

gnP

nPn

1

1

ˆ,

ˆ,1

T

T

N

j jg

N

j

Tgjgjjg

gnP

nnnPn

1

1

ˆ,

11ˆ,1

T

N

j jg

g N

nPP

T 1ˆ,

Features Used

• Localized Gabor Features• Biological relevance and computational properties.• Captures the properties of

– spatial localization,– quadrature phase relationship.

Wavelet

Anger

Surprise

Happy

Sad

Fear

Disgust

Feature Locations

• Four primary feature locations– the center of the left eye,– center of the right eye,– tip of the nose,– the center of the lips.

• 60 Extended Features

Performance Evaluations

• Database used

– Japanese Female Facial Expression (JAFFE) Database

• 213 images of 7 facial expressions (including neutral) posed by 10 Japanese female models

Subjects Training Test

Known 143 70

Unknown 170 43

FEETS vs Quadtree

• FEETS are smaller in size• Higher recognition rate

Known Subjects

85.00%

85.50%

86.00%

86.50%

87.00%

87.50%

88.00%

1 2 3 4 5Tree Node Depth

Rate

Quadtree

FEETS

Unknown Subjects

65.00%

70.00%

75.00%

80.00%

1 2 3 4 5

Tree Node Depth

Rate

Quadtree

FEETS

FEETS vs Others

Recognition Rate

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Known Subjects Unknown Subjects

Rate

Navie Bayes

SVM

KNN

FEETS

Missing Features Performance

• Perfect Condition

• Eyes Missing

• Nose & MouthMisssing

FEETS vs Others• Database Used CMU Emotion Database

Known Subjects with Missing Features

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Full No Left Eye No Right Eye No Nose No Mouth No Eyes No Nose Mouth

FEETS

SVM

KNN

Naïve Bayes

FEETS vs Others

Unknown Subjects with Missing Features

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Full No Left Eye No RightEye

No Nose No Mouth No Eyes No NoseMouth

FEETS

SVM

KNN

Naïve Bayes

Conclusions

• FacE Emotion Tree Structures (FEETS) has achieve high recognition rates

• Robust recognition when there are missing features

• Smaller footprint than Quadtrees

Thank You