Upload
juan
View
19
Download
0
Embed Size (px)
DESCRIPTION
Automated recognition of facial expressi ns and identity 2003 UCIT Progress Report. Ioulia Guizatdinova Research Group for Emotions, Sociality, and Computing University of Tampere 10.01.2004. Contents. Research problems Aims and Tasks Facial landmark extraction : Methods - PowerPoint PPT Presentation
Citation preview
TAUCHI – Tampere Unit for Computer-Human Interaction
Automated recognition of facial expressi ns and identity
2003 UCIT Progress Report
Ioulia Guizatdinova
Research Group for Emotions, Sociality, and Computing University of Tampere
10.01.2004
TAUCHI – Tampere Unit for Computer-Human Interaction
Contents
• Research problems
• Aims and Tasks
• Facial landmark extraction : Methods
• Facial landmark extraction : Results
• Future Steps
TAUCHI – Tampere Unit for Computer-Human Interaction
Research Problems
“Automated recognition of facial expressions and identity”
• Face identification
• Recognition of facial expressions
TAUCHI – Tampere Unit for Computer-Human Interaction
Face classes
1
…
N
Person A
Person Z
…… … …
Face database
Input face
Face identification
Unrecognized face
still image/video signal
Recognition system
• Classification of input face to one of existing face classes stored in database
• Rejection of input face as unrecognized/unknown face
classification
rejection
Research Problems
TAUCHI – Tampere Unit for Computer-Human Interaction
• Facial expressions affect face recognition because a variability of facial landmarks in their appearance is high
• Humans are good in recognizing facial identity regardless of changes in facial expressions
• Computer-aided systems of face recognition are dramatically compromised by changes in facial expressions
Recognition of facial expressions
Research Problems
TAUCHI – Tampere Unit for Computer-Human Interaction
Aim of research
• The primary aim of this research is to do theoretical and experimental investigation on the possibilities to automatically recognize facial identity independent of changes in facial expressions
• For that purpose two 2D recognition systems will be developed
- Recognition system of facial expressions
- Expression-invariant system of facial identity recognition
TAUCHI – Tampere Unit for Computer-Human Interaction
Tasks
• Extraction and classification of facial landmarks, namely, regions of eyes/eye-brows, nose, and mouth from still images
• Detection and recognition of facial expressions - how facial muscle activations can change appearance of a face during emotional reactions?
• Expression-invariant recognition of facial identity
TAUCHI – Tampere Unit for Computer-Human Interaction
• Four regions of interest have been selected as most informative for further recognition steps
– right eye-brow / eye– left eye-brow / eye– nose– mouth
Methods of landmark extraction
• Uses knowledge on geometrical structure of human faces
• Based on geometrical features of facial landmarks, such as position of eyes/eye-brows, nose, and mouth
Feature-b ased method
TAUCHI – Tampere Unit for Computer-Human Interaction
Methods of landmark extraction
Template-b ase method
• Represents a face as a feature map/template of original facial image
• Local oriented edges are used to construct a feature map of the facial image
• Orientation of edges has been determined with step of 22.5 and encoded as 0,1,…..15
01
2
15
3
4
5
6
789
14
13
12
11
10
22.5
°
Oriented edges extracted in left eye region
TAUCHI – Tampere Unit for Computer-Human Interaction
Methods of landmark extraction
Database
• Tests were performed using Pictures of Facial Affect [1]
• 110 images with 7 basic facial displays: happiness, surprise, fear, anger, disgust and neutral expression
• Images were first normalized to three pre-set sizes 100X150, 200X300 and 300X400 in order to test the effect of image size to the operation of the algorithms
• In sum 110 x 3 = 330 images were used for algorithm testing
[1] Ekman, P., Friesen, W. V., & Hager, J.C. (2002) Facial Action Coding System (FACS). Published by A Human Face, Salt Lake City, UTAH: USA
TAUCHI – Tampere Unit for Computer-Human Interaction
Input face (RGB)Normalization (grey-level scale)
Different resolution levels
Resolution level 0Resolution level 2
Transformation
Pre-processing algorithms
• RGB – grey-level transformation
• Multiresolution image representation was performed using a recursive Gauss transformation
Facial landmark extraction
TAUCHI – Tampere Unit for Computer-Human Interaction
Final feature map
• Final feature map has been constructed on base of local oriented edges extracted in each point of grey-level image at each resolution level with exception of points which had the contrast values less than threshold
• Extraction of local edges has been performed by calculation of difference between two oriented Gaussians with shifted kernels, which allows determining both orientation and contrast of local edge
Map of detected points of interest
Points of interest have been grouped - if the distance between points of interest was less than the threshold the points were grouped - otherwise ignored
Facial landmark extraction
TAUCHI – Tampere Unit for Computer-Human Interaction
matching
0
100
200
300
400
500
0
100
200
300
400
0
100
200
300
400
0
100
200
300
400
2 3 4 5 6 10 11 12 13 14
Edge orientationN
umbe
r of
poi
nts
of in
tere
st
Right Eye
Left Eye
Nose
Mouth
Orientation portraits of the facial landmarks• Detected regions of interest have been
compared with orientation portraits of facial landmarks I have constructed earlier
• Regions which did not correspond to the portraits have been ignored
• Pre-knowledge about facial structure have been used
Facial landmark extraction
TAUCHI – Tampere Unit for Computer-Human Interaction
Finally, facial landmarks have been detected!
Examples of feature maps of high-contrast oriented edges detected from the expressive images
Facial landmark extraction
neutral disgust fear
TAUCHI – Tampere Unit for Computer-Human Interaction
Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes
70
80
90
100
L eye R eye Nose Mouth
Per
form
ance
(%)
100X150 200X300 300X450
Results
TAUCHI – Tampere Unit for Computer-Human Interaction
Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes
70
80
90
100
L eye R eye Nose Mouth
Per
form
ance
(%)
100X150 200X300 300X450
Results
TAUCHI – Tampere Unit for Computer-Human Interaction
Right Eye Left Eye
Nose Mouth
Performance of the feature detection system for three image sizes.N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust
(a)
(c) (d)
(b)
Results
TAUCHI – Tampere Unit for Computer-Human Interaction
Right Eye Left Eye
Nose Mouth
Performance of the feature detection system for three image sizes.N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust
(a)
(c) (d)
(b)
Results
TAUCHI – Tampere Unit for Computer-Human Interaction
• Algorithms are slow – about few seconds
• Errors in groupping points of interest (red rectangles a, b, c)
• Some landmarks are undetectable (d)
Results
Problems
(a) (b) (c) (d)
TAUCHI – Tampere Unit for Computer-Human Interaction
• To improve detection of nose and mouth regions two alternatives are proposed
- The first one is selection of different thresholds for detection and groupping of points of interest for different resolution levels
- The second alternative requires more careful processing of detected regions and searching different landmark parts such as eye and mouth corners and nostrils.
Results
Recommendations
TAUCHI – Tampere Unit for Computer-Human Interaction
Future steps
• Full article about automated expression-invariant detection of facial landmarks; short article about how emotions affect cognitive functioning and how this knowledge might be implicated for HCI
• To improve landmark detection; implement prototype of 2D recognition system of facial expressions (iExpRec)
• To implement and test iExpRec
• To implement of expression-invariant 2D facial identity recognition system (iFaceRec).
TAUCHI – Tampere Unit for Computer-Human Interaction
Thank you f r your attention!