Face Recognition

Caglar Tirkaz

Biometrics – CS 516

OutlineDefinition of face recognition

Challenges in face recognition

Face recognition techniques

Databases

Conclusion

Face Recognition A face recognition system is expected to identify faces

present in images and videos automatically. It can operate in either or both of two modes:

Face verification (or authentication): involves a one-to-one match that compares a query face image against a template face image whose identity is being claimed.

Face identification (or recognition): involves one-to-many matches that compares a query face image against all the template images in the database to determine the identity of the query face.

First automatic face recognition system was developed by Kanade 1973. [1]

Why is it hard?• “The variations between the images of the same face due to illumination and viewing direction are almost always larger than the image variation due to change in identity. “[2]

•The figure demonstrates the nonlinearity and nonconvexity of face manifolds in a PCA subspace spanned by the first three principal components, where the plots are drawn from real face image data. [3]

•Face recognition evaluation reports [4] and other independent studies indicate that the performance of many state-of-the- art face recognition methods deteriorates with changes in lighting, pose, and other factors

Why is it hard? - 2Large Variability in Facial Appearance: Whereas

shape and reflectance are intrinsic properties of a face object, the appearance (i.e., the texture look) of a face is also subject to several other factors, including the facial pose (or, equivalently, camera viewpoint), illumination, facial expression. [3]

Why is it hard? - 3Highly complex and non-linear manifolds:

The entire face manifold is highly nonconvex, and so is the face manifold of any individual under various change. Linear methods such as PCA, LDA, ICA can not preserve the nonconvex variations of face manifolds necessary to differentiate among individuals.

High dimensionality and small sample size: Another challenge is the ability to generalize. A canonical face image of 112×92 resides in a 10,304-dimensional feature space. Nevertheless, the number of examples per person (typically fewer than 10, even just one) available for learning the manifold is usually much smaller than the dimensionality of the image space.

Face Recognition Processing Flow

Face detection segments the face areas from the background. In the case of video, the detected faces may need to be tracked using a face tracking component.Face alignment is aimed at achieving more accurate localization and at normalizing faces thereby whereas face detection provides coarse estimates of the location and scale of each detected face. This normalization is both in terms of geometry and illumination.Feature extraction is performed to provide effective information that is useful for distinguishing between faces of different individuals.Face matching is the step where the extracted feature vector of the input face is matched against those of enrolled faces in the database.

Methods to Use

•With the approach which was introduced by Turk and Pentland in 1991 a small number of eigenfaces are derived from a set of training face images by using PCA [5]. A face image is efficiently represented as a feature vector (i.e., a vector of weights) of low dimensionality. The use of subspace modeling techniques has significantly advanced face recognition technology.

•This method is important not only because of its success but also because it is a baseline for many other algorithms and moreover it can be used as a benchmark for recognition tests.

Subspace analysis techniques for face recognition are based on the fact that a class of patterns of interest, such as the face, resides in a subspace of the input image space. For example, a small image of 64 × 64 has 4096 pixels can express a large number of pattern classes.

Methods to Use - 2• Feature extraction: construct a “good”

feature space in which the face manifolds become simpler i.e. less nonlinear and nonconvex than those in the other spaces. This includes two levels of processing: Normalize face images geometrically and photometrically,

such as using morphing and histogram equalization Extract features in the normalized images which are stable

with respect to such variations, such as based on Gabor wavelets.

• Pattern classification: construct classification engines able to solve difficult nonlinear classification and regression problems in the feature space and to generalize better

A Taxanomy of Face Recognition Methods

• As can be seen on the figure [6] most work on face recognition is carried out using2D intensity images. Although these techniques can achieve good performancesunder controlled conditions, their performance degrade when pose, illumination, and scale parameters change this is why 3D techniques for face recognition are developed which are thought to be advantageous since the 3D shape of the face does not change due to these factors

Eigenfaces - 1The method was introduced by Turk and

Pentland [5]Very fast and efficientThe idea is that any face can be

represented as a linear combination of eigenfaces

To generate eigenfaces large number of face images taken under the same lighting conditions which are normalized with respect to eyes and mouth are used

PCA is applied to those images to create eigen faces

Eigenfaces - 2Principal component analysis (PCA), or

Karhunen-Loeve transformation, is a data-reduction method that finds an alternative set of parameters for a set of raw data (or features) such that most of the variability in the data is compressed down to the first few parameters•The transformed PCA parameters are orthogonal•The PCA diagonalizes the covariance matrix, and the resulting diagonal elements are the variances of the transformed PCA parameters [11-12]

Eigenfaces - 3A face image defines a point

in the high-dimensional image space

Different face images share a number of similarities with each other

They can be described by a relatively low-dimensional subspace

They can be projected into an appropriately chosen subspace of eigenfaces and classification can be performed by similarity computation (distance)

Eigenfaces - 4When the viewing angle and

illumination conditions change the performance drops rapidly, so the images should be frontal with similar illumination conditions

96% recognition with light variation, 85% with orientation variation, 64% with size variation [5]

Eigenfaces - 5

Typical EigenfacesSource: MIT Face Recognition Demo Page

The top left is mean face and the remaining are the eigen vectors that construct the PCA space

Fisherfaces - 1Developed in 1997 by P.

Belhumeur et al. [7]Based on Fisher’s LDA which was

developed by Robert Fisher in 1936

Has lower error rates compared to eigenfaces

Works well even if different illumination conditions appear

Fisherfaces - 2LDA seeks directions that are

efficient for discrimination between the data

Class A

Class B

Elastic Graph Matching (EGM)Each face is represented by a set of feature

vectors positioned on the nodes of a coarse 2D grid placed on the face

Each feature vector is comprised of a set of responses of 2D Gabor wavelets, differing in orientation and scale

Comparing two faces is accomplished by matching and adapting the grid of a test image to the grid of a reference image, where both grids have the same number of nodes; the test grid has initially the same structure as the reference grid.

The elasticity of the test grid allows accommodation of face distortions (e.g., due to the expression change) and to a lesser extent, changes in the view point.

The quality of match is evaluated using a distance function

EGM - 2

Active Appearance Model (AAM)

Create a model of the face to interpret the face images

Split the changes that can appear on the face into two parts – Shape and texture

Learn the ways in which the shape and texture of the face vary across a range of images

Use images annotated with a set of feature points defining correspondences across the set.

AAM - 2AAM was introduced by Cootes

et.al in 1998 [8]

• Given a new image of a face AAM locates the facial features and synthesizes a face similar to the image

Morphable Models of FacesThe idea of morphable models

was introduced by Vetter et. al. (1999)

Based on an analysis by synthesis framework

Requires a generative model able to accurately synthesize face images.

Uses a three-dimensional (3D) representation to accurately model illumination, pose and expression

Morphable Models of Faces - 2

• Here is a link that shows morphable models in action

• This is the framework used in face recognition using Morphable models

Morhable Models of Faces - 3

Identification performances on FERETThe overall mean of the table is 92.9%

Identification performance on CMU-PIE database. The overall mean of the table is 92.1%

Morhable Models of Faces - 4Morphable models address in a

natural way such difficult problems as combined variations of pose and illumination

Limited by its computational load. However this disadvantage will evaporate with time as computer increase their clock speed.

Face Recognition Vendor Test (FRVT) [4]

• Evaluation of face recognition technology.

• The FRVT 2006 established the first independent performance benchmark for 3D face recognition technology. • FRVT 2006 and Iris Cahllenge Evaluation (ICE) 2006 are the first technology evaluations that allowed iris recognition, still face recognition, and 3D face recognition performance to be compared. • The FRGC was a face recognition technology development effort that supported the development of the face recognition algorithms from high-resolution still and 3D imagery. The goal of the Face Recognition Grand Challenge (FRGC) was a decrease in the error rate of face recognition algorithms by an order of magnitude. The FRVT 2006 documented a decrease in the error rate by at least an order of magnitude over what was observed in the FRVT 2002

FRVT - 2

The reduction in error rate for state-of-the-art face recognitionalgorithms as documented through the FERET, the FRVT 2002, and theFRVT 2006 evaluations.

FRVT - 3

An example of the types of images used in FRVT 2006. The first column shows two frontal images taken under controlled illumination with neutral and smiling expressions. The second column shows two images taken under uncontrolled illumination with neutral and smiling expressions. The third contains a 3D facial image. The top image is the shape channel only and the bottom image has the texture channel on top of the shape channel.

FRVT - 4FRR of 0.01 at a FAR of 0.001 was

achieved by Neven Vision (NV1-norm algorithm) on the very high-resolution still images and Viisage (V-3D-n algorithm) on the 3D images.

The improvement in algorithm performance between FRVT 2002 and FRVT 2006 is due to advancement in algorithm design, sensors, and understanding of the importance of correcting for varying illumination across images.

FRVT - 5FRVT 2006 integrated human face

recognition performance into an evaluation for the first time.

This inclusion allowed a direct comparison between humans and state-of-the-art computer algorithms.

The study focused on recognition across changes in lighting.

The experiment matched faces taken under controlled illumination against faces taken under uncontrolled illumination.

FRVT - 6Face pairs were presented side by side on

the computer screen for two seconds. After each pair of faces was presented, subjects rated the similarity of the two faces on a scale of 1 to 5. Subjects responded, using labeled keys on the keyboard as follows:

1.) You are sure they are the same person; 2.) You think they are the same person; 3.) You don’t know; 4.) You think they are different people; 5.) You are sure they are different people.

FRVT - 7ROC of human and computer performance on matching faces across illumination changes. The ROC plots FAR against FRR. Perfect performance would be the lower left hand corner (FAR=FRR=0)

Face Databases• Below is a table of the mostly used Face Databases (Taken from Handbook of face recognition)

Overview of the recording conditions for all databases.

Cases where the exact number of conditions is not determined (either because the underlyingmeasurement is continuous or the condition was not controlled for during recording) are markedwith “++.”

ConclusionRobust face recognition systems

should be able to handle the variations that occur under practical operational scenarios

The FRGC database is currently the largest database with the most challenging practical variations which challenge simple algorithms such as PCA by yielding only 12% verification at 0.1% FAR

Conclusion - Advantages• Photos of faces are widely used in passports and

driver’s licenses where the possession authentication protocol is augmented with a photo for manual inspection purposes; there is wide public acceptance for this biometric identifier

• Face recognition systems are the least intrusive from a biometric sampling point of view, requiring no contact, nor even the awareness of the subject

• The biometric works, or at least works in theory, with legacy photograph data-bases, videotape, or other image sources

• Face recognition can, at least in theory, be used for screening of unwanted individuals in a crowd, in real time

• It is a fairly good biometric identifier for small-scale verification application

Conclusion - DisadvantagesA face needs to be well lighted by controlled

light sources in automated face authentication systems. This is only a first challenge in a long list of technical challenges that are associated with robust face authentication

Face, currently, is a poor biometric for use in a pure identification protocol

An obvious circumvention method is disguise

There is some criminal association with face identifiers since this biometric has long been used by law enforcement agencies (‘mugshots’).

References1. T. Kanade. Picture Processing by Computer Complex and Recognition of

Human Faces. Ph.D. thesis, Kyoto University, 1973.2. Y. Moses, Y. Adini, and S. Ullman. Face recognition: The problem of

compensating for changes in illumination direction. In Proceedings of the European Conference on Computer Vision, volume A, pages 286–296, 1994.

3. Stan Z. Li, Anil K. Jain. Handbook of Face Recognition. Pages 1-134. Face Recognition Vendor Tests (FRVT). http://www.frvt.org.5. M. A. Turk and A. P. Pentland. Eigenfaces for recognition. Journal of Cognitive

Neuroscience, 3(1):71–86, 1991.6. Fahri Tuncer. 3D Face Representation and recognition using spherical

harmonics. Ph.D. Thesis, Middle East Technical University, 2008.7. P. Belhumeur, J. Hespanha, D. Kriegman, Eigenfaces vs Fisherfaces:

Recognition using Class Specific Linear Projection. 19978. T.F.Cootes, G.J.Edwards, C.J.Taylor. , Active Appearance Models, ECCV'989. Volker Blanz, Thomas Vetter. A morphable model for the synthesis of 3D

faces. International Conference on Computer Graphics and Interactive Techniques,. 1999

10. Signal Processing Institute, Swiss Federal Institute of Technology http://scgwww.epfl.ch/

11. Biometric Systems Lab, University of Bologna http://bias.csr.unibo.it/research/biolab/

Questions?