http://www.cubs.buffalo.edu

Face Recognition

University at Buffalo CSE666 Lecture Slides

Resources:http://www.face-rec.org/algorithms/

http://www.cubs.buffalo.edu

Overview of face recognition algorithms

• Correlation- Pixel based correspondence between two face images

• Structural- Based on identifying landmark points

• Linear models- PCA (Principal Component Analysis)- LDA (Linear Discriminant Analysis)- ICA (Independent Component Analysis)- Combinations of above

• Non-linear models-Kernel mapping (and using PCA, LDA, ICA)-Active shape/appearance models-Manifold mapping

• 2D Model vs. 3D Model• Matching vs. Classification

http://www.cubs.buffalo.edu

Correlation MatchingTest Image Test Image captured captured by by cameracamera

Correlation Correlation Filter H Filter H (Template)(Template)

Frequency Frequency Domain arrayDomain array

FFTFFT

N x N pixelsN x N pixels

N x N pixelsN x N pixelsResulting Resulting Frequency Frequency Domain arrayDomain array

IFFTIFFT

PSRPSR

*B.V.K. Vijaya Kumar, Marios Savvides, C. Xie, K. Venkataramani, J. Thornton and A. Mahalanobis, “Biometric Verification using Correlation Filters”, Applied Optics, 2003*B.V.K. Vijaya Kumar, M. Savvides, K. Venkataramani, C. Xie, "Spatial frequency domain image processing for biometric recognition," IEEE Proc. of International Conference on Image Processing (ICIP), Vol. I, 53-56, 2002

http://www.cubs.buffalo.edu

Elastic Bunch Graph Matching

L. Wiskott, J.-M. Fellous, N. Krueuger, C. von der Malsburg, Face Recognition by Elastic Bunch GraphMatching, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, 1997

http://www.cubs.buffalo.edu

Elastic Bunch Graph Matching

L. Wiskott, J.-M. Fellous, N. Krueuger, C. von der Malsburg, Face Recognition by Elastic Bunch GraphMatching, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, 1997

• Heuristic matching algorithm

http://www.cubs.buffalo.edu

Principal Component Analysis (PCA)

Previous discussion: PCA as a feature extraction method

( )iaA =

∑=

=n

iiiy

1

ax

∑=

=m

iiiy

1

ˆ ax

- representation of original vector in

KL basis vectors

- the projection on subspace spanned by eigenvectors with largest eigenvalues

∑+=

=−n

miiE

1

2 ])ˆ[( λxx

- Karhunen-Loeve transform

ia - eigenvectors of matrix ][ Tx E xxR =

x

m

Axy = - projection of original vector to KL basisx

xAy mm = - first PCA coefficients –‘best’ features

m

Features Representation

- sum of the smallest eigenvaluesmn −

http://www.cubs.buffalo.edu

PCA as appearance model

xAy mm = - first PCA coefficients –

‘best’ features - should provide a good approximation to all the samples (for which PCA was trained)

• Distance from feature space (DFFS) should be small:

m

|ˆ| xx −=DFFS

x

x̂

• PCA provides a model on typical faces• DFFS can be used it to separate faces from non-faces

• DIFS (distance in feature space) can be used to match two faces

mTm yAx =ˆ

|ˆˆ| 21 xx −=DIFS

http://www.cubs.buffalo.edu

PCA for face recognition

• Collect a set of face images for training

• Train PCA

• During face recognition see if the distance between test and enrolled PCA feature vectors is less than threshold:

M. Turk, A. Pentland, Eigenfaces for Recognition, Journal of Cognitive Neurosicence, Vol. 3, No. 1, 1991

θ<−=− |||ˆˆ| 2121 yyxx

x̂ - reconstructed face (hyperplane in the original feature space)- projection into PCA feature spacey

http://www.cubs.buffalo.edu

If has dimension , then has dimension

PCA – subspace on training samples

M. Turk, A. Pentland, Eigenfaces for Recognition, Journal of Cognitive Neurosicence, Vol. 3, No. 1, 1991

][ Tx E xxR =x

2N 22 NN ×

Suppose we have training samplesM ][ 21 MxxxX …=

Tx XXR = matrix of dimension 22 NN ×

Instead consider of dimensionXXL T= MM ×

If is an eigenvector of then is an eigenvector of : v L Xv xR

XvXvXXvXvX λλ =⇒= TT

• The number of training samples is usually less than the dimension of image vectors , so this procedure makes sense2NM <

http://www.cubs.buffalo.edu

Linear Discriminant Analysis

Covariance matrix for class :

]))([( Tiii E µxµxS −−=

i

Within-class scatter matrix: ∑=

=M

iiiw P

1

SS

Between-class scatter matrix: ∑=

−−=M

i

Tiiib P

100 ))(( µµµµS

∑=

=M

iiiP

10 µµwhere

)det()det(

)det( 1bw

w

b SSSS −=Optimization criteria: maximize

http://www.cubs.buffalo.edu

Linear Discriminant Analysis

bw SS 1−Solution: projection is determined by the eigenvectors of

• Usually has better performance than PCA• Requires samples of the same class (same person face) to train• Need to make sure that is non-singular

wSwS

Solution:apply PCA first to reduce the number of dimensions, then perform LDA.

http://www.cubs.buffalo.edu

Independent Component Analysis

• Whereas PCA makes uncorrelated features, there might be still dependent• ICA tries to reduce higher order dependence• Search for proper projection is more difficult• Use approximationapproaches

Bartlett et al. ”Face Recognition by Independent Component Analysis”, 2002

http://www.cubs.buffalo.edu

Comparisons of subspace methods

Delac et al. “Independent Comparative Study of PCA, ICA and LDA on the FERET Data Set”, 2006

http://www.cubs.buffalo.edu

Comparisons of subspace methods

Delac et al. “Independent Comparative Study of PCA, ICA and LDA on the FERET Data Set”, 2006

http://www.cubs.buffalo.edu

Kernel Methods• Kernel mapping:

-map original image feature vectors into higher dimensional spaceusing some kernel functions:

- Covariance matrix in kernel space has elements

- Kernel trick: there exists a function

- No need to explicitly calculate - use some choosen kernel function:

- The number of eigenvectors is still limited by the number of training samples as in regular PCA

NfRR fN >>→Φ ,:

)()( ji xx Φ⋅Φ

ijjiji Kk =Φ⋅Φ= )()(),( xxxx

)( ixΦ

2

2

2),( σji

ek ji

xx

xx−

−=

http://www.cubs.buffalo.edu

Kernel Methods

M.-H. Yang, “Face Recognition Using Kernel Methods”, 2002

http://www.cubs.buffalo.edu

Local Linear Embedding

Roweis S., Saul L. “Nonlinear dimensionality reduction by locally linear embedding”, Science, 2000

http://www.cubs.buffalo.edu

• Step 1: find optimal local linear representation of point as a combination of its neighbors

• Step 2: find lower dimensional representation of

Local Linear Embedding

Roweis S., Saul L. “Nonlinear dimensionality reduction by locally linear embedding”, Science, 2000

min)( →−= ∑ ∑i j

jiji XWXW��

ε

iX�

jX�

1=∑j

ijW

0=ijW if j is not a neighbor of i

min)( →−=Φ ∑ ∑i j

jiji YWYY��

iY�

iX�

http://www.cubs.buffalo.edu

Local Linear Embedding

Roweis S., Saul L. “Nonlinear dimensionality reduction by locally linear embedding”, Science, 2000

http://www.cubs.buffalo.edu

Local Linear Embedding

http://www.cubs.buffalo.edu

3D Model Based Recognition

Blanz, Vetter “Face recognition based on fitting a 3D morphable model” PAMI 200303

http://www.cubs.buffalo.edu

3D Model Based Recognition

Blanz, Vetter “Face recognition based on fitting a 3D morphable model” PAMI 200303

http://www.cubs.buffalo.edu

3D Model Based Recognition

Blanz, Vetter “Face recognition based on fitting a 3D morphable model” PAMI 200303