1
Image Analysis
PCA and Eigenfaces
Christophoros [email protected]
PCA and Eigenfaces
University of Ioannina - Department of Computer Science
Images taken from:D. Forsyth and J. Ponce. Computer Vision: A Modern Approach, Prentice Hall, 2003.Computer Vision course by Svetlana Lazebnik, University of North Carolina at Chapel Hill.Computer Vision course by Michael Black, Brown University.Research page of Antonio Torralba, MIT.
2 Face detection and recognition
C. Nikou – Image Analysis (T-14)
2
3 Face detection and recognition
Detection Recognition “Sally”
C. Nikou – Image Analysis (T-14)
4 Consumer application: iPhoto 2009
C. Nikou – Image Analysis (T-14)
http://www.apple.com/ilife/iphoto/
3
5 Consumer application: iPhoto 2009
It can be trained to recognize pets!
C. Nikou – Image Analysis (T-14)
http://www.maclife.com/article/news/iphotos_faces_recognizes_cats
6 Consumer application: iPhoto 2009
iPhoto decides that this is a face
C. Nikou – Image Analysis (T-14)
4
7 Outline
• Face recognition– EigenfacesEigenfaces
• Face detection– The Viola and Jones algorithm.
C. Nikou – Image Analysis (T-14)
P. Viola and M. Jones. Robust real-time face detection. International Journal of Computer Vision, 57(2), 2004. Also in CVPR 2001.
M. Turk and A. Pentland. "Eigenfaces for recognition". Journal of Cognitive Neuroscience, 3(1):71–86, 1991. Also in CVPR 1991.
8 The space of all face images
• When viewed as vectors of pixel values, face images are extremely high-dimensional
– 100x100 image = 10,000 dimensions
• However, relatively few 10,000-dimensional vectors correspond to valid face
C. Nikou – Image Analysis (T-14)
correspond to valid face images.
• Is there a compact and effective representation of the subspace of face images?
5
9 The space of all face images
• Insight into principal t l icomponent analysis
(PCA).• We want to construct
a low-dimensional linear subspace that
C. Nikou – Image Analysis (T-14)
linear subspace that best explains the variation in the set of face images.
10 Principal Component Analysis
• Given: N data points x1, … ,xN in Rd
• We want to find a new set of features that are linear combinations of the original ones:
w(xi) = uT(xi – µ)( : mean of data points)
C. Nikou – Image Analysis (T-14)
(µ: mean of data points)
• What unit vector u in Rd captures the most variance of the data?
6
11Principal Component Analysis
(cont.)• The variance of the projected data:
Projection of data point
( ) ( )1 1
1 1var ( ) ( ) ( ) ( ) ( )N N TT T T
i i i i ii i
w w wN N= =
= = − −∑ ∑x x x u x μ u x μ
1 1( )( ) ( )( )N N
T T Ti i i i
⎡ ⎤= − − = − −⎢ ⎥∑ ∑u x μ x μ u u x μ x μ u
C. Nikou – Image Analysis (T-14)
Covariance matrix of data
1 1
( )( ) ( )( )i i i ii iN N= =
⎢ ⎥⎣ ⎦∑ ∑u x μ x μ u u x μ x μ u
T= u Σu
12Principal Component Analysis
(cont.)• We now estimate vector u maximizing the variance:
Tu Σusubject to: 2|| || 1T = =u u ubecause any multiple of u maximizes the objective function.
• The Lagrangian is ( ; ) (1 )T TJ λ λ= + −u u Σu u u
u Σu
C. Nikou – Image Analysis (T-14)
The Lagrangian isleading to the solution:
( ; ) (1 )J λ λ= +u u Σu u uλ=Σu u
which is an eigenvector of Σ. The one maximizing J corresponds to the largest eigenvalue of Σ.
7
13Principal Component Analysis
(cont.)• The direction that captures the maximum
covariance of the data is the eigenvector gcorresponding to the largest eigenvalue of the data covariance matrix.
• The top k orthogonal directions that
C. Nikou – Image Analysis (T-14)
capture the most variance of the data are the k eigenvectors corresponding to the klargest eigenvalues.
14 Eigenfaces: Key idea
• Assume that most face images lie on a low-dimensional subspace determineda low dimensional subspace determined by the first k (k<d) directions of maximum variance.
• Use PCA to determine the vectors or “eigenfaces” u1,…uk that span that
b
C. Nikou – Image Analysis (T-14)
subspace.• Represent all face images in the dataset
as linear combinations of eigenfaces.
8
15 Eigenfaces example
Training images
C. Nikou – Image Analysis (T-14)
16 Eigenfaces example (cont.)Top eigenvectors
Mean: μ
C. Nikou – Image Analysis (T-14)
9
17 Eigenfaces example (cont.)
uk
3μ - uk kλ
C. Nikou – Image Analysis (T-14)
3μ uk kλ+
18 Eigenfaces example (cont.)
• Face x in “face space” coordinates:
( ) ( ) ( )TT T T⎡ ⎤
[ ]1 2
1 2
( ), ( ),..., ( )
, ,...,
x u x μ u x μ u x μT T Tk
Tkw w w
⎡ ⎤→ − − −⎣ ⎦
=• Reconstruction:
C. Nikou – Image Analysis (T-14)
+=
+x̂ = μ 1 1 2 2 ...u u uk kw w w+ + +
10
19 Eigenfaces example (cont.)
• Any face of the training set may be expressed as a linear combination of a pnumber of eigenfaces.
• In matrix-vector form:
[ ]1
21 2x μ u u uk
ww⎡ ⎤⎢ ⎥⎢ ⎥= +⎢ ⎥
1=K
ii kk KE
λ
λ= +∑
∑
Error:
C. Nikou – Image Analysis (T-14)
+=
[ ]1 2μ k
kw⎢ ⎥⎢ ⎥⎣ ⎦
1
Kiiλ
=∑
20 Eigenfaces example (cont.)
Intra-personal subspace (variations in expression)
C. Nikou – Image Analysis (T-14)
Extra-personal subspace (variations between people)
11
21 Face Recognition with eigenfaces
Process the training images:• Find mean µ and covariance matrix Σ.• Find k principal components (eigenvectors of Σ) u u• Find k principal components (eigenvectors of Σ) u1,…uk.• Project each training image xi onto subspace spanned by
principal components:(wi1,…,wik) = (u1
T(xi – µ), … , ukT(xi – µ))
Given a new image x:• Project onto subspace:
T T
C. Nikou – Image Analysis (T-14)
(w1,…,wk) = (u1T(x – µ), … , uk
T(x – µ)).• Optional: check the reconstruction error x – x to determine
whether the new image is really a face.• Classify as closest training face in k-dimensional subspace.
^
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
22 Computation of eigenvectors
• Given the d-dimensional vectorized images xi(i=1,2…,N), we form matrix X=[x1, x2,…, xN], d >>N.
• Covariance matrix:
1 TXXN
Σ =
• Σ has dimensions dxd which, for example, for a
C. Nikou – Image Analysis (T-14)
200x200 image is very large (40000x40000).• How do we compute its eigenvalues and
eigenvectors?
12
23 Computation of eigenvectors (cont.)
• We create the NxN matrix:1 TT X X1 TT X XN
=
• T has N eigevalues (γi) and eigenvectors (ei):
i i iTe eγ= 1 Ti i iX Xe e
Nγ⇔ =
1 Ti i iXX Xe Xe
Nγ⇔ =
C. Nikou – Image Analysis (T-14)
N N( ) ( )i i iXe Xeγ⇔ Σ =
• γi is an eigenvalue of Σ and Xei is the correspoding eigenvector.
24The importance of the mean image
and the variations
C. Nikou – Image Analysis (T-14)
Average of 100 of the images from the Caltech-101 dataset. Reproduced from the web page of A. Torralba http://web.mit.edu/torralba/www/gallery/gallery11.html
What is this?
13
25The importance of the mean image
and the variations (cont.)
C. Nikou – Image Analysis (T-14)
The context plays an important role. Is it a car or a person? Both blobs correspond to the same shape after a 90 degrees rotation.
Reproduced from the web page of A. Torralba.http://web.mit.edu/torralba/www/gallery/gallery11.html
26The importance of the mean image
and the variations (cont.)
C. Nikou – Image Analysis (T-14)
Average fashion model. Average of 60 aligned face images of fashion models.
Reproduced from the Perception Lab, University of St Andrews in Fife, UK. http://perception.st-and.ac.uk
14
27Application: MRF features for texture
representation• Each pixel is described by its 7x7 neighborhood
in every Lab channel.• This results in a 7x7x3 = 147-dimensional vector
per pixel.• Applying PCA to these vectors leads to a
compact 6- to 8-dimensional representation of natural images.
C. Nikou – Image Analysis (T-14)
• It captures more than 90% of the variation of the image pixels neighborhoods.
28Application: MRF features for texture
representation (cont.)
C. Nikou – Image Analysis (T-14)
15
29 Application: view-based modeling
• PCA on various views of a 3D object.
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
30Application: view-based modeling
(cont.)• Subspace of the first three PC (varying view angle).
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
16
31Application: view-based modeling
(cont.)• View angle recognition.
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
32Application: view-based modeling
(cont.)• Extension to more 3D objects with varying pose and
illumination.
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
17
33 Application: EigenheadsPrincipal components 1 and 2.
C. Nikou – Image Analysis (T-14)
34 Application: Eigenheads (cont.)Principal components 3 and 4.
C. Nikou – Image Analysis (T-14)
18
35 Limitations
• Global appearance method– not robust to misalignment, backgroundnot robust to misalignment, background
variation.
C. Nikou – Image Analysis (T-14)
36 Limitations (cont.)
• Projection may suppress important detail– The smallest variance directions may not be
unimportant. PCA assumes Gaussian data.
C. Nikou – Image Analysis (T-14)
The shape of this dataset is not well described by its principal components
19
37 Limitations (cont.)
• PCA does not take the discriminative task into account
– Typically, we wish to compute features that allow good discrimination.
• Projection onto the major axis can not separate the green from the red data.
C. Nikou – Image Analysis (T-14)
• The second principal component captures what is required for classification.
38 Canonical Variates
• Also called “Linear Discriminant Analysis”• A labeled training set is necessaryA labeled training set is necessary.• We wish to choose linear functions of the
features that allow good discrimination.– Assume class-conditional covariances are
the same.
C. Nikou – Image Analysis (T-14)
– We seek for linear feature maximizing the spread of class means for a fixed within-class variance.
20
39 Canonical Variates (cont.)
• We have a set of vectorized images xi(i=1,2…,N).( , , )
• The images belong to C categories each having a mean μj, ( j=1,2,…,C ).
• The mean of the class means:
1 C
C. Nikou – Image Analysis (T-14)
1
1j
jCμ μ
=
= ∑
40 Canonical Variates (cont.)
• Between class covariance:C
T( )( )TB i i i
i 1
S N μ μ μ μ=
= − −∑with Ni being the number of images in class i.
• Within class covariance:
C. Nikou – Image Analysis (T-14)
• Within class covariance:
( )( )iNC T
W k ,i i k ,i ii 1 k 1
S x xμ μ= =
= − −∑∑
21
41 Canonical Variates (cont.)
Using the same argument as in PCA (and graph-based segmentation) we seek vector u maximizing
which is equivalent to
maxT
BTu
W
u S uu S u
⎧ ⎫⎨ ⎬⎩ ⎭
max{ }TBu
u S u s.t. 1TWu S u =
C. Nikou – Image Analysis (T-14)
B WS u S uλ=
The solution is the top eigenvector of the generalized eigenvalue problem:
42 Canonical Variates (cont.)
C. Nikou – Image Analysis (T-14)
• 71 views of 10 objects at a variety of poses at a black background.• 60 images were used to determine a set of canonical variates.• 11 images were used for testing.
22
43 Canonical Variates (cont.)
C. Nikou – Image Analysis (T-14)
• The first two canonical variates. The clusters are tight and well separated (a different symbol per object is used).
• We could probably get a quite good classification.