View
232
Download
0
Category
Preview:
Citation preview
Pattern Recognition for VisionFall 2004
Principal Component Analysis & Independent Component Analysis
Overview
Principal Component Analysis•Purpose•Derivation•Examples
Independent Component Analysis•Generative Model•Purpose•Restrictions•Computation•Example
Literature
Pattern Recognition for VisionFall 2004
Notation & Basics
Vector Matrix
, , Dot product written as matrix product
Product of a row vector with
T N
a
∈
uUu v u v R
u
22
scalar as matrix product, and not
squared normRules for matrix multiplication:
( ) ( )( )( )
T
T T T
a
= =
= =+ = +
=
u
u u u u
UVW UV W U VWU V W UW VWUV V U
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
PurposeFor a set of samples of a random vector
,discover or reduce the dimensionality and identify meaningful variables.
N∈x R
1u
2u 1u
, : ,p N p N= × <y Ux U
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
PCA by Variance Maximization
AuBu
2 2A B
σ σ>u u
{ } { }{ } { }
{ }
1
1
1
1
2 21 1
21 1 1 1
21 1
1
Find the vector ,such that the variance of the data along this direction is maximized:
( ) , 0, 1
( )( ) ,
,
The solution is the eigenvector of wi
T
T T T T
T T
E E
E E
E
σ
σ
σ
= = =
= =
= =
u
u
u
u
u x x u
u x x u u xx u
u Cu C xx
e C
1
12
1 1 1 1 1 1 1
th the largesteigenvalue .
, T
λ
λ λ σ λ= = ⇔ =uCe e e Ce
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
{ }
For a given , find orthonormal basis vectors such that the variance of the data along these vectors is maximally large, under the constraint of decorrelation:
( )( ) 0, 0
The sol
i
T T Ti n i n
p N p
E n p
<
= = ≠
u
u x u x u u
{ } { }
1 1 2 2 1 2
orthogonal
ution are the eigenvectors of ordered according to decreasing eigenvalues :
, ,..., , ...
Proof of decorrelation for eigenvectors:
( )( )
p p p
T T T T T Ti n i n i n i n nE E
λ
λ λ λ
λ
= = = > >
= = = =
C
u e u e u e
e x e x e xx e e Ce e e 0
PCA by Variance Maximization
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
PCA by Mean Square Error Compression
x
x̂
ku
{ } ( )2
1
For a given , find orthonormal basis vectors such that ˆthe between and its projection into the subspace spanned
by the orthonormal basis vectors is mimimum:
ˆ ˆ, ,p
Ti k k k
k
p N pmse
p
mse E=
<
= − =∑
x x
x x x u x u u ,Tk m k mδ=u
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
{ }
{ }
{ }
{ }( )
2
2
1 scalar
2
1 1 1
2 2 2
1 1
2 2
1
1
ˆ ( )
2 ( ) ( ) ( )
2 ( ) ( )
( )
trace ,
pT
k kk
p p pT T T T T
k k n n k kk n k
p pT T
k kk k
pT
kk
pTk k
k
mse E E
E E E
E E E
E E
=
= = =
= =
=
=
= − = −
= − +
= − +
= −
= −
∑
∑ ∑ ∑
∑ ∑
∑
∑
x x x u x u
x x u x u x u u u x u
x x u x u
x x u
C u Cu C { }TE= xx
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
( ) { }1
maximize
trace ,P
T Tk k
k
mse E=
= − =∑C u Cu C xx
1
Solution to minimizing is any (orthonormal) basis of the subspace spanned by the first eigenvectors
,..., of .p
msep
e e C
( )1 1
tracep N
k kk k p
mse λ λ= = +
= − =∑ ∑C
11
The is the sum of the eigenvalues corresponding to
the discarded eigenvectors ,..., of : N
P N kk p
mse
mse λ+= +
= ∑e e C
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)How to determine the number of principal components ?p
{ } { }{ } { } { } { }
1,1 1,
,1 ,
2
0
Linear signal model with unknown number of signals:
,
Signal have 0 mean and are uncorrelated, is white noise:
,
( )
p
N N p
i
T Tn
T T T T
p N
a aN p
a a
s
E E
E E E E
σ
=
<
= + = ×
= =
= = + +
x As n A
n
ss I nn I
C xx As As nn Asn
{ } { } 2
21 2 1 2... ...
cut off when eigenvalues become constants
T T T Tn
p p p N n
E E
d d d d d d
σ
σ+ +
= + = +
> > > > = = = =
→
A ss A nn AA I
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
Computing the PCA
{ }1
1
1
Given a set of samples ,..., of a random vector calculate mean and covariance.
1 ,
1 ( )( )
Compute eigenvecotors of e.g. with QR algorithm
M
M
ii
MT
i ii
M
M
=
=
= → −
= − −
∑
∑
x x x
µ x x x µ
C x µ x µ
C
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
Computing the PCA
1,1 1,
,1 ,
If the number of samples is smaller than the dimensionality of :
, , : , :
( ) ( ),
MT T
N N M
T
T
T
MN
x xN M M N
x x
λ
λ
λλ λ
= = × × =
′ ′ ′=
′ ′ ′=′ ′= =
x
B C BB B B
BB e eB Be eBB Be Bee Be 2 2
Reducing complexity from( ) to ( ) O N O M
→
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
Examples
Eigenfaces for face recognition (Turk&Pentland):
Training:-Calculate the eigenspace for all faces in the training database-Project each face into the eigenspace feature reduction
Classification:-Projec
→
t new face into eigenspace-Nearest neighbor in the eigenspace
Pattern Recognition for VisionFall 2004
Principal Component Analysis (PCA)
Examples cont.
Feature reduction/extraction
Original Reconstruction with 20 PC
http://www.nist.gov/
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Generative model
( )( )
1
1
1
1,1 1,
1, ,
,1 ,
Noise free, linear signal model:
,..., Observed variables
,..., latent signals, independent components
unknown mixing matrix, ( ,..., )
N
i ii
TN
TN
NT
i i N i
N N N
s
x x
s s
a aa a
a a
=
= =
=
=
= =
∑x As a
x
s
A a
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)Task
For the linear, noise free signal model, compute and given the measurements .
A sx
1 2 3
1 2 3
Blind source separation :separate the three original signals
, , and from their mixtures , , and .
s s sx x x
Figure by MIT OCW.
S3
S2
S1
X1
X2
X3
S1
S 1
S1
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Restrictions
{ } { } { } { }
1 2 1 1 2 2
1 1 2 2 1 1 2 2
1.) Statistical independenceThe signals must be statistically independent:
( , ,..., ) ( ) ( )... ( )Independent variables satisfy:
( ) ( )... ( ) ( ) ( ) ... ( )
for any
i
N N N
N N N N
sp s s s p s p s p s
E g s g s g s E g s E g s E g s
g
=
=
{ }
{ } { }
1
1 1 2 2 1 1 2 2 1 2 1 2
1 1 1 1 2 2 2 2 1 1 2 2
( )
( ) ( ) ( ) ( ) ( , )
( ) ( ) ( ) ( ) ( ) ( )
i s L
E g s g s g s g s p s s ds ds
g s p s ds g s p s ds E g s E g s
∞ ∞
−∞ −∞
∞ ∞
−∞ −∞
∈
=
= =
∫ ∫∫ ∫
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Restrictions
{ } { } { } { }
{ } { } { }
1 1 2 2 1 1 2 2
Statistical independence cont.
( ) ( )... ( ) ( ) ( ) ... ( )Independence includes uncorrelatedness:
( )( ) ( ) ( ) 0
N N N N
i i n n i i n n
E g s g s g s E g s E g s E g s
E s s E s E sµ µ µ µ
=
− − = − − =
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Restrictions
2 21 2
1 2 2 21 2 1 2
2.) Nongaussian componentsThe components must have a nongaussian distributionotherwise there is no unique solution.Example:given and two gaussian signals:
1( , ) exp( )2 2 2
gene
is
s sp s sπσ σ σ σ
= − −
A
2 21 1 2
1 22
Scaling matrix
rate new signals1/ 0 1( , ) exp( )exp( )
0 1/ 2 2 2s sp s s
σσ π
′ ′ ′ ′ ′= ⇒ = − −
S
s s
1σ
2σ
11
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Restrictions
2 21 2
1 2
Rotation matrix
Nongaussian components cont.
under rotation the components remain independent:cos sin 1, ( , ) exp( )exp( )sin cos 2 2 2
combine whitening and rotatio
s sp s sϕ ϕϕ ϕ π
′′ ′′ ′′ ′ ′′ ′′= = − − − R
s s
1
1
n :
is also a solution to the ICA problem.
−
−
=
′′= =
B RSx As AB sAB
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Restrictions
3.) Mixing matrix must be invertibleThe number of independent components is equal to the number of observerd variables.Which means that there are no redundant mixtures.
In case mixing matrix is not invertible apply PCA on measurements first to remove redundancy.
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Ambiguities
{ }1 1
2
1.) Scale1( )( )
Reduce ambiguity by enforcing 1
2.) OrderWe cannot determine an order of the independant components
N N
i i i i ii i i
i
s s
E s
αα= =
= =
=
∑ ∑x a a
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Computing ICA
1
1
2
a) Minimizing mutual information: ˆˆ
ˆ ˆˆMutual information: ( ) ( ) ( )
is the differential etnropy:
ˆ ˆ ˆ ˆ( ) ( ) log ( ( ))
ˆ is always nonnegative and 0 only if the areindepende
N
ii
i
I H s H
H
H p p d
I s
−
=
=
= −
= −
∑
∫
s A x
s s
s s s s
1
nt.ˆ ˆIteratively modify such that ( ) is minimized.I−A s
Pattern Recognition for VisionFall 2004
Independent Component Analysis (ICA)
Computing ICA cont.
b) Maximizing Nongaussianity1
introduce and :From central limit theorem:
is more gaussian than any of the and becomes least gaussian if .
Iteratively modify such that the "gaussianity"of is m
T T T
Ti
i
T
y y
y sy s
y
−=
= = =
=
=
s A xb b x b As q s
q s
b
1
inimized. When a local minimum is reached, is a row vector of .T −b A
Pattern Recognition for VisionFall 2004
PCA Applied to Faces
1x
Mx
1,1x
,1Nx
1,Mx
,N Mx
1u
2u
Each pixel is a feature, each face image a point in the feature space.Dimension of feature vector is given by the size of the image.
1x
2x
3x
1x
Mx
1
are the eigenvectors which can berepresented as pixel images in the originalcooridinate system ...
i
Nx x
u
…
Pattern Recognition for VisionFall 2004
ICA Applied to Faces
…
1,1x
1,Mx
,1Nx
,N Mx
Nx1x
Now each image corresponds to a particular observed variable measured over time (M samples). N is the number of images.
( )( )
1
1
1
,..., Observed variables
,..., latent signals, independent components
N
i ii
TN
TN
s
x x
s s
=
= =
=
=
∑x As a
x
s
S1
S1
Pattern Recognition for VisionFall 2004
PCA and ICA for Faces
Features for face recognition
Image removed due to copyright considerations. See Figure 1 in: Baek, Kyungim et. al. "PCA vs. ICA: A comparison on the FERET data set." International Conference of Computer Vision, Pattern Recognition, and Image Processing, in conjunction with the 6th JCIS. Durham, NC, March 8-14 2002, June 2001.
Recommended