Principle Component Analysis

Principle Component Principle Component AnalysisAnalysis

Presented by:Sabbir Ahmed

Roll: FH-227

OverviewOverviewVariance and CovarianceEigenvector and EigenvaluePrinciple Component AnalysisApplication of PCA in Image

Processing

2

Variance and Variance and Covariance(1/2)Covariance(1/2)The variance is a measure of

how far a set of numbers is spread out.

The equation of variance is

1)var( 1

n

xxxxx

n

iii

3

Variance and Variance and Covariance(2/2)Covariance(2/2)Covariance is a measure of how

much two random variables change together.

The equation of variance is

1

))((),cov( 1

n

yyxxyx

n

iii

4

Covariance MatrixCovariance MatrixCovariance Matrix is a n*n matrix

where each element can be define as

A covariance matrix over 2 dimensional dataset is

),cov( jiM ij

),cov(),cov(),cov(),cov(yyxyyxxx

M

5

EigenvectorEigenvectorThe eigenvectors of a square matrix A are

the non-zero vectors x such that, after being multiplied by the matrix, remain parallel to the original vector.

1112

33

33

6

EigenvalueEigenvalueFor each Eigenvector, the corresponding Eigenvalue is the factor by which the eigenvector is scaled when multiplied by the matrix.

1112

33

33

1

7

Eigenvector and Eigenvalue Eigenvector and Eigenvalue (1/2)(1/2)The vector x is an eigenvector of

the matrix A with eigenvalue λ (lambda) if the following equation holds:

0)(,0,

xIAorxAxor

xAx

8

Eigenvector and Eigenvalue Eigenvector and Eigenvalue (2/2)(2/2)Calculating Eigenvalues

Calculating Eigenvector

0 IA

0)( xIA

9

Principle Component Analysis Principle Component Analysis (1/3)(1/3)PCA (Principle Component

Analysis) is defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance comes to lie on the first coordinate, the second greatest variance on the second coordinate and so on.

10

Principle Component Analysis Principle Component Analysis (2/3)(2/3)

11

Principle Component Analysis Principle Component Analysis (3/3)(3/3)

12

Principle ComponentPrinciple ComponentEach Coordinate in Principle

Component Analysis is called Principle Component.

Ci = bi1 (x1) + bi2 (x2) + … + bin(xn)

where, Ci is the ith principle component, bij is the regression coefficient for observed variable j for the principle component i and xi are the variables/dimensions.

13

Eigenvector and Principle Eigenvector and Principle ComponentComponentIt turns out that the Eigenvectors

of covariance matrix of the data set are the principle components of the data set.

Eigenvector with the highest eigenvalue is first principle component and with the 2nd highest eigenvalue is the second principle component and so on.

14

Steps to find Principle Steps to find Principle ComponentsComponents1. Adjust the dataset to zero mean

dataset.2. Find the Covariance Matrix M3. Calculate the normalized

Eigenvectors and Eigenvalues of M4. Sort the Eigenvectors according to

Eigenvalues from highest to lowest5. Form the Feature vector F using the

transpose of Eigenvectors.6. Multiply the transposed dataset with

F 15

ExampleExample

X Y2.5 2.40.5 0.72.2 2.91.9 2.23.1 3.02.3 2.72 1.61 1.1

1.5 1.61.1 0.9

X Y0.69 0.49-1.31 -1.210.39 0.990.09 0.291.29 1.090.49 0.790.19 -0.31-0.81 -0.81-0.31 -0.31-0.71 -1.01

Original Data Adjusted Dataset16

AdjustedDataSet = OriginalDataSet - Mean

Covariance Matrix Covariance Matrix

716555556.0615444444.0615444444.060.61655555

M

17

Eigenvalues and Eigenvalues and EigenvectorsEigenvectorsThe eigenvalues of matrix M are

Normalized Eigenvectors with corresponding eigenvales are

28402771.10490833989.0

seigenvalue

735178656.0677873399.0677873399.0735178656.0

rseigenvecto

18

Feature VectorFeature VectorSorted eigenvector

Feature vector

677873399.0735178656.0735178656.0677873399.0

rseigenvecto

677873399.0735178656.0735178656.0677873399.0

,

677873399.0735178656.0735178656.0677873399.0

For

FT

19

Final Data (1/2)Final Data (1/2)

X Y-

0.827970186

-0.17511530

71.77758033 0.14285722

7-

0.992197494

0.384374989

-0.27421041

6

0.130417207

-1.67580142 -0.20949846

1-

0.912949103

0.175282444

-0.09910943

7

-0.34982469

81.14457216 0.04641725

820.43804613

70.01776462

971.22382056 -

0.162675287

FinalData = F x AdjustedDataSetTransposed

20

Final Data (2/2)Final Data (2/2)FinalData = F x

AdjustedDataSetTransposed

21

X-0.8279701861.77758033

-0.992197494-0.274210416-1.67580142-0.9129491030.09910943751.144572160.4380461371.22382056

Retrieving Original Retrieving Original Data(1/2)Data(1/2)

FinalData = F x AdjustedDataSetTransposed

AdjustedDataSetTransposed = F-1 x FinalDatabut, F-1=FT

So, AdjustedDataSetTransposed =FT x FinalData

and, OriginalDataSet = AdjustedDataSet + Mean

22

Retrieving Original Retrieving Original Data(2/2)Data(2/2)

23

Application of PCA in Image Application of PCA in Image ProcessingProcessingPattern RecognitionImage CompressionDetermination of Object

Orientation and Rotation

24

QuestionQuestion

?25

ReferencesReferencesPrinciple Component Analysis in

Wikipedia http://en.wikipedia.org/wiki/Principal_component_analysis

A tutorial on Principal Components Analysisby Lindsay I Smith http://www.sccg.sk/~haladova/principal_components.pdf

Principle Component Analysis in Image Processing by M. Mudrov´, A. Proch´zkahttp://dsp.vscht.cz/konference_matlab/matlab05/prispevky/mudrova/mudrova.pdf

26