240-650 Principles of Pattern Recognition

240-572: Appendix A: Mathematical Foundations

1

Montri [email protected]://fivedots.coe.psu.ac.th/~montri

240-650 Principles of Pattern

Recognition


2

Appendix A

Mathematical Foundations


3

Linear Algebra

• Notation and Preliminaries• Inner Product• Outer Product• Derivatives of Matrices• Determinant and Trace• Matrix Inversion• Eigenvalues and Eigenvectors


4

Notation and Preliminaries

• A d-dimensional column vector x and its transpose xt can be written as

d

d

xxx

x

x

x

.. and

.

. 21

2

1

txx


5

Inner Product

• The inner product of two vectors having the same dimensionality will be denoted as xty and yields a scalar:

xyyx tt

d

iii yx

1


6

Euclidian Norm (Length of vector)

• We call a vector normalized if ||x|| = 1

• The angle between two vectors

xxx t

yx

yxtcos


7

Cauchy-Schwarz Inequality

• If xty = 0 then the vectors are orthogonal

• If ||xty|| = ||x||||y| then the vectors are colinear.

yxyxt


8

Linear Independence

• A set of vectors {x1, x2, x3, …, xn} is linearly independent if no vector in the set can be written as a linear combination of any of the others.

• A set of d L.I. vectors spans a d-dimensional vector space, i.e. any vector in that space can be written as a linear combination of such spanning vectors.


9

Outer Product

• The outer product of 2 vectors yields a matrix

mnn

m

m

n yxyx

yx

yxyxyx

yyy

x

x

x

...

.....

.....

....

..

...:

1

12

12111

212

1

txyM


10

Determinant and Trace

• Determinant of a matrix is a scalar• It reveals properties of the matrix• If columns are considered as vectors,

and if these vector are not L.I. then the determinant vanishes.

• Trace is the sum of the matrix’s diagonal elements

d

iiim

1

tr M


11

Eigenvectors and Eigenvalues

• A very important class of linear equations is of the form

• The solution vector x=ei and corresponding scalar are called the eigenvector and associated eigenvalue, respectively

• Eigenvalues can be obtained by solving the characteristic equation:

scalar for or 0xIMxMx

i

0IM det


12

Example

• Let find eigenvalues and associated

eigenvectorsCharacteristic Eqn:

12

13M

0 54

0 213

012

13det

2


13

Example (cont’d)

j 2Solution:

Eigenvalues are:

Each eigenvector can be found by substituting each eigenvalue into the equation

then solving for x1 in term of x2 (or vice versa)

jj 2 ,2 21

xMx


14

Example (cont’d)

• The eigenvectors associated with both eigenvalues are:

j1

1 :

1

1 :

2

1

e

je


15

Trace and Determinant

• Trace = sum of eigenvalues• Determinant = product of eigenvalues

d

ii

d

iitr

1

1

M

M


16

Probability Theory

• Let x be a discrete RV that can assume any of the finite number of m of different values in the set X = {v1, v2, …, vm}. We denote pi the probability that x assumes the value vi :

pi = Pr[x=vi], i = 1..m

• pi must satisfy 2 conditions

m

iii pp

1

1 and 0


17

Probability Mass Function

• Sometimes it is more convenient to express the set of probabilities {p1, p2, …, pm} in terms of the probability mass function P(x), which must satisfy the following conditions:

m

i

xPxP1

1 and 0

For Discrete x


18

Expected Value

• The expected value, mean or average of the random variable x is defined by

• If f(x) is any function of x, the expected value of f is defined by

Xx

m

iii pvxxPx

1

Xx

xPxfxf )()(


19

Second Moment and Variance

• Second moment

• Variance

• Where is the standard deviation of x

Xx

xPxx 22

Xx

xPxxxVar 222 )()(


20

Variance and Standard Deviation

• Variance can be viewed as the moment of inertia of the probability mass function. The variance is never negative.

• Standard deviation tells us how far values of x are likely to depart from the mean.


21

Pairs of Discrete Random Variables

• Joint probability

• Joint probability mass function

• Marginal distributions

jiij wyvxp ,Pr

yxP ,

Xxy

Yyx

yxPyP

yxPxP

),()(

),()(


22

Statistical Independence

• Variables x and y are said to be statistically independent if and only if

• Knowing the value of x did not give any knowledge about the possible values of y

yPxPyxP yx,


23

Expected Values of Functions of Two Variables

• The expected value of a function f(x,y) of two random variables x and y is defined by


24

Means and Variances


25

Covariance

• Using vector notation, the notations of mean and covariance become

tXYx

P

μxμxΣ

xxxμ


26

Uncorrelated

• The covariance is one measure of the degree of statistical dependence between x and y.

• If x and y are statistically independent then

and The variables x and y are said to be uncorrelated

0xy


27

Conditional Probability

• conditional probability of x given y

• In terms of mass functions


28

The Law of Total Probability

• If an event A can occur in m different ways, A1, A2, …, Am, and if these m subevents are mutually exclusive then the probability of A occurring is the sum of the probabilities of the subevents Ai.


29

Bayes Rule

• Likelihood = P(y|x)

• Prior probability = P(x)

• Posterior distribution P(x|y)

Xx

xPxyP

xPxyPyxP

)()|(

)()|(|

X = causeY = effect


30

Normal Distributions

)/)((2/1 22

2

1)(

xexp

Documents

240-650 Principles of Pattern Recognition