View
240
Download
0
Category
Preview:
Citation preview
Linear Discriminant Functions Linear Discriminant Functions
Discriminant Functions
Least Squares Method
Fisher’s Linear Discriminant
Probabilistic Generative Models
Linear Discriminant FunctionsLinear Discriminant Functions
A discriminant function is a linear combination of the components of x:
g(x) = wt x + w0
where • wt is the weight vector• w0 is the bias or threshold weight
For the two-class problem we can use the following decision rule:
Decide c1 if g(x) > 0 and c2 if g(x) < 0.
For the general case we will have one discriminant function for each class.
Figure 5.1Figure 5.1
The Normal Vector wThe Normal Vector w
The hyperplane H divides the feature space into two regions: Region R1 for class c1 and Region R2 for class c2.
For two points x1 and x2 on the decision boundary:
wt x1 + w0 = wt x2 + w0
which means
wt (x1 – x2) = 0
Thus w is normal to any vector in the hyperplane.
Geometry for Linear ModelsGeometry for Linear Models
The Problem with Multiple ClassesThe Problem with Multiple Classes
How do we use a linear discriminant when we have more than two classes?
There are two approaches:
1. Learn one discriminant function for each class 2. Learn a discriminant function for all pairs of classes
If c is the number of classes, in the first case we have c functions andin the second case we have c(c-1) / 2 functions.
In both cases we are left with ambiguous regions.
Figure 5.3Figure 5.3
Linear MachinesLinear Machines
To avoid the problem of ambiguous regions we can use linear machines:
We define c linear discriminant functions and choose the onewith highest value for a given x.
gk(x) = wkt x + wk0 k = 1, …, c
In this case the decision regions are convex and thus are limitedin flexibility and accuracy.
Figure 5.4Figure 5.4
Generalized Linear Discriminant FunctionsGeneralized Linear Discriminant Functions
A linear discriminant function g(x) can be written as:
g(x) = w0 + Σi wixi i = 1, …, d (d is the number of features).
We could add additional terms to obtain a quadratic discriminant function:
g(x) = w0 + Σi wixi + Σi Σj wij xixj
The quadratic discriminant function introduces d(d-1)/2 coefficientscorresponding to the products of attributes. The surfaces are thus morecomplicated (hyperquadric surfaces).
Generalized Linear Discriminant FunctionsGeneralized Linear Discriminant Functions
We could even add more terms wijk xi xj xk and obtain the classof polynomial discriminant functions. The generalized form is
g(x) = Σi wi yi(x)
g(x) = wt y
Where the summation goes over all functions yi(x).The yi(x) functions are called the phi or φ functions.The function is now linear on the yi(x).
The functions map a d-dimensional x-space into a d’ dimensionaly-space.
Example: g(x) = w1 + w2x + w3x2 y = (1 x x2 ) t
Figure 5.5Figure 5.5
Mapping to other spaceMapping to other space
Mapping from x to y:
If x follows certain probability distribution, the corresponding distribution on the new space will be degenerate.
Even with simple functions for y, the decision surfaces in x can be quite complicated.
With a larger space we have more degrees of freedom (parameters to specify). Thus, we need larger samples.
Figure 5.6Figure 5.6
Linear Discriminant Functions Linear Discriminant Functions
Discriminant Functions
Least Squares Method
Fisher’s Linear Discriminant
Probabilistic Generative Models
Least SquaresLeast Squares
And how do we compute y(x)? How do we find the values of w0, w1, w2, …, wd?
We can simply find the w that minimizes an error function E(w):
E(w) = ½ Σ (g(x,w) – t)2
Problems: Lacks robustness; assumes target vector is Gaussian.
Least SquaresLeast Squares
Least squares vs Logistic regression
Least SquaresLeast Squares
Least squares Logistic regression
Carl Friedrich GaussCarl Friedrich Gauss German 1777 – 1855German 1777 – 1855
Carl F. Gauss is known as the scientist whodeveloped the idea of “least squares method”.
He came up with this idea at the early age of eighteen years old!
He is considered one of the greatest mathematicianof all times.
He made major discoveries in geometry, number theory, magnetism,astronomy, among other fields.
Anecdotes: solved a problem posed by his teacher (sum all integers from 1 – 100).
Linear Discriminant Functions Linear Discriminant Functions
Discriminant Functions
Least Squares Method
Fisher’s Linear Discriminant
Probabilistic Generative Models
Fisher’s Linear DiscriminantFisher’s Linear Discriminant
The idea is to project the data on one single dimension.
We choose a projection that maximizes class separation,
and minimizes the variance within each class.
Find w that maximizes an error function
J(w) = (m2 – m1)2 / s12 + s22
J(w) = wT SB w / wT Sw w
Where SB is the between-class covariance matrixAnd SW is the within class covariance matrix
Fisher’s Linear DiscriminantFisher’s Linear DiscriminantSB = (m2 – m1) (m2 – m1)T
SW = ∑ (x – m1)(x – m1)T + ∑(x – m2)(x – m2)T
Fisher’s Linear DiscriminantFisher’s Linear Discriminant
Wrong Right
Linear Discriminant Functions Linear Discriminant Functions
Discriminant Functions
Least Squares Method
Fisher’s Linear Discriminant
Probabilistic Generative Models
Probabilistic Generative ModelsProbabilistic Generative Models
We first compute g(x) = w1x1 + w2x2 + … + wdxd + w0
But instead we wish to have P(Ck|x).
To get conditional probabilities we compute a logistic function:
L(g(x)) = 1 / ( 1 + exp(-g(x)) )
And L(y) = P(Ck|x) if the two classes can be modeled as a Gaussian distribution with equal covariance matrix.
Recommended