Linear Models for Classification -...

Discriminant Functions Generative Models Discriminative Models

Linear Models for ClassificationMachine Learning

Torsten Möller and Thomas Torsney-Weir

©Möller/Mori 1

Reading

• Chapter 4 of “Pattern Recognition and Machine Learning”by Bishop

©Möller/Mori 2

Classification

• goal in classification is to take input x and assign it to oneof K discrete classes Ck

• Ck typically disjoint (unique class membership)• input space divided into decision regions• boundaries: decision boundaries or decision surfaces• linear models: decision boundaries are hyperplanes:

linearly separable classes

©Möller/Mori 3

Classification: Hand-written Digit Recognition

!"#$%&'"()#*$#+ !,- )( !.,$#/+ +%((/#/01/! 233456 334

7/#()#-! 8/#9 */:: )0 "'%! /$!9 +$"$;$!/ +,/ ") "'/ :$1< )(

8$#%$"%)0 %0 :%&'"%0& =>?@ 2ABC D,!" -$</! %" ($!"/#56E'/ 7#)")"97/ !/:/1"%)0 $:&)#%"'- %! %::,!"#$"/+ %0 F%&6 GH6

C! !//0I 8%/*! $#/ $::)1$"/+ -$%0:9 ()# -)#/ 1)-7:/J1$"/&)#%/! *%"' '%&' *%"'%0 1:$!! 8$#%$;%:%"96 E'/ 1,#8/-$#</+ 3BK7#)") %0 F%&6 L !')*! "'/ %-7#)8/+ 1:$!!%(%1$"%)07/#()#-$01/ ,!%0& "'%! 7#)")"97/ !/:/1"%)0 !"#$"/&9 %0!"/$+)( /.,$::9K!7$1/+ 8%/*!6 M)"/ "'$" */ );"$%0 $ >6? 7/#1/0"

/##)# #$"/ *%"' $0 $8/#$&/ )( )0:9 (),# "*)K+%-/0!%)0$:

8%/*! ()# /$1' "'#//K+%-/0!%)0$: );D/1"I "'$0<! ") "'/

(:/J%;%:%"9 7#)8%+/+ ;9 "'/ -$"1'%0& $:&)#%"'-6

!"# $%&'() *+,-. */0+12.33. 4,3,5,6.

N,# 0/J" /J7/#%-/0" %08):8/! "'/ OAPQKR !'$7/ !%:'),/""/

+$"$;$!/I !7/1%(%1$::9 B)#/ PJ7/#%-/0" BPK3'$7/KG 7$#" SI

*'%1' -/$!,#/! 7/#()#-$01/ )( !%-%:$#%"9K;$!/+ #/"#%/8$:

=>T@6 E'/ +$"$;$!/ 1)0!%!"! )( GI?HH %-$&/!U RH !'$7/

1$"/&)#%/!I >H %-$&/! 7/# 1$"/&)#96 E'/ 7/#()#-$01/ %!

-/$!,#/+ ,!%0& "'/ !)K1$::/+ V;,::!/9/ "/!"IW %0 *'%1' /$1'

!"# $%%% &'()*(+&$,)* ,) -(&&%') ()(./*$* ()0 1(+2$)% $)&%..$3%)+%4 5,.6 784 ),6 784 (-'$. 7997

:;<6 #6 (== >? @AB C;DE=FDD;?;BG 1)$*& @BD@ G;<;@D HD;I< >HJ CB@A>G KLM >H@ >? "94999N6 &AB @BO@ FP>QB BFEA G;<;@ ;IG;EF@BD @AB BOFCR=B IHCPBJ

?>==>SBG PT @AB @JHB =FPB= FIG @AB FDD;<IBG =FPB=6

:;<6 U6 M0 >PVBE@ JBE><I;@;>I HD;I< @AB +,$.W79 GF@F DB@6 +>CRFJ;D>I >?@BD@ DB@ BJJ>J ?>J **04 *AFRB 0;D@FIEB K*0N4 FIG *AFRB 0;D@FIEB S;@A!!"#$%&$' RJ>@>@TRBD K*0WRJ>@>N QBJDHD IHCPBJ >? RJ>@>@TRB Q;BSD6 :>J**0 FIG *04 SB QFJ;BG @AB IHCPBJ >? RJ>@>@TRBD HI;?>JC=T ?>J F==>PVBE@D6 :>J *0WRJ>@>4 @AB IHCPBJ >? RJ>@>@TRBD RBJ >PVBE@ GBRBIGBG >I@AB S;@A;IW>PVBE@ QFJ;F@;>I FD SB== FD @AB PB@SBBIW>PVBE@ D;C;=FJ;@T6

:;<6 "96 -J>@>@TRB Q;BSD DB=BE@BG ?>J @S> G;??BJBI@ M0 >PVBE@D ?J>C @AB+,$. GF@F DB@ HD;I< @AB F=<>J;@AC GBDEJ;PBG ;I *BE@;>I !676 X;@A @A;DFRRJ>FEA4 Q;BSD FJB F==>EF@BG FGFR@;QB=T GBRBIG;I< >I @AB Q;DHF=E>CR=BO;@T >? FI >PVBE@ S;@A JBDRBE@ @> Q;BS;I< FI<=B6

ti = (0, 0, 0, 1, 0, 0, 0, 0, 0, 0)

• Each input vector classified into one of K discrete classes• Denote classes by Ck

• Represent input image as a vector xi ∈ R784.• We have target vector ti ∈ {0, 1}10• Given a training set {(x1, t1), . . . , (xN , tN )}, learning

problem is to construct a “good” function y(x) from these.• y : R784 → R10

©Möller/Mori 4

Generalized Linear Models

• Similar to previous chapter on linear models for regression,we will use a “linear” model for classification:

y(x) = f(wTx+ w0)

• This is called a generalized linear model• f(·) is a fixed non-linear function

• e.g.f(u) =

{1 if u ≥ 00 otherwise

• Decision boundary between classes will be linear functionof x

• Can also apply non-linearity to x, as in ϕi(x) for regression

©Möller/Mori 5

y(x) = f(wTx+ w0)

• e.g.f(u) =

©Möller/Mori 6

y(x) = f(wTx+ w0)

• e.g.f(u) =

©Möller/Mori 7

Outline

Discriminant Functions

Generative Models

Discriminative Models

©Möller/Mori 8

Outline

Generative Models

©Möller/Mori 9

Discriminant Functions with Two Classes

y(x)‖w‖

−w0‖w‖

y = 0y < 0

• Start with 2 class problem,ti ∈ {0, 1}

• Simple linear discriminant

y(x) = wTx+ w0

apply threshold function to getclassification

• Projection of x in w dir. is wTx||w||

©Möller/Mori 10

Multiple Classes

not C1

not C2

• A linear discriminant between two classes separates with ahyperplane

• How to use this for multiple classes?• One-versus-the-rest method: build K − 1 classifiers,

between Ck and all others• One-versus-one method: build K(K − 1)/2 classifiers,

between all pairs©Möller/Mori 11

Multiple Classes

not C1

not C2

Multiple Classes

not C1

not C2

Multiple Classes

not C1

not C2

Multiple Classes

not C1

not C2

Multiple Classes

not C1

not C2

Multiple Classes

• A solution is to build K linear functions:yk(x) = wT

k x+ wk0

assign x to class arg maxk yk(x)• Gives connected, convex decision regions

x̂ = λxA + (1− λ)xB

yk(x̂) = λyk(xA) + (1− λ)yk(xB)

⇒ yk(x̂) > yj(x̂),∀j ̸= k©Möller/Mori 17

Multiple Classes

• A solution is to build K linear functions:yk(x) = wT

k x+ wk0

assign x to class arg maxk yk(x)• Gives connected, convex decision regions

x̂ = λxA + (1− λ)xB

yk(x̂) = λyk(xA) + (1− λ)yk(xB)

⇒ yk(x̂) > yj(x̂),∀j ̸= k©Möller/Mori 18

Least Squares for Classification

• How do we learn the decision boundaries (wk, wk0)?• One approach is to use least squares, similar to regression• Find W to minimize squared error over all examples and

all components of the label vector:

E(W ) =1

N∑n=1

K∑k=1

(yk(xn)− tnk)2

• Some algebra, we get a solution using the pseudo-inverseas in regression

©Möller/Mori 19

E(W ) =1

N∑n=1

K∑k=1

(yk(xn)− tnk)2

©Möller/Mori 20

E(W ) =1

N∑n=1

K∑k=1

(yk(xn)− tnk)2

©Möller/Mori 21

Problems with Least Squares

−4 −2 0 2 4 6 8

• Looks okay... least squaresdecision boundary

• Similar to logistic regressiondecision boundary (more later)

• Gets worse by adding easypoints?!

• Why?• If target value is 1, points far

from boundary will have highvalue, say 10; this is a largeerror so the boundary is moved

©Möller/Mori 22

−4 −2 0 2 4 6 8

from boundary will have highvalue, say 10; this is a largeerror so the boundary is moved

©Möller/Mori 23

−4 −2 0 2 4 6 8

• Why?

• If target value is 1, points farfrom boundary will have highvalue, say 10; this is a largeerror so the boundary is moved

©Möller/Mori 24

−4 −2 0 2 4 6 8

from boundary will have highvalue, say 10; this is a largeerror so the boundary is moved©Möller/Mori 25

More Least Squares Problems

−6 −4 −2 0 2 4 6−6

• Easily separated by hyperplanes, but not found using leastsquares!

• We’ll address these problems later with better models• First, a look at a different criterion for linear discriminant

©Möller/Mori 26

Fisher’s Linear Discriminant

• The two-class linear discriminant acts as a projection:

y = wTx

≥ −w0

followed by a threshold• In which direction w should we project?• One which separates classes “well”

©Möller/Mori 27

y = wTx ≥ −w0

followed by a threshold

• In which direction w should we project?• One which separates classes “well”

©Möller/Mori 28

y = wTx ≥ −w0

followed by a threshold• In which direction w should we project?

• One which separates classes “well”

©Möller/Mori 29

Fisher’s Linear Discriminant• The two-class linear discriminant acts as a projection:

y = wTx ≥ −w0

followed by a threshold• In which direction w should we project?• One which separates classes “well”

−4 −2 0 2 4 6 8

©Möller/Mori 30

−2 2 6

• A natural idea would be to project in the direction of the lineconnecting class means

• However, problematic if classes have variance in thisdirection

• Fisher criterion: maximize ratio of inter-class separation(between) to intra-class variance (inside)

©Möller/Mori 31

−2 2 6

©Möller/Mori 32

−2 2 6

©Möller/Mori 33

−2 2 6

©Möller/Mori 34

Math time - FLD• Projection yn = wTxn

• Inter-class separation is distance between class means(good):

∑n∈Ck

• Intra-class variance (bad):

s2k =∑n∈Ck

(yn −mk)2

• Fisher criterion:

J(w) =(m2 −m1)

s21 + s22

maximize wrt w©Möller/Mori 35

∑n∈Ck

s2k =∑n∈Ck

(yn −mk)2

J(w) =(m2 −m1)

s21 + s22

∑n∈Ck

s2k =∑n∈Ck

(yn −mk)2

J(w) =(m2 −m1)

s21 + s22

∑n∈Ck

s2k =∑n∈Ck

(yn −mk)2

J(w) =(m2 −m1)

s21 + s22

Math time - FLD

J(w) =(m2 −m1)

s21 + s22=

Between-class covariance:SB = (m2 −m1)(m2 −m1)

Within-class covariance:SW =

∑n∈C1

(xn −m1)(xn −m1)T +

∑n∈C2

(xn −m2)(xn −m2)T

Lots of math:

w ∝ S−1W (m2 −m1)

If covariance SW is isotropic, reduces to class mean differencevector

©Möller/Mori 39

Math time - FLD

J(w) =(m2 −m1)

s21 + s22=

Between-class covariance:SB = (m2 −m1)(m2 −m1)

Within-class covariance:SW =

∑n∈C1

(xn −m1)(xn −m1)T +

∑n∈C2

(xn −m2)(xn −m2)T

Lots of math:

w ∝ S−1W (m2 −m1)

If covariance SW is isotropic, reduces to class mean differencevector

©Möller/Mori 40

FLD Summary

• FLD is a dimensionality reduction technique (more later inthe course)

• Criterion for choosing projection based on class labels• Still suffers from outliers (e.g. earlier least squares

example)

©Möller/Mori 41

Perceptrons

• Perceptrons is used to refer to many neural networkstructures

• The classic type is a fixed non-linear transformation ofinput, one layer of adaptive weights, and a threshold:

y(x) = f(wTϕ(x))

• Developed by Rosenblatt in the 50s• The main difference compared to the methods we’ve seen

so far is the learning algorithm

©Möller/Mori 42

Perceptron Learning

• Two class problem• For ease of notation, we will use t = 1 for class C1 and

t = −1 for class C2• We saw that squared error was problematic• Instead, we’d like to minimize the number of misclassified

examples• An example is mis-classified if wTϕ(xn)tn < 0• Perceptron criterion:

EP (w) = −∑n∈M

wTϕ(xn)tn

sum over mis-classified examples only

©Möller/Mori 43

Perceptron Learning

EP (w) = −∑n∈M

wTϕ(xn)tn

©Möller/Mori 44

Perceptron Learning

EP (w) = −∑n∈M

wTϕ(xn)tn

©Möller/Mori 45

Perceptron Learning

EP (w) = −∑n∈M

wTϕ(xn)tn

©Möller/Mori 46

Perceptron Learning Algorithm

• Minimize the error function using stochastic gradientdescent (gradient descent per example):

w(τ+1) = w(τ) − η∇EP (w)

= w(τ) + ηϕ(xn)tn︸︷︷︸if incorrect

• Iterate over all training examples, only change w if theexample is mis-classified

• Guaranteed to converge if data are linearly separable• Will not converge if not• May take many iterations• Sensitive to initialization

©Möller/Mori 47

w(τ+1) = w(τ) − η∇EP (w) = w(τ) + ηϕ(xn)tn︸︷︷︸if incorrect

©Möller/Mori 48

w(τ+1) = w(τ) − η∇EP (w) = w(τ) + ηϕ(xn)tn︸︷︷︸if incorrect

©Möller/Mori 49

Perceptron Learning Illustration

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

• Note there are many hyperplanes with 0 error• Support vector machines (in a few weeks) have a nice way

of choosing one

©Möller/Mori 50

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

of choosing one

©Möller/Mori 51

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

of choosing one

©Möller/Mori 52

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

of choosing one

©Möller/Mori 53

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

−1 −0.5 0 0.5 1−1

−0.5

of choosing one ©Möller/Mori 54

Limitations of Perceptrons

• Perceptrons can only solve linearly separable problems infeature space

• Same as the other models in this chapter• Canonical example of non-separable problem is X-OR

• Real datasets can look like this too

Expressiveness of perceptrons

Consider a perceptron with g = step function (Rosenblatt, 1957, 1960)

Can represent AND, OR, NOT, majority, etc.

Represents a linear separator in input space:

!jWjxj > 0 or W · x > 0

(a) (b) (c)

0 1 0 1

xor I 2I 1orI 1 I 2and I 1 I 2

Chapter 20 11

©Möller/Mori 55

Outline

Generative Models

©Möller/Mori 56

Generative vs. Discriminative

• Generative models• Can generate synthetic

example data• Perhaps accurate

classification is equivalent toaccurate synthesis

• e.g. vision and graphics• Tend to have more parameters• Require good model of class

distributions

• Discriminative models• Only usable for classification• Don’t solve a harder problem

than you need to• Tend to have fewer parameters• Require good model of

decision boundary

©Möller/Mori 57

Probabilistic Generative Models• Up to now we’ve looked at learning classification by

choosing parameters to minimize an error function• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck)which generates the data for each class

©Möller/Mori 58

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

©Möller/Mori 59

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

©Möller/Mori 60

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

©Möller/Mori 61

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

©Möller/Mori 62

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

©Möller/Mori 63

Probabilistic Generative Models - Example

• Let’s say we observe x which is the current temperature• Determine if we are in Vienna (C1) or Honolulu (C2)• Generative model:

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

• p(x|C1) is distribution over typical temperatures in Vienna• e.g. p(x|C1) = N (x; 10, 5)

• p(x|C2) is distribution over typical temperatures in Honolulu• e.g. p(x|C2) = N (x; 25, 5)

• Class priors p(C1) = 0.1, p(C2) = 0.9

• p(C1|x = 15) = 0.0484·0.10.0484·0.1+0.0108·0.9 ≈ 0.33

©Möller/Mori 64

Probabilistic Generative Models - Example

• Let’s say we observe x which is the current temperature• Determine if we are in Vienna (C1) or Honolulu (C2)• Generative model:

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

• p(x|C1) is distribution over typical temperatures in Vienna• e.g. p(x|C1) = N (x; 10, 5)

• p(x|C2) is distribution over typical temperatures in Honolulu• e.g. p(x|C2) = N (x; 25, 5)

• Class priors p(C1) = 0.1, p(C2) = 0.9

• p(C1|x = 15) = 0.0484·0.10.0484·0.1+0.0108·0.9 ≈ 0.33

©Möller/Mori 65

• We can write the classifier in another form

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

1 + exp(−a)≡ σ(a)

where a = ln p(x|C1)p(C1)p(x|C2)p(C2)

• This looks like gratuitous math, but if a takes a simple formthis is another generalized linear model which we havebeen studying

• Of course, we will see how such a simple forma = wTx+ w0 arises naturally

©Möller/Mori 66

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

1 + exp(−a)≡ σ(a)

©Möller/Mori 67

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

1 + exp(−a)≡ σ(a)

©Möller/Mori 68

Logistic Sigmoid

−5 0 50

• The function σ(a) = 11+exp(−a) is known as the logistic

sigmoid• It squashes the real axis down to [0, 1]

• It is continuous and differentiable• It avoids the problems encountered with the too correct

least-squares error fitting (later)

©Möller/Mori 69

Multi-class Extension

• There is a generalization of the logistic sigmoid to K > 2classes:

p(Ck|x) =p(x|Ck)p(Ck)∑j p(x|Cj)p(Cj)

=exp(ak)∑j exp(aj)

where ak = ln p(x|Ck)p(Ck)

• a. k. a. softmax function• If some ak ≫ aj , p(Ck|x) goes to 1

©Möller/Mori 70

Multi-class Extension

• There is a generalization of the logistic sigmoid to K > 2classes:

p(Ck|x) =p(x|Ck)p(Ck)∑j p(x|Cj)p(Cj)

=exp(ak)∑j exp(aj)

where ak = ln p(x|Ck)p(Ck)

• a. k. a. softmax function• If some ak ≫ aj , p(Ck|x) goes to 1

©Möller/Mori 71

Gaussian Class-Conditional Densities

• Back to that a in the logistic sigmoid for 2 classes• Let’s assume the class-conditional densities p(x|Ck) are

Gaussians, and have the same covariance matrix Σ:

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

2(x− µk)

TΣ−1(x− µk)

}• a takes a simple form:

a = ln p(x|C1)p(C1)p(x|C2)p(C2)

= wTx+ w0

• Note that quadratic terms xTΣ−1x cancel©Möller/Mori 72

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

2(x− µk)

TΣ−1(x− µk)

= wTx+ w0

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

2(x− µk)

TΣ−1(x− µk)

= wTx+ w0

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

2(x− µk)

TΣ−1(x− µk)

= wTx+ w0

Maximum Likelihood Learning

• We can fit the parameters to this model using maximumlikelihood

• Parameters are µ1, µ2, Σ−1, p(C1) ≡ π, p(C2) ≡ 1− π• Refer to as θ

• For a datapoint xn from class C1 (tn = 1):

p(xn, C1) = p(C1)p(xn|C1) = πN (xn|µ1,Σ)

p(xn, C2) = p(C2)p(xn|C2) = (1− π)N (xn|µ2,Σ)

• The likelihood of the training data is:

p(t|π,µ1,µ2,Σ) =N∏

[πN (xn|µ1,Σ)]tn [(1−π)N (xn|µ2,Σ)]1−tn

• As usual, ln is our friend:

ℓ(t; θ) =

N∑n=1

tn lnπ + (1− tn) ln(1− π)︸︷︷︸π

+ tn lnN1 + (1− tn) lnN2︸︷︷︸µ1,µ2,Σ

• Maximize for each separately

• The likelihood of the training data is:

p(t|π,µ1,µ2,Σ) =N∏

[πN (xn|µ1,Σ)]tn [(1−π)N (xn|µ2,Σ)]1−tn

• As usual, ln is our friend:

ℓ(t; θ) =

N∑n=1

tn lnπ + (1− tn) ln(1− π)︸︷︷︸π

+ tn lnN1 + (1− tn) lnN2︸︷︷︸µ1,µ2,Σ

• Maximize for each separately

Maximum Likelihood Learning - Class Priors

• Maximization with respect to the class priors parameter πis straightforward:

∂πℓ(t; θ) =

N∑n=1

− 1− tn1− π

⇒ π =N1

N1 +N2

• N1 and N2 are the number of training points in each class• Prior is simply the fraction of points in each class

∂πℓ(t; θ) =

N∑n=1

− 1− tn1− π

⇒ π =N1

N1 +N2

∂πℓ(t; θ) =

N∑n=1

− 1− tn1− π

⇒ π =N1

N1 +N2

Maximum Likelihood Learning - Gaussian Parameters• The other parameters can also be found in the same

fashion• Class means:

µ1 =1

N∑n=1

µ2 =1

N∑n=1

(1− tn)xn

• Means of training examples from each class• Shared covariance matrix:

Σ =N1

∑n∈C1

(xn−µ1)(xn−µ1)T+

∑n∈C2

(xn−µ2)(xn−µ2)T

Maximum Likelihood Learning - Gaussian Parameters• The other parameters can also be found in the same

fashion• Class means:

µ1 =1

N∑n=1

µ2 =1

N∑n=1

(1− tn)xn

• Means of training examples from each class• Shared covariance matrix:

Σ =N1

∑n∈C1

(xn−µ1)(xn−µ1)T+

∑n∈C2

(xn−µ2)(xn−µ2)T

Probabilistic Generative Models Summary

• Fitting Gaussian using ML criterion is sensitive to outliers• Simple linear form for a in logistic sigmoid occurs for more

than just Gaussian distributions• Arises for any distribution in the exponential family, a large

class of distributions

Outline

Generative Models

Probabilistic Discriminative Models

• Generative model made assumptions about form ofclass-conditional distributions (e.g. Gaussian)

• Resulted in logistic sigmoid of linear function of x• Discriminative model - explicitly use functional form

p(C1|x) =1

1 + exp(−wTx+ w0)

and find w directly• For the generative model we had 2M +M(M + 1)/2 + 1

parameters• M is dimensionality of x

• Discriminative model will have M + 1 parameters

p(C1|x) =1

1 + exp(−wTx+ w0)

p(C1|x) =1

1 + exp(−wTx+ w0)

Maximum Likelihood Learning - Discriminative Model• As usual we can use the maximum likelihood criterion for

learning

p(t|w) =

N∏n=1

ytnn {1− yn}1−tn ; where yn = p(C1|xn)

• Taking ln and derivative gives:

∇ℓ(w) =

N∑n=1

(tn − yn)xn

• This time no closed-form solution since yn = σ(wTx)

• Could use (stochastic) gradient descent• But there’s a better iterative technique

learning

p(t|w) =

N∏n=1

∇ℓ(w) =

N∑n=1

(tn − yn)xn

learning

p(t|w) =

N∏n=1

∇ℓ(w) =

N∑n=1

(tn − yn)xn

learning

p(t|w) =

N∏n=1

∇ℓ(w) =

N∑n=1

(tn − yn)xn

Iterative Reweighted Least Squares• Iterative reweighted least squares (IRLS) is a descent

method• As in gradient descent, start with an initial guess, improve it• Gradient descent - take a step (how large?) in the gradient

direction• IRLS is a special case of a Newton-Raphson method

• Approximate function using second-order Taylor expansion:

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

• Closed-form solution to minimize this is straight-forward:quadratic, derivatives linear

• In IRLS this second-order Taylor expansion ends up beinga weighted least-squares problem, as in the regressioncase from last week

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

Newton-Raphson484 9 Unconstrained minimization

PSfrag replacements

(x, f(x))

(x + !xnt, f(x + !xnt))

Figure 9.16 The function f (shown solid) and its second-order approximation!f at x (dashed). The Newton step !xnt is what must be added to x to give

the minimizer of !f .

9.5 Newton’s method

9.5.1 The Newton step

For x ! dom f , the vector

!xnt = "#2f(x)!1#f(x)

is called the Newton step (for f , at x). Positive definiteness of #2f(x) implies that

#f(x)T !xnt = "#f(x)T#2f(x)!1#f(x) < 0

unless #f(x) = 0, so the Newton step is a descent direction (unless x is optimal).The Newton step can be interpreted and motivated in several ways.

Minimizer of second-order approximation

The second-order Taylor approximation (or model) !f of f at x is

!f(x + v) = f(x) + #f(x)T v +1

2vT#2f(x)v, (9.28)

which is a convex quadratic function of v, and is minimized when v = !xnt. Thus,the Newton step !xnt is what should be added to the point x to minimize thesecond-order approximation of f at x. This is illustrated in figure 9.16.

This interpretation gives us some insight into the Newton step. If the functionf is quadratic, then x + !xnt is the exact minimizer of f . If the function f isnearly quadratic, intuition suggests that x + !xnt should be a very good estimateof the minimizer of f , i.e., x!. Since f is twice di"erentiable, the quadratic modelof f will be very accurate when x is near x!. It follows that when x is near x!,the point x + !xnt should be a very good estimate of x!. We will see that thisintuition is correct.

• Figure from Boyd and Vandenberghe, Convex Optimization

• Excellent reference, free for download onlinehttp://www.stanford.edu/~boyd/cvxbook/

Conclusion

• Readings: Ch. 4.1.1-4.1.4, 4.1.7, 4.2.1-4.2.2, 4.3.1-4.3.3• Generalized linear models y(x) = f(wTx+ w0)

• Threshold/max function for f(·)• Minimize with least squares• Fisher criterion - class separation• Perceptron criterion - mis-classified examples

• Probabilistic models: logistic sigmoid / softmax for f(·)• Generative model - assume class conditional densities in

exponential family; obtain sigmoid• Discriminative model - directly model posterior using

sigmoid (a. k. a. logistic regression, though classification)• Can learn either using maximum likelihood

• All of these models are limited to linear decisionboundaries in feature space

Linear Models for Classification -...

Documents

Machine Learning Torsten Möller - univie.ac.atvda.univie.ac.at/Teaching/ML/15s/LectureNotes/03_regression.pdf · Regression Basis Functions Loss Functions Weights Regularization

Linear Models for Classiﬁcation - univie.ac.atvda.univie.ac.at/Teaching/ML/15s/LectureNotes/04_classification.pdf · Linear Models for Classiﬁcation Bishop PRML Ch. 4 ... *+,-

3D Viewing - univie.ac.atvda.univie.ac.at/Teaching/Graphics/13w/LectureNotes/09_viewing.pdf · Perspective vs. parallel • Perspective projection: – Realistic, mimicking our human

Deubiquitinase Inhibition of 19S Regulatory Particles …...Small Molecule Therapeutics Deubiquitinase Inhibition of 19S Regulatory Particles by 4-Arylidene Curcumin Analog AC17 Causes

19S Macm Tor - Raincoast Books

Technical data PULSAR Thermal imaging scopes Quantum HD/LD 19S, 38S | Optics Trade

Indirect volume rendering - univie.ac.atvda.univie.ac.at/.../07_indirect_volume_rendering.pdf · – Surface is found by linear interpolation along cell edges – Uses gradients as

3D Transformations - univie.ac.atvda.univie.ac.at/Teaching/Graphics/17s/LectureNotes/05_3... · 2017. 2. 3. · • Afﬁne transformations • Use of homogeneous coordinates •

Usability Heuristics - univie.ac.atvda.univie.ac.at/Teaching/HCI/19s/LectureNotes/02... · VU Human Computer Interaction und Psychologie Human-Centered Design Process Human centered

Introduction to WebGL - univie.ac.atvda.univie.ac.at › ... › webgl_tutorial › webgl_tutorial.pdfIntroduction to WebGL Wolfgang Ost Foundations of Computer Graphics, WS 2018/19

Free Bus For Under 19s - Final Consultation PDF Version

Data+Dataset Types/Semantics Tasks - univie.ac.atvda.univie.ac.at/Teaching/Vis/14s/LectureNotes/04_data_types_and... · Data types • categorical (nominal) apples, oranges, bananas

VisuNet GXP RM-GXP1100-19S RM-GXP1200-19S PC-GXP1100 …

& NEO EM. 18gG DROMANA ROSEBUD UNDER 19s Ground …€¦ · U19s Coach: Matt Baker UNDER 19s Ground Location: Dromana Recreation Reserve - Beach Street, Dromana Club Colours: Green,

Spasticity in under 19s: management

Reaction of 7S and 19S Components of Immune Rabbit Antisera

Sedation in under 19s: using sedation for diagnostic and

model Ypc Paraflow Two-stage Direct-fired Absorption Chiller · Model YPC ParaFlow Two-Stage Direct-Fired Absorption Chiller-Heaters Style A Models YPC‑DF‑12SC through YPC‑DF‑19S

Model YPC Two-Stage Steam-Fired Absorption … 155.19-EG3 (1108) Model YPC Two-Stage Steam-Fired Absorption Chiller Style C Models YPC-ST-14SC through YPC-ST-19S 300 through 675 Tons

06 1 viewing 18w - univie.ac.atvda.univie.ac.at/Teaching/Graphics/19s/LectureNotes/06_1_viewing_18w.pdfPerspective vs. parallel • Perspective projection: – Realistic, mimicking