1

1) Unsupervised Learning2) K-means clustering3) The EM Algorithm4) Competetive Learning &

SOM Networks

IES 511

Machine Learning Dr. Türker İnce

(Lecture notes by Prof. T. M. Mitchell, Machine Learning course at CMU & Prof. E. Alpaydın, Introduction to Machine Learning, MIT Press, 2004.)

2

3

4

5

6

7

8

9

10

EM for two Gaussian Mixture

11

Simple Competitive Learning• Unsupervised learning• Goal:

– Learn to form classes/clusters of examplers/sample patterns according to similarities of these examplers.

– Patterns in a cluster would have similar features– No prior knowledge as what features are important for

classification, and how many classes are there.• Architecture:

– Output nodes: Y_1,…….Y_m, representing the m classes

– They are competitors

(Winner-Take-All algorithm)

12

• Training: – Train the network such that the weight vector wj associated

with jth output node becomes the representative vector of a class of similar input patterns.

– Initially all weights are randomly assigned– Two phase unsupervised learning

• competing phase:– apply an input vector randomly chosen from sample set.– compute output for all output nodes: – determine the winner among all output nodes (winner is

not given in training samples so this is unsupervised) • rewarding phase:

– the winner is rewarded by updating its weights to be closer to (weights associated with all other output nodes are not updated: kind of WTA)

• repeat the two phases many times (and gradually reduce the learning rate) until all weights are stabilized.

*jjlj wio

li

li*j

w

13

• Weight update: – Method 1: Method 2:

In each method, is moved closer to il

– Normalize the weight vector to unit length after it is updated

– Sample input vectors are also normalized

– Distance

)( jlj wiw lj iw

jjj www

jjj www /

wj

il

il – wj

η (il - wj)

wj + η(il - wj)

jw

il

wjwj + ηil

ηil

il + wj

lll iii /

i ijiljljl wiwiwi 2,,2)(

14

• wj is moving to the center of a cluster of sample vectors after repeated weight updates – Node j wins for three training

samples: i1 , i2 and i3

– Initial weight vector wj(0)– After successively trained

by i1 , i2 and i3 ,the weight vector

changes to wj(1),

wj(2), and wj(3),

i2

i1

i3

wj(0)

wj(1)

wj(2)

wj(3)

15

16

17

• Input vectors are uniformly distributed in the region, and randomly drawn from the region

• Weight vectors are initially drawn from the same region randomly (not necessarily uniformly)

• Weight vectors become ordered according to the given topology (neighborhood), at the end of training

SOM Examples

18

1-D Lattice

19

2-D Lattice

20

WEBSOM

• See http://www.cis.hut.fi/research/som-research.

• WEBSOM: http://websom.hut.fi/websom/Self-organizing maps of document collections.

– Goal:Automatically order and organize arbitrary

free-form textual document collections to enable their easier browsing and exploration.

– Reference paper for next slides:• S. Kaki et al. Statistical aspects of the WEBSOM system in

organizing document collections, Computing Science and Statistics 29:281-290, 1998.

21

WEBSOM

All words of document are mapped into the word category map

Histogram of “hits” on it is formed

Self-organizing map.Largest experiments have used:• word-category map

315 neurons with 270 inputs each

• Document-map 104040 neurons with 315 inputs each

Self-organizing semantic map.15x21 neurons

Interrelated words that have similar contexts appear close to each other on the map

• Training done with 1124134 documents

22

23

Word histogram

24

Word categories

25

A map of documents