10_1_compet

8/2/2019 10_1_compet

1/26

Self Organization:

Competitive Learning

CS/CMPE 537Neural Networks

8/2/2019 10_1_compet

2/26

CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 2

Introduction

Competition for limited resources is a way of

diversifying and optimizing a distributed system E.g. a sports tournament, evolution theory

Local competition leads to optimization at local and

global levels without the need for global control to

assign system resources

Neurons compete among themselves to decide the one

(winner-takes-all) or ones that are activated

Through competitive learning the output neurons form atopological map unto the input neurons

Sefl-organizing feature maps, where the networks learns the

input and output maps in a geometrical sense

8/2/2019 10_1_compet

3/26


Competition

Competition is intrinsically a nonlinear operation

Two types of competition

Hard competition means that only one neuron wins the

resources

Soft competition means that there is a clear winner, but its

neighbors also share a small percentage of the system

resources

8/2/2019 10_1_compet

4/26


Winner-Takes-All Network

Inputs xi (i = 1, p); output is

yi = 1 if neuron i wins the competition

yi = 0 otherwise

The winning criterion can be any rule, as long as one clear

winner is found (e.g. largest output, smallest output)

The winner-takes-all network has one layer with p

neurons, self-excitatory connections, and lateral

inhibitory connections to all other neurons

When a input is applied, the network evolves until the outputof the winning neuron tends to 1 while the outputs of the

other neurons tends to 0

Winner-takes-all network acts as a selector

8/2/2019 10_1_compet

5/26


Instar Network (1)

Instar network is made up of McCulloch-Pitts neurons

Instar neuron is able to recognize a single vectorpattern based on a modified Hebbian rule that

automatically accomplishes weight normalization for

normalized inputs

p

p

8/2/2019 10_1_compet

6/26


Instar Network (2)

Learning rule

wji(n + 1) = wji(n) + yj(n)[xi(n)wji(n)]

y is the hard-limiter output of either 0 or 1

The rule modifies the weights for neuron j if neuron j is

active (wins). Otherwise, the weights remain unchanged.

This rule is similar to Ojas rule, except that the weights are

not multiplied by the output

8/2/2019 10_1_compet

7/26CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 7

Instar Network (3)

The effect of the instar rule is to move the weight

closer to the input in a straight line proportional to

A trained instar neuron provides a response of 1 fordata samples located near the training pattern (the bias

controls the size of the neighborhood)

8/2/2019 10_1_compet

8/26

8/2/2019 10_1_compet


Criterion for Competition (1)

What criterion should be used to determine the winning neuron?

The criterion should have some geometrical relevance Inner product

A neuron wins if the input is closest to the neuron (i.e. its weights)

However, this only works when the input is normalized

Winning neuron is the one with the largest inner product

8/2/2019 10_1_compet


8/2/2019 10_1_compet


Criterion for Competition (2)

Euclidean distance

||xw|| = [i (xiwi)2]1/2Winning neuron is the one with the smallest Euclidean

distance

Less efficient than inner product

L1 norm or Manhattan distance

|xw| = i (xiwi)

Winning neuron is the one with the smallest L1 norm

8/2/2019 10_1_compet


Clustering

The competitive rule allows a single-layer linear

network to group and represent data samples that lie ina neighborhood of the input space

Each neighborhood is represented by a single output neuron

The weights of the neuron represent the center of the cluster

Thus, a single-layer linear competitive network performs

clustering

Clustering is a divide and conquer process where an

input space is divided into groups of proximal patterns Vector quantizationan example of clustering

8/2/2019 10_1_compet


k-Means Clustering Algorithm (1)

Classical algorithm for data clustering

Cluster data vectors xi (i = 1, N) into k cluster Ci (i = 1,k) such that the distance between the vectors and their

respective cluster centers ti (i = 1, k) is minimized

Algorithm

Randomly assign vectors to class Ci

Find center of each cluster Ci

ti = (1/Ni) nCixn

Reassign the vectors to their nearest cluster, i.e., assign

vector i to cluster k if ||xitk||2 is minimum for all k

Repeat the last two steps until no changes occur

8/2/2019 10_1_compet


k-Means Clustering Algorithm (2)

Performance measure or function

J = i=1knCi ||xnti||2 This measure is minimized in the k-means algorithm

Estimating J with its instantaneous value, then takingderivative wrt ti

J(n)/ti = -[x(n)ti(n)]

and, the change becomes (to minimize J)

ti(n) = [x(n)ti(n)]

This is identical to the competitive update rule, when tiis replaced by wi

Thus, the single-layer linear competitive network performs k-means clustering of the input data

8/2/2019 10_1_compet


k-Means Clustering - Example

Cluster [0, 1, 3, 6, 8. 9, 10, 11] into 2 groups. N = 8, k

= 2 C1 = [0, 1, 3, 6]; C2 = [8, 9, 10, 11]

t1 = 10/4 = 2.5; t2 = 38/4 = 9.5

C1 = [0, 1, 3]; C2 = [6, 8, 9, 10, 11] t1 = 4/3 = 1.33; t2 = 44/5 = 8.8

C1 = [0, 1, 3]; C2 = [6, 8, 9, 10, 11]

No change. Thus, these are the final groupings and clusters

8/2/2019 10_1_compet

16/26

8/2/2019 10_1_compet


Soft Competition

Hard competition = winner-takes-all. That is, the

winner neuron is active while the rest are inactive Soft competition = winner-shares-reward. That is, the

winning neuron activates its neighboring neurons as

well

The winning neuron has the highest output, while its

neighboring neurons outputs decrease with increase in

distance

Consequently, weights of all active neurons are modified, not

just that of the winning neuron

Soft competition is common in biological neural

systems

8/2/2019 10_1_compet


Kohonen Self-Organizing Maps (1)

Kohonen SOMs are single-layer competitive networks

with a soft competition rule They perform a mapping from a continuous input space

to a discrete output space, preserving the topological

properties of the input

Topological map

points close to each other in the input space are mapped to

the same or neighboring neurons in the output space

There is topological ordering or spatial significance to theneurons in the output and input layer

SO

8/2/2019 10_1_compet


Kohonen SOMs (2)

K h SOM ( )

8/2/2019 10_1_compet


Kohonen SOMs (3)

1-D SOM has output neurons arranged as a string, with

each neuron having two adjacent neighbors 2-D SOM has output neurons arranged as an 2-D array,

with more flexible definition of neighborhoods

K h L i Al i h (1)

8/2/2019 10_1_compet


Kohonen Learning Algorithm (1)

Competitive rule

wj(n + 1) = wj(n) + hj,i*(n) (n) [x(n)wj(n)] hj,j* = neighborhood function for winning neuron i*

A common neighboring function is the Gaussian

hj,i*(n) = exp(- dj,i*2

/ 2(n)2

)

K h L i Al i h (2)

8/2/2019 10_1_compet

22/26


Kohonen Learning Algorithm (2)

Initialization: Choose random values for the initial

weight vectors. Make sure the weights for each neuronare different and of small magnitude

Sampling and similarity matching: select an input

vector x. Find the best-matching (winning) neuron at

time n, using the minimum-distance Euclidean

criterion.

Updating: adjust the weight vectors using the Kohonen

learning rule Continuation: Repeat the previous two steps until no

noticeable changes occur in the feature map are

observed

E l (1)

8/2/2019 10_1_compet

23/26


Example (1)

Create a 1-D feature map of a 2-D input space. The

input vectors x lie on a unit circle. No. of inputs = 2 (defined by the problem, 2-D)

No. of outputs = 20 (our choice, 1-D linear array or

lattice) Randomly initialize the weights so that each neuron has

a different weight (in this example weights are

initialized along the x-axis, i.e, the y component is

zero) See the MATLAB code for details

E l (2)

8/2/2019 10_1_compet

24/26


Example (2)

Consider the input x = [0.7071 0.7071]T

Assume the current weights areW = [ 00:19] = [0 0; 0 1; 0 2; 0 3;;0 19]

Winning neuron is the one with smallest norm from theinput vector

Winning neuron is 2 (weight vector w2 = [0 1]T)

Norm = (0.70712 + 0.29292)1/2 = 0.7674 (this is the smallestdistance from the weight vectors)

Update of weights

wj(n + 1) = wj(n) + hj,j*(n) (n) [x(n)wj(n)]

Assume Gaussian neighborhood function with variance of0.5

E l (3)

8/2/2019 10_1_compet

25/26


Example (3)

Neuron 1

w1(n + 1) = w1(n) + h1,2*(n) (n) [x(n)w1(n)]

= [0 0] + 0.3679([0.7071 0.7071][0 0])

= [0.2601 0.2601]

Neuron 2 (winning neuron)w2(n + 1) = w2(n) + h2,2*(n) (n) [x(n)w2(n)]

= [0 1] + (1)([0.7071 0.7071][0 1])

= [0.7071 0.7071]

E l (4)

8/2/2019 10_1_compet

26/26

Example (4)

Neuron 3

w3(n + 1) = w3(n) + h3,2*(n) (n) [x(n)w3(n)]

= [0 2] + 0.3679([0.7071 0.7071][0 2])

= [0.2601 1.5243]

Neuron 4w4(n + 1) = w4(n) + h4,2*(n) (n) [x(n)w4(n)]

= [0 3] + 0.0183([0.7071 0.7071][0 3])

= [0.0129 2.9580] Neuron 5 and beyond

There update will be negligible because the Gaussian tends to

zero with increasing distance

Documents

10_1_compet