A New Unsupervised Competive Learning Algorithm for Vector Quantization

7/29/2019 A New Unsupervised Competive Learning Algorithm for Vector Quantization

1/5

Proceedings of the 9th International Conference onNeural Information Processing (ICONIPOP), Vol. 2Lip0Wang, JagathC. Rajapakse, KunihikoFukushima,Soo-Young Lee, andXin Y ao (Editors)

A NEW UNSUPERVISED COMPETIVELEARNING ALGORITHM FORVECTOR QUANTIZATION

Tzu-Chao Lin and Pao-Ta Yu*Departmentof Computer Science-and nformation Engineering

National Chung Cheng UniversityChiayi, Taiwan62107,R.O.C.

Phone: 886-5-272041 1 ext. 33104,E-mail: [email protected]

In this paper, a novel unsupervised competitive learningalgorithm, called the centroid neural network adaptiveresonance theory (CNN-ART) algorithm, is to beproposed to relieve the dependence on the initialcodewords of the codebook in contrast to the conventionalalgorithms with vector quantization in lossy imagecompression. The design of the CNN-ART algorithm ismainly based on the adaptive resonance theory (ART)structure, and then a gradient-descent based learning ruleis derived so that the CNN-ART algorithm does notrequire a predetermined schedule for learning rate. Theappropriate initial weights obtained kom the CNN-ARTalgorithm can be applied as an initial codebook of theLinde-Buzo-Gray (LBG) algorithm such that thecompression performance can be greatly improved.Inthispaper, the extensive simulations demonstrate that theCNN-ART algorithm does outperform other algorithmslike LBG, SOFM and DCL.

1. INTRODUCTIONThe design of a codebook is an essential process withvector quantization (VQ) techniques in lossy imagecompression and usually based on the mnimzation of anaverage distortion measure between the training vectorsand the codebook vectors. Several variations of vectorquantization have been successfully used in imagecompression [I]. The well-known k -means clusteringalgorithm is oneof the most popular competitive learningvector quantization schemes [2]. Although the k -means

algorithm is simple and intuitively appealing, it has someinevitable problems [3]. The first one is that the numberk of clusters must be pre-known and fixed. Anotherproblem is that the initial codebook strongly affects theperformance of the k -means algorithm. Still anotherproblem is that the algorithm may not converge towardsan optimal solution for k > 1. A variation of the k -means algorithm,known as the LBG (Linde, BUZO,Gray)algorithm, has been proposed@ [4]. The LBG algorithmstill suffers kom these problems as the k -meansalgorithm. The same thing goes for many of the so-calledcompetitive learning algorithms based on a gradientmethod associated with a neural network whose weightvectors represent the codewords. The self-organizingfeature map (SOFM) proposed by Kohonen has a strongconnection with the k -means algorithm [5 ] . During thetraining procedure, the SOFM algorithm finds a winnerneuron and updates the synaptic weights of both thewinner and its neighbors. Inorder to obtain the best resultsfiom SOFM, the updated neighborhood, the initiallearning rate and the total numberof iterations for a givenset of data should be chosen carefully. However, it isdifficult to determne an appropriate set of initial synapticweights (initial codebook) and the total number ofiterations corresponding to it before the algorithm is runsuch that the final learning performance can be acceptable.The differential competitive learning (DCL)algorithm, proposed by Kong and Kosko [6],providesconcepts of reward and punishment to the competitivelearning algorithm like the learning vector quantization(LVQ) proposed by Kohonen [ 5 ] . The significance ofDCL is that it allows punishment by using negativelearning gain in unsupervised learning. However, it is

* Corresponding authorThis work is supported by National Science Council of the Republicof China under GrantNSC90-2213-E-194-043

944


2/5

diEcult to determine an appropriate set of initial synapticweights, and a schedule for the optimal learning, rate asmentioned in the previous paragraph is still nowhere to befound.Based on the adaptive resonance theory (ART)structure, a new unsupervised competitive learningalgorithm, called the centroid neural network .adaptiveresonance theory (CNN-ART) algorithm, is proposed inthis paper. The CNN-ART algorithm can determrte howa

new training pattern should be classified into it clusterthrough a matching threshold called vigilance. Due to thissignificant property of the CNN-ART algorithm, theappropriate initial codebook can be easily obtained incontrast to the conventional algorithm. The CNN-ARTalgorithm derives a gradient-descent based learning rule,so one of the very advantageous features is that CNN-ART does not require a schedule for the learning rate. Asthe case with DCL, CNN-ART provides reward andpunishment for the winner and the loser respectively inaccordance with the learning rate. After iterations, theweights can converge fast towards the centroids ofclusters for the expected codebook size.In Section2, the design of CNN-ART is described indetails. In Section3, the computer simulations, the designof a codebook for image compression, are exhibitedFinally, conclusions are discussed n Section4.

2. A NEW COMPETITIVE L EAR (GALGOlUTElM- NN-ART2.1.The CNN-ART ArchitectureThe adaptive resonance theory (ART) algorithm proposedby Grossberg is a special type of neural network that canrealize classification without supervision [7]. Based on theART algorithm, we develop the centroid neural networkadaptive resonance theory (CNN-ART) algorithmin orderto design a codebook. The graphical representation of theCNN-ART architectureis shown in Figure 1.The CNN-ART network consists of an input layer and ;moutputlayer called MINNET that is a simple net for determningthe nearest cluster exemplar [7]. Each node in the inputlayer is connected directly to the neurons in the outputlayer. A synaptic prototype weight wp p=0,1, * ., ,that has the same dimension as the input data X, isassociated with each neuron in the output layer as shownin Figure 1. As a result, the set of synaptic weightswP ,p=0,1, - a , k, can be treated as a codebook. Inthe MINNET subnet, the number of neurons starts withone and grows by one each time until the desired

codebook size is reached; that is, the number of neurons isproportional to the size of the codebook. Due to thisproperty, the CNN-ART algorithm does not require theinitial values of clusters; that is, it is not necessary toselect any appropriate initial codewords in advance whenwe design a codebook. Each neuron of the h4IET isweighted and interconnected so as to maintain its ownvalue (+1) and attempt to be lateral inhibition ( - E ) forthe others, and the output value07 of the neuron j attime t is given in terms of the output values at time -1,by the relationship01= ;(q-&E@-),i=O,l,.-.,n; t21

it j

whereOy =(Ixi-wj I2,/3 7 i f p ={

and n is the current number of output layer neurons,1nE >- The process of the MINNET is repeated until

only one output OJIremains to be negative with all theothers being 0, and that is when the minimum input isfound. In other words, the competitive output layer is awinner-take-all type of net. We can consider themnimum Euclidean distance between input vector x i andwp p=0,1, , , n the CNN-ART algorithm.2.2. Learning Rules in CNN-ARTWhen the cluster j has hdJ members and x , ( j )denotes the input data vector X, in the cluster j for oneinstance, the synaptic weights for a given output neuronshould be chosen in a way to minimize the total Euclideandistance in L , norm from the input vectors in its cluster

MI[SI:

wJ =mi& I IXl(i>-w112-By the following Theorem 1, the local minimum can beachieved; that is, the centroid of data in a cluster is thesolution.

w ,=o

945


3/5

[Theorem 11 The centroid of data in a cluster is thesolution that gives the mnimum Euclidean distance in L ,nom[si,~91.Inthe CNN-ART algorithm, according to Theorem1, thecentroid of every cluster is computed by the followingequation:

1 M Jw. =--Cxi( j ) .A1v1 . ;=

In this paper, the CNN-ART algorithm provides rewardand punishment to the competitive learning rules. Whenan input data vector Xi is applied to the network at time t,the CNN-ART algorithm rewards the winner neuron j(updates its weights w? : the weights of a winner neuronin the current iteration are updated only when the winnerneuron does notwinthe data xi in the previous iteration).The adaptive equationwy) can be writtenas follows:

J

(1-1)M j xwj XIw( . ?= +J M j+l M j+l

. jM j +1

At the same time, the CNN-ART algorithm will punish theloser neuron JJ (update its weights w: : the weights ofthe loser neuron in the current iteration are updated onlywhen the loser neuron win the data xi in the previousiteration). The adaptive equation w:) can be written asfollows:

[x; -Wy ] .w- -~M y-11

From Equations (1) and (2), we obtain that CNN-ARTadapts its weights according to the centroids to achieve thelocalmnimumEuclidean distance; that is, the mechanismis a greedy approach, so it can minimize the totalEuclidean distance for designing a codebook. However, itdoes not guarantee the convergence towards the global

mnimum. However, according to the following adaptivevector quantization (AVQ) convergence theorem [9],AVQ can be viewed as a way to learn prototype vectorpatterns of real numbers; it can guarantee that averagesynaptic vectors converge to centroids exponentiallyquickly, so the CNN-ART algorithm makes theconvergence to the centroids of clusters much faster thanconventional learning algorithms.[AVQ Convergence Theorem] Competitive synapticvectors converge exponentially quickly to pattern-classcentroids.From Equations (1) and (2), a gradient-descent basedlearning rule is derived. The learning rate can besummarized as

1 if j isa winner,if y is a loser,( t)= (-1)- (3)1M y -1

0, otherwise.1hrough Equation (3), CNN-ART provides reward andpunishment to the competitive learning algorithm by usingpositive and negative learning gain respectively.Furthermore, the learning rate is adaptive according to thenumber of members in the winner or loser cluster, soCNN-ART does not require a predetermned schedule.

2.3. The CNN-ART AlgorithmInthe CNN-ART competitive learning algorithm, at first,an input training vector xo is selected as the centroid ofthe first cluster, and then the next input training vector iscompared to the fist cluster centroid. It is classified to thefirst cluster if its Euclidean distance is smaller than thevalue of vigilance. Otherwise, it forms the centroid of anew cluster. This process-is repeated for all the traininginput vectors. The following steps show a pseudo code ofthe CNN-ART algorithm.Step 1: [Initialize]

Set CodebookSizek ,Total-input-vectors N ;n =0;w0=xo; //n :the number of neurons.Step2: [Set vigilance] Select vigilance;Step3: P o oop for designing a codebook]

For (i =0;i I N ; i++)Repeat(a) [Calculate Euclidean distances]

946


4/5

Apply an input vector xi to the network,and calculate Euclidean distance between xiand existing clusters.Find the smallest Euclidean distanced .(b) pecide the winner neuron j ]

(c)[Test tolerance]If (dj>vigilance)AND ( nvigilance) AND ( n


5/5

descent based learning rule is derived to provide rewardand punishment to the neuron in the process so that theCNN-ART algorithm does not require a predeterminedschedule for learning rate. Furthermore, CNN-ARTadaptively learns the training vectors and converges muchfaster than conventional learning algorithms, and it doesnot require the total number of iterations in advance. Dueto the significant vigilance property of the CNN-ARTalgorithm, the appropriate initial weights obtained fiomthe CNN-ART algorithm can be applied as an initialcodebook to the Linde-Buzo-Gray (LBG) algorithm suchthat the compression performance can be greatly improved.Moreover, experimental results have also demonstratedthat CNN-ART for image compression outperforms othercompetitive earning algorithms.

43ori"LBG

CNN-AKr-LBG

5. REFERENCES

h a Peppers0.5 0.5625 0625 0.5 0.5625 0.625

29.23 30.41 32.14 29.28 30.58 32.3129.95 31.92 36.21 30.16 32.25 35.50

[l] N. M. Nasrabadi and Y. Feng, Image compressionusing address-vector quantization, IEEE Trans. onCommunication,pp. 2166-2173, 1990.

Algorithms

[2] A. Gersho andR. M. Gray, Vector Quantization andSignal Compression, MA: Kluwer Academic, Boston,1992.

Bitsper pixd (b.p.p.)0.5625 I 0.625 I 0.6875

[3]Lei Xu, Bayesian Ying-Yang machine, clustering andnumber of clusters, Pattern Recognition Letters, 18, pp.1167-1178, 1997.

LBG I 27.59

[4] Y. Linde,A. Buzo andR. M. Gray,An algorithm forvector quantizatizer design, IEEE Trans. onCommunication,28, pp. 84-95, 1980.

27.96 I 28.28

[5] T. Kohonen, Self-organization and AssociativeMemory, Germany: Springer-Velag, Berlin, third edition,1989.

DCL I 29.98

[6] S. Kong and B. Kosko, Differential competitivelearning for centroid estimation and phoneme recognition,IEEE Trans. on Neural Networks, pp. 118-124,1991.

30.39 I 30.74[7] Y. H. Pao, Adaptive Pattern Recognition and NeuralNetworks,AddisonWesleyReading,MA,1989.[8] Dong-Chul Park, Centroid neural network forunsupervised competitive learning, IEEE Trans. onNeural Networks, pp. 520-528, March 2000.

Lateral nhibum Createa newnemm$ynamically\

MINNET

_ '' YY Y YP x z x 2 1,'x , = [x ; , x : , x:; , x:] '

Figure 1 The architectureof CNN-ART.

30.530$$ 29.52 2928.528 0 20 40 60

Number of Iterations

& BG+NN-ART

Figure 2: Comparison inPSNR ( dB)between CNN-ARTand LBG with different numbers of iterations.

CNN-AKr I 29.97 I 31.97 1 36.63 I 30.18 I 32.33 I 3621

I I

[9] B. Kosko, Neural Networh and Fuzzy System: ALlynamc Systems Approach to Machine Intelligence,Prentice-Hall, Englewood Cliffs,NJ , 1991.

948

Documents

A New Unsupervised Competive Learning Algorithm for Vector Quantization