8/13/2019 Particle Swarm Optimisation for Evolving Artificial Neural Network

1/4

Particle Swarm Optimisation for Evolving Artificial Neural NetworkChunkai Zhang, Huihe ShaoInstitute of Automation, Shanghai Jiao TongUniversityShanghai, 200030, P. R. China

AbstractThe information processing capability of artificial neuralnetworks (ANNs) is closely related to its architecture andweights. T he paper describes a new evolutionary system forevolving artificial neural networks, which is based on theparticle swarm optimisation PSO) algorithm. Both thearchitecture and the weights o f ANNS are adaptivelyadjusted according to the quality of the neural network.This process is repeated until the best ANNs is accepted orthe maximum number o f generations has been reached. Anda strategy of evolving added nodes and a partial trainingalgorithm are used to maintain a close behavioural linkbetween the parents and their offspring. This system hasbeen tested on two real problems in the medical domain.The results show that ANNS evolved by PSONN havegood accuracy and generalisation ability.

1 IntroductionThere have been many attempts to design artificial neuralnetworks architectures automatically, such as variousconstructive and pruning algorithms [1 2]. However, Suchstructural hill climbing methods are susceptible tobecoming trapped at structure local optima, and the resultdepends on initial network architectures. [3].Design of a near optimal ANNs architecture can beformulated s a search problem in the architecture spacewhere each point represents an architecture. Miller et al.indicated that evolutionary algorithms are better candidatesfor searching the architecture space than those constructiveand pruning algorithms mentioned above [4]. GAS was usedto evolve ANNs, but the evolution of ANNs architecturesoften suffers from the permutation problem [5]. Thisproblem not only makes the evolution inefficient, but alsomakes crossover operators more difficult to produce highlyfit offspring.This paper proposes an evolutionary system for evolvingfeed-forward ANNs called PSONN, which combines thearchitectural evolution with weight learning. And a strategyof evolving added nodes a nd a partial training algorithm areused to maintain the behavioural link between a parent andits offspring to prevent destruction of the behaviour alreadylearned by the parent Its purpose is to produce very

Yu LiInstitute of LSI, Shanghai Jiao Tong UniversityShanghai, 200030, P. R. China

compact ANNs with good generalization ability.The rest of this paper is organised as follows. Section 2describes the PSO algorithm and the PSONN system, andthe motivations on how to evolve the entire ANNs. Section3 presents experimental results on PSONN in the medicaldomain. The paper is concluded in Section 4.

2 PSONN system2 1 Particle sw arm optimisation PSO)In PSONN system, the learning algorithm is the PSOalgorithm. PSO is a population based optimisationalgorithm that is motivated from the simulation of socialbehaviour [ 6 ] .Each individual in PSO flies in the searchspace with a velocity that is dynamically adjusted accordingto its own flying experience and its companions flyingexperience. Compared with other evolutionary algorithms,such as GAS, PSO algorithm possesses some attractiveproperties such as memory and constructive cooperationbetween individuals, so it has more chance to fly into thebetter solution areas more quickly and discover reasonablequality solution much faster. In this paper we propose animproved PSO algorithm, which is as follows:

Initialise positions Pesentx and associated velocity ofall individuals (potential solutions) in the populationrandomly in the D imension space.Evaluate the fitness value of all individuals.Compare the PBEST[] of every individual with itscurrent fitness value. If the current fitness value isbetter, assign the current fitness value to PBEST[] andassign the current coordinates to PBESTx[][d]coordinates. Here PBEST[] represents the best fitnessvalue of the nth individual, PBESTxg[d] represents thedth compon ent of a n individual.Determine the current best fitness value in the entirepopulation and its coordinates. If the current bestfitness value is better than the GBEST, then assign thecurrent best fitness value to GBEST and assign thecurrent coordinates to GBESTx[d].Change velocities and positions using the followingrules:

0-7803-6583-6/00/ 10.00 IEEE 2487

8/13/2019 Particle Swarm Optimisation for Evolving Artificial Neural Network

2/4

v[l[dI= W [ l [ d l+Cl*r nd PBESTd] d] Pesentxf][ d ]+C 2 rand (GBESTdd] -Peserrbq] [ d ] )PesentA] [d]= Pesenbq][d]+v[] [d]where C1= C2 = 2.0, t and K are the number of currentiterations and total generation. The balance between theglobal and local search is adjusted through the parameter

Repeat step 2 - 6) until a stop criterion is satisfied or apredefined num ber o f iterations is completed.

W=W + WOW,)[l- ]

W E WO,W,).6

Because there are not selection and crossover operator inPSO, each individual in an original population has acorresponding partner in a new population. It can avoid thepremature convergence and stagnation in GAS to someextent.2.2 PSONN systemIn PSONN all the data sets are partitioned into three sets: atraining set, a validation set, and a testing set. Such use ofthe validation set improves the generalisation ability ofevolved ANNs and introduces little overhead incomputation time.The major steps of PSONN can be described s follows:

Generate an initial population of M networks. Thenumber of hidden nodes and the initial connectiondensity for each network are uniformly generated atrandom within certain ranges. The random initialweights are uniformly distributed inside a small range.Using the PT algorithm to train each networks nodeson the training set (this process can evaluate the qualityof a given network a rchitecture).a. Choose a network as a parent network, thenrandomly generate N-1 initial networks s apopulation where each networks initial weightsuniformly generated at random within certainranges, but its architectures are the same as theparents architecture, and then the parent networkis added into the population.b. Employ the PSO algorithm to evolve thispopulation for a certain number of epochs. Thenumber of epochs, K s specified by the user.Here the encoding scheme is that each individualis to parameterise a whole group of g nodes inANNs, this means that every component of eachindividual represents a connection weight.c. The best network that survived will join thenetwork architecture evolution.All survived networks form a new population. Rank thenetworks in the population according to their errorvalues on the validation set.If the best network found is accepted or the maximum

number of generations has been reached, go to step 7 .Otherwise continue.Employ the PSO algorithm to evolve the networksarchitecture. Here each individual in PSO representsthe number of the hidden nodes and the connectiondensity of a network. When the network architecture ofan individu al cha nges, thlere are two situations:a. If some hidden nodes need to be added to anetwork, the PT algorithm only evolves the newadded nodes to explain s much of the remainingoutput variance s possible. It can decrease thecomputation time than evolving the entirenetwork and prevent from destroying thebehaviour already Icarned by the parentIf some hidden nodes need to be deleted from anetwork, the nodes will be deleted in the reverseorder in which they were originally added to thenetwork or uniformly at random. Then the PTalgorithm evolves the weights of the entirenetwork.Go to Step 3 .Train the best ANN further using the P T algorithm onthe combined training and validation set until itconverges.

b.

PSONN, evolving ANNs architectures and weightlearning are alternated. This process can avoid a movingtarget problem resulted from .the simultaneous evolution ofboth architectures and weights [7], and maintain thebehavioural link between a parent and its offspring [ l I].And the network architectures are adaptively evolved byPSO algorithm, starting from the parents weights instead ofrandomly initialised weights, so this can preferably solvethe problem of the noisy fitness evaluation that can misleadthe evolution [3].

3 Experimental studiesIn order to evaluate the ability of PSONN in evolvingANNs, it was applied to two real problems in the medicaldomain, i.e., breast cancer a nti heart disease. All data setswere obtained from the UCI machine learning benchmarkrepository. These medical diagnosis problems have thefollowing characteristics [121:

The input attributes used are similar to those ahuman expert would w e in order to solve the sameproblem.The outputs represent either the classification of anumber of un derstanda.ble classes or the p redictionof a set of understandable quantities.Examples are expensive to get. This has theconsequence that the tmining sets are not very large.

The purpose of the breast cancer data set is to classify atumour s either benign or malignant based on cell

2488

8/13/2019 Particle Swarm Optimisation for Evolving Artificial Neural Network

3/4

descriptions gathered by microscopic examination. It first 349 examples of the whole data set were used forcontains 9 attributes and 699 examples of which 485 are training, the following 175 examples for validation, and thebenign examples and 241 are malignant examples. The final 17 5 examples for testing. For the heart d isease set, thepurpose of the h eart disease data set is to predict the first 134 examples were used for training set, the followingpresence or absence of heart disease given the results of 68 examples for the validation set, and the final 68various medical tests carried out on a patient, it contains 13 examples for the testing set. Table 1 show the experimentalattributes and 270 examples. For the breast cancer set, the results averaged 30 runs for the testing data set.

- Breast cancer data set Heart disease data setMean Min Max Mean Min MaxError 1.322 0.153 3.362 11.724 9.958. 13.637Errorrate 0.011 0.000 0.051 0.146 0.112 0.186

Breast cancer data setTable 2 Comparison with other work

Heart disease data set0.0145rror rate(Mean) 0.0115 0.0112 0.1653 0.1478 0.1460

We compare the results of PSONN with other works. Forthe breast cancer problem, Setiono and Hui h ave proposed anew ANNs constructive algorithm called FNNCA [13].Prechelt also reported results on manu ally constructed ANNusing HDANN [12]. For the heart disease problem, Bennetreported a testing result with their M SM l method [14], andthe best manually designed ANN S also gave a result. Theresults are show in Table 2. From the results, we know that PSONN is able to evolveboth the architecture and weights of A NN s, and the AN Nsevolved by PSON N has better accuracy and generalisationability than th e o ther algorithms.

4 ConclusionsPSONN perfectly harmonises the architecture and weightsevolution of artificial neural networks, which effectivelyalleviate the noisy fitness evaluation problem and themoving target problem. ANN s evolved by PSONN has verycompact ANNs with good g eneralization ability.

AcknowledgeThis work is supported by National 973 FundamentalResearch Program of China. The author thanks Prof. JunGu of Hong Kong University of Science and Technologyfor his encouragement and support in this research.

ReferencesN.Burgess. A constructive algorithm that convergesfor real-valued input p attems. Int. J. Neural Syst., vol5, no. 1, pp. 59-66, 1994R. Reed. Pruning algorithms-A survey, IEEE trans.Neural Networks, vol 4, pp. 740-747,1995.D. B. Fogel. Evolutionary computation: toward a newphilosophy of machine intelligence. New York: IEEEPress, 1995.G. F. Miller, P. M. Todd, and S. U . Hegde. Designingneural networks using genetic algorithms. In Proc.3rd Int. Con Gene tic Algorithms Their Applications,CA: Morgan Kaufhan n, pp. 379-384, 1989.R. K. Belew, J. MchInemey and N. N. Schraudolph.Evolving networks: Using G AS with con nectionistlearning. Computer Sci. Eng. Dep t., Univ. California-San Diego, Tech. Rep. CS90-174 revised, Feb. 1991.Kennedy, J., and Eberhart, R. C . Particle swarmoptimization, in Proc. IEEE InternationalConference on Neural Networks, IEEE Service Center,Piscataway, NJ, pp. 39-43, 1 995.X. Yao. A review of evolutionary artificial neuralnetworks. Int. J Intell. Syst., vol. 8, no. 4, pp. 539-567,1993.J R. McDo nnell and D. W aagen. Evolving recurrentperceptrons for time-series modeling. IEEE Dan.Neural Networks, vol. 5 , no. 1, pp. 24-38, 1994.V . Maniezzo. Genetic evolution of the topology andweight distribution of neural netwo rks. IEEE Trans.Neural Networks, vol. 5 , no. 1, pp. 39-53, 1994.

2489

8/13/2019 Particle Swarm Optimisation for Evolving Artificial Neural Network

4/4

[lo] Fahlman, S.E., and Lebiere, C.. The cascade- benchmarking. Neurocomputing, vol. 9, no. 3, pp.343-347. 1995.[13] R. Setiono and L. C. K. Hui. Use of a quasinewtonmethod in a feedforward neural network constructionalgorithm. IEEE Tram. Neural Network, vol. 6 , pp.

correlation learning architecture. In D.S.Touretzhy.Advances in neural information processing systems 2,pp. .524-532, San Mateo, CA: Morgan Kaufmann,1990.[l 17 P.J. Angeline, G.M.Sauders, and J.B. Pollack. An 740-747, 1995.evolutionary algorithm that constructs recurrent neuranetworks. IEEE Trans. Neural Networks, vol. 5 , ;:p. [14] K. P . Bennett and 0 IL Mangasarian. Robust linearprogramming discrimination of two linearlyinsemirable sets. Ovtirnization Methods Sofm are. vol.4-65. 1994.[121 L. Prechelt. Some notes on neural learning algorithm 1, pp. 23-34, 1992

249