2005 First International Conference on Neural Interface and Control Proceedings; 26-28 May 2005; Wuhan, China

NEW EVOLUTIONARY NEURAL NETWORKS WE1 GAO

Wuhan Polytechnic University, Wuhan 430023, China Email: wgaowh@,hotmail.com

Abstract-The evolutionary neural network can be generated combining the evolutionary computation and neural network. Based on analysis of merits and dements of previously proposed evolutionary neural network models, combining the immunized evolutionary programming proposed by author and BP neural network, a new evolutionary neural network model whose architecture and connection weights evolve simultaneously is proposed. A t last, through the typical XOR problem, the new model is compared and analyzed with BP neural network and traditional evolutionary neurd network. The computing results show that the precision and elficiency of the new model are all good.

Keywords - evolutionary neural network; evolutionary programming; genetic algorithm; immunized evolutionary programming;

1 INTRODUCTION

As two important branches of Computation Intelligence (GI), the combination of evolutionary computation and neural network is necessary and inevitable [I]. And then, a new neural network model- Evolutionary Neural Network (ENN) model is generated [2-31. In ENN model, the auto-adaptability of evolutionary computation and learning capability of neural network is combined effectively. And then, almost all demerits of traditional neural network model can be overcome in ENN model. So, the ENN is a very promising neural network model. Someone has already predicated that the EN” model is the next generation neural network model [4]. Based on above ideas, a lot of researchers have studied the ENN and already proposed many ENN models [l-111. But because the complexity of ENN research, this study is not very good. So, it is very necessary to still do some work on this field. In this paper, based on analysis of merits and demerits of previously proposed evolutionary neural network models, combining the immunized evolutionary programming proposed by author and BP neural network, a new evolutionary neural network model whose architecture and connection weights evolve simultaneously is proposed. At last, this new model is verified by the typical XOR problem.

11 EVOLUTIONARY NEURAL NETWORK MODEL

A . Shortcomings of traditional evobfionary neural network model

Through carefully analyzing the existing evolutionary neural network models, we can see that, the evolutionary computation used in evolutionary

neural network models is mainly genetic algorithm and evolutionary programming. And the evolutionary neural network model based on genetic algorithm is the one kind that has been studied for a long time and has obtained lot of achievements. But the evolutionary neural network model based on evolutionary programming is a newcomer. As deepening of study on ENN model, more and more researchers think that it should be a tendency to study ENN model based on evolutionary programming [ 12-1 31. Becauie the traditional ENN model based on genetic algorithm has follow shortcomings. (1) Problems produced by coding operation

The coding operation is the first operation of genetic algorithm. The binary code is generally used in genetic algorithm, which corresponds to coding individuals into discrete space. When coding the architecture of neural network, this operation will make . the architecture fixed and make some good architecture lost. While, when coding connection weights, it will make the expressing precision low. (2) Problems produced by genetic operation In genetic algorithm, the crossover operation is the

main searching operator. But it will likely destroy the produced good network architecture. And also the crossover operation can generate the “interconversion”, that is, two different genotypes are essentially the right-and-left interchange of same network architecture, The mutation operation of genetic algorithm will produce large jump, and makes the searching process of algorithm unstable. (3) Multi-multi mapping of genotype and representation type

First, it will generate estimating error when fitness of representation type approximates the fitness of genotype. Second, there will exist arraying problem, that is to say, one representation type is corresponding with many genotypes. (4) When neural network’s architecture and connection weights evolve simultaneously, genetic algorithm will restrict the network architecture very strongly.

By contrary, application of evolutionary programming in ENN model has some obvious advantages.

(1) Evolutionary programming operates directly with representation type, which not only omits the complicated operations of coding and decoding in genetic algorithm but also avoids the problems produced by crossover operation and multi-expression.

Evolutionary programming is easy and suitable to make neural network’s architecture and connection weights to be evolved simultaneously, which is favorable to decrease the estimating error of individual

(2)

0-7803-8902-6/05/%20.00 02005 IEEE 167

fitness. (3) Evolutionary programming emphasizes the

generation linking of parent and offspring, and pay attention to behavior evolution of neural network.

According to above analysis, Yao et a] [9-IO] proposed one kind of typical ENN model called EP Net. In this model, evolutionary programming and modified BP network is combined. And then, the shortcomings of traditional ENN are overcome effectively. But in this model, users must only use their experiences to estimate some parameters that have strong relation with the performance of ENN model, which makes the robustness of ENN model very poor.

Based on above discussion, and in order to improve the performance of ENN model, based on the Immunized Evolutionary Programming proposed by author, a new ENN model whose architecture and connection weights evulve simultaneously is proposed in this paper. The theories of neural network have proved that, three-layer feed forward neural network can approximate any mapping from input space to output space with any precision [ 141. Also, the previous studies have proved that [14], in order to improve the generalization of neural network, its architecture should be as simple as possible. So, in this ENN model, the three-layer feed forward neural network is applied. Also, to make the model as simple as possible, the full-linking network is applied, that is to say, the designed ENN model must satisfy follow conditions.

@ The input neuron has no input linking; @ The output neuron has no output linking; @ There is only one linking between any two

neurons; @ There is only feed forward linking between any

two neurons.

B. New evolutionary neural network model

To express clearly, the flow chart of new ENN model

The detailed explaining of above flow chart is given is given firstly as fig. I .

as follows. (1) Giving initial parameters

Giving number of input neuron and output neuron and also giving the change scope of hidden-layer neuron and initial scope of its linking weight. (2) Generating initial population

Randomly generating certain number of neural networks as initial population, whose hidden neuron number and linking weights are generated in their initial scope. (3) Fitness computation

According to the giving sample set, training network by “sample counterchanging method”, and transforming the computation error of each network as the fimess of training network individual. (4) Mutation operation

First, according to the fitness of each individual, deciding the mutation object, which is network architecture or linking weight, If it is network architecture to be mutated, the number of hidden neuron is mutated using uniform mutation method. If

the number of hidden neuron is increased, the increased linking weights are generated randomly. If the number of hidden neuron is decreased, its linking is directly deleted. If it is linking weight to be mutated, each linking weight is mutated by “adaptive Cauchy mutation method”. (5) Selection operation

According to “random competition method” and thickness of each individual, the new score of each individual is calculated by individual fitness and its thickness. Then selection is performed according to new score of each individual. (6) Termination criterion

generation and the best individual fitness.

follows. (1) Individual expression

According to requirement of new ENN model, the individual should include number of hidden neuron and linking weights and thresholds of whole network. In order to express network simply, the threshold of neuron is put into the matrix of linking weight. To express easily, the individual expression is taken structural data type. So, the pseudocode of individual is as follows.

Termination criterion is the threshold of evolutionary

Some details of the new ENN model are given as

Type geti Integer yinjiedianshu Real W[i]bJ

End type geti Where, W[i]b] is the matrix of linking weight, that

include two matrixes which are matrix of linking weight between input layer and hidden layer and matrix of linking weight between hidden layer and output layer. Parameter of yinjiedianshu is the number of hidden neuron. (2) Expression of fitness function

In order to improve the generalization of ENN model, the network individual is trained by “sample counterchanging method”, and then its fitness is calculated. The sample counterchanging method is that, as to network individual training of each generation, not the whole training sample set is used, that is to say, a part of training sample set ( the 80% of whole sample set) is randomly drawn out to train the individual of each generation. So, the used training sample set for neural network of each generation is changed, then the fitness of individual whose generalization capacity is poor is small while the fitness of individual whose generalization capacity is strong is large. Consequently, the performance of the whole ENN model is improved through selection.

The error function of neural network is expressed as follows.

The individual fitness of neural network is expressed by follow transformation of error function of neural network.

1 I + E

f =-

168

(3) Mutation operation

selected by network temperature of individual.

as follows.

Before mutation operation, the mutation type. is

The network temperature of individual is expressed

f q = 1 - - fm,

where, f is fitness of individual, h a x is fitness of best individual in population. In the operation, the mutation probability of

individual architecture is TI. The reason is that, when TI is large, the performance of network individual is poor, and then the probability of its architecture change should be large, which i s that, the influence on network performance of network architecture change is stronger than the influence of linking weights change.

In this ENN model, the mutation of network architecture is change of number of hidden neuron in network. In order to operate simply, here the uniform mutation is applied, that is to say, a value that is in changeable scope of hidden neuron number is taken out randomly to replace the original number of ~ hidden neuron.

The mutation of network linking weight is adaptive Cauchy mutation method. The advantage of this mutation is that, when individual performance is poor, the search scope is large. While when individual performance is better, the search scope is small. And then the search precision is guaranteed.

The mutation expression is as follows.

where, 0 i is the normal deviation of parameters, Ci (0,1) is a random value obeying Cauchy

F(X) is the fitness of individual,

pi

distribution, which is different to each weight,

and yi is particularparameter, where p,=1

and yi=O, n is the total number of network linking weight, T i s a adaptive parameter, which expression is as

follows. T = r*

Flax -L where, TO is the initial parameter, whose value can

be 2.5, Fmax and Favr is the best fitness and average fitness

of current generation. We can see that, the mutation operation in this ENN

model is a simpIer operation, and it can improve the independence of network evolution strongly and avoid the subjective disturbance in many previous models effectively at the condition that the evolutionary effect of the ENN model can not be affected strongly.

( 4 ) Selection operation In this ENN model, the selection based on individual

thickness adjustment in immunized evolutionary programming is applied. The details of this operation

are as follows. In the immune system theory, as to the invading

external antigen, the immune system can produce relevant antibody. After antibody integrates with, antigen, the antigen can be destroyed through a series of chemic reactions. At the same time, the immune system will produce prompting and restraining reaction between antibodies, which is done according to the thickness of antibody. From above introduction, we can see that, the immunized process can guarantee the diversity of antibody, and make the antibody adaptive to the environment through prompting and restraining reaction between antibodies.

According to above analysis, here we introduce the prompting and restraining action between antibodies in immune system into selection operation of traditional evolutionary programming, and the score of individual can be adjusted by thickness term. If individual thickness is too high, the evolutionary process will be easy to be trapped into premature convergence. So, this individual must be restrained in evolutionary process. At the same time, it must be guaranteed that the selection probability of individual whose fitness is large is high. The detailed operation is that, the adjustment terms of individual thickness and fitness are added to the original score of individual, which is as follows,

where, a: and p are adjustable parameters, and here we can use this equation, a = p =OS;

C is the thickness of individual, whose definition is the rate of number of individual whose fitness is largest or approximative to largest fitness to total number of papulation, here it is as follows,

t - (O .S .maxF+ m a x F ) N

C =

where, the numerator is the total number of individual whose fitness is largest and individual whose fitness is the 80% of largest fitness,

F(i) is the fitness of individual i, MaxF is the largest fitness in population. From above score equation we can see that, as to the

individual whose thickness is high, its score adjustment is small when its fitness i s high ( the second term of equation). If the thickness of individual i s low, the third term of equation can guarantee its score adjustment is large when its fitness is high.

TIT. SIMULATION EXPERIMENT

In order to verify the new ENN model proposed here, the typical problem of XOR is applied to test the new ENN model.

The training sample set of XOR problem is as follow table 1.

169

Table 1 Training set of XOR problem x 1 0 1 1 0 1 0 - 1 - 1 - 1 x2 0 0 1 1 -1 -1 - 1 0 1

Y 0 1 1 1 0 1 1 1 0

In new ENN model, the controlling parameters are as follows. Number of input neuron is 2. Number of output neuron is 1. The number scope of hidden neuron is 1 tolO. The initial value scope of linking weight is -1.0 to 1.0. The population size of evolutionary algorithm is 100. The generation threshold is 200. At the same time, the maximum error of best individual is given as 10-5.

In order to compare the new ENN model with BP network and traditional ENN model based on genetic algorithm, the two network models are applied to solve the same XOR problem. In BP network model, the controlling parameters are as follows, leaming-rate v = 0.6, momentum /z = 0.2. The architecture of network is 2-2-1. In traditional ENN model, the controlling parameters are as follows, P, = 0.05,P‘ = 0.8,N = 30 . The evolutionary generation threshold is 200.

Training three network models with training set in table 1 , we can get the follow results.

The network architecture of new ENN model is 2-2-1. The comparison results are in follow table 2.

From above experiment results we can see that, the computation effect of new ENN model is obviously better than that of BP network and traditional ENN model. Its computation precision is higher and iterative time is fewer. So, the new ENN model proposed in this paper is a very good ENN model and can be applied in many complicated engineering problems.

CONCLUSiON

ENN model is a new kind of neural network model combining evolutionary computation and neural network theory. Because in this model, the

auto-adaptability of evolutionary computation and leaming capability of neural network can be combined effectively, the ENN model has become the inevitable tendency of neural network model. As to the importance of this study, based on analysis of merits and dements of previously proposed ENN models, combining the immunized evolutionary programming proposed by author and BP neural network, a new evolutionary neural network model whose architecture and connection weights evolve simultaneously is proposed in this paper. At last, this new ENN model is verified by typical XOR problem, and is compared with BP network and traditional ENN model based on genetic algorithm. The results show that, the new ENN model can obviously improve the computation precision and computation efficiency and is a very good neural network model.

REFERENCES

Chen Yingqi, Yu Songyu, and Zhao Yi, “Combination of Evolutionary Computation and Artificial Neural Network”, Tnfrared and Laser Engineering, 28(4), pp.6-91, 1999 Xin Yao, “A Review of Amficial Neural Networks”, Int. J. of Intelligent Systems, 8(4),

Xin Yao, “Evolutionary Artificial Neural Networks”, Int. J. of Neural Systems, 4(3),

H. Muhlenbein, “Limitations of Multi-Layer Preceptron Networks-Steps Towards Genetic Neural Networks”, Parallel Computing, vol 14,

S. Baluja, “Evolution o f an Artificial Neural Network Based Autonomous Land Vehicle Controller”, E E E Trans. on SMC. , vol 26, pp.450-463, I996 G E Millar, P. M. Todd, and S. U. Hegde, “Designing Neural Network Using Genetic Algorithms”, In Proc. 3rd Int. Conf. Genetic

pp.539-567, I993

pp.203-222, I993

pp.249-260, I990

Table 2 Computation results of three layers neural network model input BP network Traditional New ENN model Expectation

’

model ENN model in this paper output (0,O) 0.003864 0.000041 0.000000 0

(1,O) 0.998074 0.999937 1 .oooooo I (1, 11 1 .OOOOOO 0.999745 0.999997 1

0.996921 0.999688 1 .oooooo 1

(1, -1) 0.000773 0.000013 0.000002 0 (0, -1 ) 0.999867 0.999945 1 .oooooo 1

( -1 , -1) 0.997784 0.999991 1 .oooooo 1 (-190) 0.999777 0.999798 1 .oooooo 1

(-1, 1 ) 0.001 867 0.000137 0.000005 0 Training error 0.000014 0.000007 0.000000

Iterative time 700000 41 14

170

Algorithms and Their Applications, J. D. Schaffer Eds. San Mateo, CA: Morgan Kaufmann, pp.379-384, 1989

171 A. N. Skourikhine, “An Evolutionary Algorithm for Designing Feed-forward Neural Networks”, In 7th Int. Conf. on EP’98, V. W. Porto, N. Saravanan, and D. Waagen, et al, Eds. New York: Springer, pp.629-635, 1998

[XI H. Takahashi, T. Agui, and H. Nagahshi, “Designing Adaptive Neural Network Architectures and Their Learning Parameters Using Genetic Algorithms”, In Proc. Science of ANN U , D. W. Ruck Eds. Orlando, Florido,

Xin Yao, Yong Liu, “A New Evolutionary System for Evolving Artificial NeuraI Networks”, IEEE Trans. on NN, 8(3),

[lo] Jian Fang, Yugeng Xi, “Neural Network Design Based on Evolutionary Programming”,

pp.208-2 15,1993 [9]

pp.694-713, 1997

Artificial Intelligence in Engineering, vol 1 I ,

[ l I ] V. Maniezzo, “Genetic Evolution of the Topology and Weight Distribution o f Neural Networks”, IEEE Trans. on NN, 5(1),

[12] P. J. Angeline, G M. Saunders, and J. B. Pollack, “An Evofutionary Algorithm that Constructs Recurrent Neural Networks”, IEEE Trans. on NN, 5 ( l), pp. 54-65, 1994

[I31 J. R. McDonnell, D. Waagen, “Evolving Recurrent Percephms fur Time-Series Modeling”, IEEE Trans. on NN, 5(1),

Jin Fan, “Foundation of Neural Computation Intelligence-Principles and Methods”, Chengdu: Southwest Jiaotong University Press, 2000

pp.155-161, 1997

pp.39-53, 1994

pp.24-38, I994 [ 141

171