An Improved Internal Dynamics in Binary Neural

Network for Solving Optimization Problems

Rong Long WANGFaculty of Engineering,

Fukui UniversityFukui-shi, Japan

910-8507wang@fuee.fudkui-u.ac.jp

Shan Shan GUOBeijing Information

Science and TechnologyUniversity, Beijing china

100085shanshan9331@msn.com

Shinichi FUKUTA Pei ZHANG Kozo OKAZAKFaculty of Engineering

Fukui UniversityFukui-shi, Japan

910-8507zhangpei210(hotmail.com

Abstract In this paper, an improved internal dynamics inbinary neural network is proposed for efficiently solvingcombinatorial optimization problems. The improved internaldynamics provides a mechanism for increasing the exchange ofinformation between neurons, by using a stabilizationparameter to control the updating of neurons inputs. Theperformance was evaluated through simulating the N-queensproblem. Simulation results showed that the proposedtechnique has improved the searching capability of thenetwork and decreased computation time.Keywords: Binary neural network, internal dynamics,combinatorial optimization problem, N-queensproblem

I. INTRODUCTION

Combinatorial optimization problems (COPs) are oftenencountered in engineering and business. Since many COPsare NP-hard and are difficult to solve, heuristic algorithmssuch as simulated annealing and tabu search, etc. have beenwidely used to provide near-optimal solutions for reasonablecomputational efforts. However, for engineeringapplications that require fast solutions and robust hardwareimplementation, artificial neural networks are becoming apowerful tool for their inherent parallel computationalarchitecture and fault tolerance [1]. The first artificial neuralnetwork representation for solving a COP was introduced byHopfield and Tank [2]. However, artificial neural networkoften fails to converge to valid solutions. Furthermore, whenit does converge, the obtained solutions are often far fromthe optimal ones.

This paper analyzes the effects of the internal dynamicsin the binary neural network on the quality of the solutionsand proposes an improved internal dynamics to improve theperformance of the network.

II. INTERNAL DYNAMICS IN BINARY NEURALNETWORK

The binary neural network for a COP consists of a largenumber of interconnected neurons.-The neural network canbe used to compute solutions to a COP by determining theinterconnection weights (wy) between neurons, whichdescribe the COP. Typically, the network is provided by a

random initial set of inputs, each neuron updates its input Uiaccording to the internal dynamic and sends an output, Vi, inresponse to the input and according to the neuron nonlinearfunction. Consequently, the whole network converges to astable state which represents the solution of the COP. Theinternal dynamic of the neuron is described by the followingequation:

(1)

where E(V1,V2,..., Vn) is the network energy function.The output is updated from Ui using a neuron nonlinear

function called neuron model. The following two binaryneuron models are usually used in the binary neuralnetwork:

1. The McCulloch-Pitts binary neuron model [3]:V { 1 if Ui > ° i_ 1 129 (2)

{ otherwise

2. The hysteresis McCulloch-Pitts neuron model [4]:1 if Ui >UTP

Vi = if Ui <LTP i=1,2, ...,n (3)unchanged otherwise

where, UTP and LTP are constant parameters satisfyingUTP>LTP.Hence, the effects of the internal dynamics in thebinary neural network on the quality of the solutions may beanalyzed by unconditionally updating the input U,{t+l),according to Eq.(l). Besides giving solutions of low quality,this approach has another drawback, which could be noticedonly as time progresses by, that is; change ofneuron input incondition settings (i.e., U,Q)>0, Vl(t)=l, and(-aEl/Vi) > 0 or U,{t)<0, Vt)=0, and

(-aE I aVi ) < 0 ) will be ineffective and neurons remaininsensitive to input variations. When considering thecondition U1(t)>O, V,(t)=l, and (-aE / aVi ) > 0, theoutput will not be changed, as the input U,(t+1) keeps onincreasing. In fact, the inputs Ui may become so large that

0-7803-9422-4/05/$20.00 C2005 IEEE1364

Ui(t + 1) = Ui(t) OE(V,,V2,...,V,,). i--1.2. ... ,n10 V.I

TABLE I.

Simulation resultsQueens Original internal dynamics Improved internal dynamics

Convergence Rate(%) Steps Convergence Rate(%) steps50 29 462 100 44

100 27 626 100 74150 28 720 100 83200 14 793 100 116300 0 ... 93 224400 0 ... 100 190500 0 ... 100 227

the neurons become insensitive to their inputs. Similarly, inthe case of Ui(t)<0, V1Q)=0, and (-&E /aVi)< 0, theinputs Ui may become so small that the neurons becomeinsensitive to their inputs. Therefore, the internal dynamicswill provide less opportunity for exchange of informationbetween neurons. This undesirable fact has been ignored byprevious authors and it is the aim of this paper to present analternative technique to tackle these problems.

III. IMPROVED INTERNAL DYNAMICS IN THE BINARYNEURALNETWORK

In this work, an improved internal dynamics in binaryneural network is proposed, which is described by thefollowing equation.

Ui (t + 1) = a(t)* Ui (t)i=1,2,...,n (4)-E(V,V2,- ,Vn)

a-where 0 < ac(t) < 1 and t is the updating iterations. The

improved internal dynamic behavior means that the changeof the input is now controlled by the new parameter a(t),which represents neuron stabilization. In the initial stage ofupdating, the neuron stabilization is very low (a(t) is near0). Thus, the input is mainly determined by the weight stateof another neuron (the second term of Eq.(4)). In otherwords, the exchange of information between neurons is veryeasy. As time proceeds, the neurons will become more stable,increasing the value of a(t) close to 1. Hence, theinternal dynamic behavior of neuron will tend toward theoriginal internal dynamics, guarantying the networkconvergence to a stable state. In this work, the parametera(t) was defined as follow:

a(t)=l-e 2 (5)

where t is the updating iterations and A is a positiveconstant, which controls the response ofa(t) . The smallerthe constant 2 is, the faster the convergence (to a stablestate) is. For practical purposes the value of A should bechosen to be as small as possible. However, although thisoffered fast convergence, too small values in A causeslocal minima easily. Simulation tests showed that a value ofaround 10 was appropriate for the N-queens problem

IV. SIMULATION RUSULTS ON N-QUEENS PROBLEM

The performance of the improved internal dynamics wasevaluated through simulations, and was compared with thatof the original internal dynamics. In this researchevaluations were based on the N-queens problem, whoseenergy function is described in Eq.(6),

E Z( Vik 1) +-2 ( 'V -1f +2i=1 k=1l j=1 k=1

below: f

- EI E Vi-k,j-k + E i-k,j+k2ij=1 j=1 l1i-k,j-k.N 1.i-k,j+k<Nk#O k*O

(6)Although Eq.(4) described the improved internal

dynamics of one-dimensional neural network, this conceptmay be extended to any dimension of neural network. Forthe N-Queens problem, the neural network istwo-dimensional and the improved internal dynamics is asfollow:

Uij (t + 1) = a(t) -Uy (t)

(7)"E(VlVI 2,8v...I Vnn)

aviiSimulations referred to an initial parameters set; A=B=1.0

and UTP=LTP=O. In the experiments, A in Eq.(7) was

1365

selected to be 10. The detail simulation results are shown inTable 1, which summarizes the convergence rates and theaverage numbers of iterations required in 100 runs. In thisexperiment, the compared results of the original internaldynamics were calculated by setting a(t)-1 . Thesimulation results showed that the improved internaldynamics could improve the searching capability of thenetwork with shorter computation times

V. CONCLUSIONS

An improved internal dynamics in the binary neuralnetwork was proposed to efficiently solve combinatorialoptimization problems. The improved internal dynamicswas evaluated experimentally through simulating theN-queens problem. The simulation results showed that thenetwork with the improved internal dynamics could improvethe performance of the network.

REFERENCES

[1] J. J. Hopfield, "Neurons with graded response have collectivecomputation properties like those of two-state neurons." Proc. Nat.Acad Sci. U.S., Vol.81, (1982)3088-3092.

[2] J. J. Hopfield and D.W.Tank, " 'Neural' computation of decisions inoptimization problems," Bio. Cybern., Vol.52, May, (1985)141-152.

[3] W. S. McCulloch and W. H. Pitts, "A logical calculus of ideas immanentin nervous activity," Bull. Math. Biophys, vol.5, (1943)115-133.

[4] Y. Takefuji and K. C. Lee, "An artificial hysteresis binary neuron: Amodel suppressing the oscillatory behaviors of neural dynamics," Biol.Cybern., vol.64, (1991)353-356.

1366