P.A. Castillo et al- Optimisation of Multilayer Perceptrons Using a Distributed Evolutionary Algorithm with SOAP

8/3/2019 P.A. Castillo et al- Optimisation of Multilayer Perceptrons Using a Distributed Evolutionary Algorithm with SOAP

1/10

Optimisation of Multilayer Perceptrons Using aDistributed Evolutionary Algorithm with SOAP

P.A. Castillo 1 , M.G. Arenas 1 , J.G. Castellano 1 ,J.J. Merelo 1 , V.M. Rivas 2 , and G. Romero 1

1 Department of Architecture and Computer TechnologyUniversity of Granada. Campus de Fuentenueva. E. 18071 Granada (Spain)

2 Department of Computer ScienceUniversity of Jaen. Avda. Madrid, 35. E. 23071 Jaen (Spain)

[email protected]

Abstract. SOAP (simple object access protocol) is a protocol that al-lows the access to remote objects independently of the computer archi-tecture and the language. A client using SOAP can send or receive objects, or access remote object methods. Unlike other remote procedurecall methods, like XML-RPC or RMI, SOAP can use many differenttransport types (for instance, it could be called as a CGI or as sockets).In this paper an approach to evolutionary distributed optimisation of multilayer perceptrons (MLP) using SOAP and language Perl has been

done.Obtained results show that the parallel version of the developed programsobtains similar or better results using much less time than the sequentialversion, obtaining a good speedup. Also it can be shown that obtainedresults are better than those obtained by other authors using differentmethods.

1 Introduction

SOAP is a standard protocol proposed by the W3C ([ 36], [4]) that extends theremote procedure call, to allow the remote access to objects [ 9]. SOAP is theevolution of XML-RPC protocol that allows remote procedure call using XMLusing HTTP as transport protocol. A SOAP client can access remote servicesusing the interface of resident objects in remote servers, using any programminglanguage. At the moment complete implementations of SOAP are available inPerl, Java, Python, C++ and other languages [31]. Opposed to other remoteprocedure call methods, such as RMI ( remote method invocation , used by theJava language) and XML-RPC, SOAP has two main advantages: it can be usedwith any programming language, and it can use any type of transport (HTTP,SHTTP, TCP, SMTP, POP and other protocols).

SOAP sends and receives messages using XML [ 27,16,6], wrapped HTTP-like headings. SOAP services specify the method interfaces that can be accessedusing language WSDL (Web Services Description Language) [ 30,33]. WSDL isan interface description language that species those calls that can be made tothe SOAP server and the response it should return.

J.J. Merelo Guerv os et al. (Eds.): PPSN VII, LNCS 2439, pp. 676 685, 2002.c Springer-Verlag Berlin Heidelberg 2002


2/10

Optimisation of Multilayer Perceptrons 677

Using a WSDL le, it can be specied a service for different languages, sothat a Java client can access a Perl server. In some cases, if the client and theserver are written using the same language, a WSDL specication is not needed,

SOAP main advantages are: it is a lightweight protocol it is simple and extensible it is used for application communications it is designed to use HTTP as transport protocol it is not bound to any components technology it is not bound to any programming language it is based on XML and coordinated by the W3C it is the core of the Microsofts .NET technology

One of the main interests in the knowledge and use of this protocol is itsrelation with the Microsofts initiative .NET [ 5], that bases most of its softwaresupplies in remote services, using the SOAP protocol to carry out the commu-nications (although it is Microsofts version, incompatible in some cases withthe standard). This must probably means that in the near future there will bemany clients and servers offering services and using them, sharing SOAP andthe .NET specication as common protocol.

The use of SOAP for distributed computation has not been proposed yet,

except in some exceptions [ 20]. Nevertheless, SOAP is a high level protocol, whichmakes easy the task of distributing objects between different servers, withouthaving to worry about the message formats, nor the explicit call to remoteservers.

In this work we propose using SOAP for distributed computation, and wedemonstrate how it could be used for evolutionary computation.

A future in which different remote computers offer services to the scienticcommunity can be imagined: by example, all the services available at the momentby means of HTML forms could be implemented easily as SOAP services.

SOAP could be also used for distributed P2P (Peer to Peer) optimisation,where all the computers act as clients and servers to each other, interchangingpopulation elements.

In this paper we intend to explore these abilities, implementing a distributedevolutionary algorithm (EA) using Perl and SOAP, to tune learning parametersand to set the initial weights and hidden layer size of a MLP, based on an EAand Quick Propagation [ 11] (QP). This paper continues the research on evolu-tionary optimisation of MLP ( G-Prop method) presented in [ 25,8]. This methodleverages the capabilities of two classes of algorithms: the ability of EA to nd asolution close to the global optimum, and the ability of the back-propagation al-gorithm (BP) to tune a solution and reach the nearest local minimum by means of local search from the solution found by the EA. Instead of using a pre-establishedtopology, the population is initialised with different hidden layer sizes, with somespecic operators designed to change them ( mutation, multi-point crossover, ad-dition and elimination of hidden units, and QP training applied as operator).


3/10

678 P.A. Castillo et al.

The EA searches and optimises the architecture (number of hidden units), theinitial weight setting for that architecture and the learning rate for that net.

The remainder of this paper is structured as follows: in section 2 it is ex-

plained how to implement an EA using SOAP and Perl. Section 3 describesthe experiments, and section 4 presents the results obtained, followed by a brief conclusion in Section 5.

2 Distributed Evolutionary Algorithms ImplementationUsing SOAP and Perl

There are many implementations of distributed genetic algorithms [ 13], usuallyusing PVM or MPI [17], thus this paper does not intend to innovate in thatsense, but in the implementation.

There are many ways to implement a distributed genetic algorithm, one of which is the global paralelization (Farming), in which, as Fogarty and Huangpropose [12], Abramson and Abela [ 2], or Hauser and M anner [26], individualevaluation and/or genetic operator application are parallelized. A master pro-cessor supervises the population and select individuals to mate; then slave pro-cessors receive the individuals to evaluate them and to apply genetic operators.

Another way to implement paralelization is the migration: the population isdivided into small subpopulations of the same size assigned to different proces-sors. From time to time each processor selects the best individuals in its sub-population and it sends them to his nearer processors, receiving as well copiesof the best individuals of his neighbours (migration of individuals). All proces-sors replace the worst individuals of their populations. This kind of algorithmsis also known as distributed evolutionary algorithms (Tanese [32], Pettey et al.[14], Cant u-Paz and Goldberg [7]). .

An ideal client-server implementation of a distributed evolutionary algorithmcould be a server process with several threads. Each thread would include a pop-ulation, and would communicate with other threads through the shared codeamong them. Each thread would use an own tail of individuals to send to otherthreads. Each thread would evaluate its individuals in different remote comput-ers, carrying out the communication using a SOAP server.

As we cannot use a threaded version of Perl, our implementation is basedon a ring topology, with N populations (computers) sending individuals to thenext population. The parallel algorithm code has been split in two processes: theevolutionary algorithm and the code that shares individuals (migrator) betweenpopulations (islands). The system uses asynchronous communications, that, be-tween EA process and migrator process within an island (as well as betweenmigrators within different islands) are carried out using SOAP on HTTP trans-port protocol (migrators act as web servers).

Implementation was carried out using the SOAP::Lite module [19] for thePerl programming language, for its stability and the familiarity of the authorswith this language. In addition, servers are easy to implement using the computerinfrastructure that exists in our department.


4/10


Fig. 1. Architecture: each computer runs two processes, one of them contains the pop-ulation (represented as a group of people) and the other (Migrator) acts as transporter,being used to write elements of the population; thus, elements of the population arereceived from other computers; readings and writings are done by the Migrator object,that abstracts the physical architecture of the application.

SOAP was included from the beginning in the OPEAL EC library, and sofar, several distributed evolutionary algorithms congurations have been testedon EC benchmark problems such as MaxOnes and tide [ 23,18].

2.1 Evolutionary Optimisation of MLP

The evolutionary algorithm has been implemented using the OPEAL library[22], available at http://opeal.sourceforge.net under GPL license.

G-Prop method has been fully described and analysed out in previous papers(see [25,8]), thus we refer to these papers for a full description. In most cases,evolved MLP should be coded into chromosomes to be handled by the geneticoperators, however, G-Prop uses no binary codication, instead, the initial pa-rameters of the network are evolved using specic variation operators such asmutation, multi-point crossover, addition and elimination of hidden units, and QP training applied as operator to the individuals of the population. The EAoptimises the classication ability of the MLP, and at the same time it searchesfor the number of hidden units (architecture), the initial weight setting and thelearning rate for that net.

Although evolved MLP are not coded as bit strings nor other kind of cod-ication (they are made evolve directly) when they are sent from an island toanother one, they must be coded as an XML document to be sent using SOAP.

The migration policy is as follows: each n generations the algorithm sendsto the migrator e individuals, and takes i individuals. Usually, e > = i , in orderthat always are available individuals so that other islands can take them.

A steady state algorithm was used because it was empirically found to befaster at obtaining solutions than other selection algorithms [ 35]. For each gen-eration, the best 30% individuals of the population are selected to mate usingthe genetic operators. After 5 generations, several individuals are taken from


5/10


the migrator, and they are put together with the new offspring that replace theworst individual in the population.

Only default parameters listed above have been used. Genetic operators

were applied using the same application rate. No parameter tuning has beendone, because our aim is to prove how speedup improves as the number of islands grows.

3 Experiments

The tests used to assess the accuracy of a method must be chosen carefully,because some of them (toy problems) are not suitable for certain capacities of the BP algorithm, such as generalization [ 10]. Our opinion, along with Prechelt[24], is that, in order to test an algorithm, real world problems should be used.

3.1 Glass

This problem consists of the classication of glass types, and is also taken from[24]. The results of a chemical analysis of glass splinters (percent content of 8different elements) plus the refractive index are used to classify the sample to beeither oat processed or non oat processed building windows, vehicle windows,containers, tableware, or head lamps. This task is motivated by forensic needsin criminal investigation. This dataset was created based on the glass problemdataset from the UCI repository of machine learning databases. The data setcontains 214 instances. Each sample has 9 attributes plus the class attribute:refractive index, sodium, magnesium, aluminium, silicon, potassium, calcium,barium, iron, and the class attribute (type of glass).

The main data set was divided into three disjoint parts, for training, vali-dating and testing. In order to obtain the tness of an individual, the MLP istrained with the training set and its tness is established from the classicationerror with the validating set. Once the EA is nished (when it reaches the limitof generations), the classication error with the testing set is calculated: this isthe result shown.

Up to six computers have been used to run the algorithm and to obtainresults both in sequential and parallel versions of the program. Computer speedsrange from 400 Mhz to 800 Mhz and are connected using the 10Mbits Ethernet network of the University (with a high communication latency). No experimentsusing homogeneous computer network have been done, because our aim is todemonstrate potential of distributed asynchronous EA using web services.

The problem has been studied in the sequential and parallel case using a ring migration scheme : sending some individuals to the next island (computer), andgetting the best individuals from the previous island.

Each single-computer EA was executed using the parameters shown in Ta-ble 1.

Each generation 30 new individual are generated (100 generations), thus 3100individuals are generated each run. Taking into account that 2 up to 6 computers


6/10


Table 1. List of parameters used to execute the EA that runs in a computer (island).

Parameter Valuenumber of generations 100

individuals in the population 100% of the population replaced 30%

hidden units ranging from 2 to 90BP epochs to calculate tness 300

Table 2. Equivalence between number of computers in parallel runs and number of generations in sequential runs (to generate the same number of individuals).

Number of Total number Number of generationsislands of individuals (sequential version)

1 3100 1002 6200 2003 9300 3004 12400 4005 15500 5006 18600 600

have been used, then 3100 to 18600 individual are created and evaluated each

run (depending on the number of computers used).To compare with the sequential version of the method, the sequential EAshave been executed using the number of generations shown in Table 2, so thatthe number of individuals generated in each run is equivalent to the number of individuals generated using the parallel version:

4 Results

Time was measured using the Unix time command. Sequential version of theprogram was run in the faster machine; and in parallel runs, time spent bythe faster machine too was taken and used in results shown. With sequentialversion, 10 simulations were run, and 5 runs with 2, 3, 4, 5 and 6 machines. Eachsimulation was carried out adding a new computer from the set of computersused previously.

Results obtained using the sequential version can be shown in Table 3:Results obtained using the parallel version can be shown in Table 4. As can

be seen, better results in time and classication error are obtained dividing theproblem between several computers.

Figure 2 shows that speedup does not equals the number of computers used;however, simulation time is improved using several computers. Moreover, up to6 computers in the ring, the speedup grows, and as can be seen in Figure 2 itcould continue growing for a higher number of computers.

Results could be better if a dedicated communication network was used,however, the University Ethernet network is overloaded and that implies a highlatency in communications between processes.


7/10


Table 3. Results (error % and time) obtained using the sequential version of themethod.

Generations Error (%) Time (seconds)

100 35 1 20 2200 35 1 30 5300 33 2 44 8400 32 2 70 9500 32 2 91 4600 30 3 104 2

Table 4. Results (error % and time) obtained using the parallel version of the method,and the speedup for each experiment.

Islands Error (%) Time (seconds) Speedup1 35 1 20 2 12 33 2 18 1 1.63 33 1 20 1 2.24 32 1 20 2 3.55 31 2 22 3 4.16 30 1 19 2 5.5

1

2

3

4

5

6

1 2 3 4 5 6

s p e e

d u p

Number of computers

Fig. 2. Plot of the speedup (solid line) and f (x ) = x function (dashed line). Althoughspeedup is not lineal, it can be seen that simulation time is improved using severalcomputers. Moreover, up to 6 computers in the ring, the speedup grows, and it couldcontinue growing for a higher number of computers.

Although it is not the aim of this paper to compare the G-Prop method withthose of other authors, we do so in order to prove the capacity of both versionsof G-Prop (parallel and sequential) to solve pattern classication problems, andhow it outperforms other methods. Thus, obtained results (% of error in test) arebetter than those presented by Prechelt [ 24] using RPROP [ 28] (32.08), Gr onroos[15] using a hybrid algorithm (32 5), and Castillo et al. [ 25] using a previousversion of the G-Prop method (31 2).


8/10


9/10


References

1. Actas XII Jornadas de Paralelismo . Universidad Politecnica de Valencia, 2001.2. Abramson; Abela J. A. Parallel genetic algorithm for solving the school timetabling

problem. In Proceedings of the Fifteenth Australian Computer Science Conference(ACSC-15), vol. 14, p.1-11 , 1992.

3. M.G. Arenas, L. Foucart, J.J. Merelo, and P. A. Castillo. Jeo: a framework forevolving objects in java. In Actas Jornadas de Paralelismo [1].

4. Paco Avila. SOAP: revoluci=n en la red. Linux actual , (19):5559, 2001.5. K. Ballinger, J. Hawkins, and P. Kumar. SOAP in the microsoft .NET framework

and visual Studio.NET. Available from http://msdn.microsoft.com/library/-techart/Hawksoap.htm .

6. D. Box. Inside SOAP. Available from http://www.xml.com/pub/a/2000/02/09/-feature/index.html .

7. E. Cantu-Paz and D. E. Goldberg. Modeling idealized bounding cases of parallelgenetic algorithms. In Koza J., Deb K., Dorigo M., Fogel D., Garz0n M., Iba H., Riolo R. Eds. Genetic Programming 1997: Proceedings of the Second Annual Conference, Morgan Kaufmann (San Francisco. CA) , 1997.

8. P. A. Castillo, J. J. Merelo, V. Rivas, G. Romero, and A. Prieto. G-Prop: GlobalOptimization of Multilayer Perceptrons using GAs. Neurocomputing, Vol.35/1-4,pp.149-163 , 2000.

9. DevelopMentor. SOAP Frequently Asked Questions. Available fromhttp://www.develop.com/soap/soapfaq.htm .

10. S. Fahlman. An empirical study of learning speed in back-propagation networks.Technical report, Carnegie Mellon University, 1988.11. S.E. Fahlman. Faster-Learning Variations on Back-Propagation: An Empirical

Study. Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann , 1988.

12. T. Fogarty and R. Huang. Implementing the genetic algorithm on transputer basedparallel processing systems. Parallel Problem Solving From Nature,p.145-149 , 1991.

13. David E. Goldberg. Genetic Algorithms in search, optimization and machine learn-ing . Addison Wesley, 1989.

14. C. B. Pettey; M. R. Leuze; J. J. Grefenstette. A parallel genetic algorithm. In J. J.

Grefenstette (Ed.), Proceedings of the Second International Conference on GeneticAlgorithms, Lawrence Erlbaum Associates, pp. 155-162 , 1987.15. M.A. Gr onroos. Evolutionary Design of Neural Networks. Master of Science Thesis

in Computer Science. Dept. of Mathematical Sciences. University of Turku , 1998.16. Elliotte Rusty Harold. XML Bible . IDG Books worldwide, 1991.17. J.G.Castellano, M.Garca-Arenas, P.A.Castillo, J.Carpio, M.Cillero, J.J.Merelo,

A.Prieto, V.Rivas, and G.Romero. Objetos evolutivos paralelos. In Universidadde Granada Dept. ATC, editor, XI Jornadas de Paralelismo , pages 247252, 2000.

18. J.J.Merelo, J.G.Castellano, and P.A.Castillo. Algoritmos evolutivos P2P usandoSOAP. pages 3137. Universidad de Extremadura, Febrero 2002.

19. P. Kuchenko. SOAP::Lite. Available from http://www.soaplite.com .20. D. Marcato. Distributed computing with soap. Available from

http://www.devx.com/upload/free/features/vcdj/2000/04apr00/dm0400/-dm0400.asp .

21. J. Atienza; M. Garca; J. Gonz alez; J. J. Merelo. Jenetic: a distributed, ne-grained, asynchronous evolutionary algorithm using jini. pages 10871089, 2000.ISBN: 0-9643456-9-2.


10/10


22. J. J. Merelo. OPEAL, una librera de algoritmos evolutivos. Actas del Primer Congreso Espa nol de Algoritmos Evolutivos y Bioinspirados. ISBN:84-607-3913-9.pp.54-59. Merida, Spain, febrero , 2002.

23. J. J. Merelo, J.G. Castellano, P.A. Castillo, and G. Romero. Algoritmos gen ticosdistribuidos usando soap. In Actas Jornadas de Paralelismo [1].

24. Lutz Prechelt. PROBEN1 A set of benchmarks and benchmarking rulesfor neural network training algorithms. Technical Report 21/94, Fakult at f ur Informatik, Universit at Karlsruhe, D-76128 Karlsruhe, Germany. (Also in:http://wwwipd.ira.uka.de/prechelt/) , 1994.

25. P.A. Castillo; J. Carpio; J. J. Merelo; V. Rivas; G. Romero; A. Prieto. Evolvingmultilayer perceptrons. Neural Processing Letters , 12:115127, October 2000.

26. Hauser R.; M anner R. Implementation of standard genetic algorithm on mimdmachines. In Davidor Y., Schwefel H. P., M anner R., Eds., Parallel Problem Solving from Nature, PPSN III, p. 504-513, Springer-Verlag (Berlin) , 1994.

27. Erik T. Ray. Learning XML: creating self-describing data . OReilly, January 2001.28. M. Riedmiller and H. Braun. A Direct Adaptive Method for Faster Backpropaga-

tion Learning: The RPROP Algorithm. In Ruspini, H., (Ed.) Proc. of the ICNN93,San Francisco, pp. 586-591 , 1993.

29. M. Keijzer; J. J. Merelo; G. Romero; and M. Schoenauer. Evolving objects: ageneral purpose evolutionary computation library. Springer-Verlag, October 2001.

30. A. Ryman. Understanding web services. Available fromhttp://www7.software.ibm.com/vad.nsf/Data/-Document4362?OpenDocument&p=1&BCT=1&Footer=1 .

31. soaprpc.com. SOAP software. Available from http://www.soaprpc.com/software.32. R. Tanese. Parallel genetic algorithms for a hypercube. In J. J. Grefenstette(Ed.), Proceedings of the second International Conference on Genetic Algorithms,Lawrence Erlbaum Associates, pp. 177-184 , 1987.

33. V. Vasudevan. A web services primer. Available fromhttp://www.xml.com/pub/a/2001/04/04/webservices/index.html .

34. Bill Venners. Jini FAQ (frequently asked questions). Available fromhttp://www.artima.com/jini/faq.html .

35. D. Whitley. The GENITOR Algorithm and Selection Presure: Why rank-based allocation of reproductive trials is best . in J.D. Schaffer (Ed.), Proceedings of

The Third International Conference on Genetic Algorithms, Morgan Kauffmann,Publishers, 116-121, 1989.36. D. Box; D. Ehnebuske; G. Kakivaya; A. Layman; N. Mendelsohn; H.F. Nielsen; S.

Thatte; D. Winer. Simple Object Access Protocol (SOAP) 1.1, W3C Note 08 May2000. Available from http://www.w3.org/TR/SOAP .

Documents

P.A. Castillo et al- Optimisation of Multilayer Perceptrons Using a Distributed Evolutionary Algorithm with SOAP