Embed Size (px)
Citation preview
The Pennsylvania State University
The Graduate School
College of Engineering
EVOLVING INTELLIGENT CONTROL
RULES FOR A COLLABORATIVE
MULTI-AGENT MANUFACTURING CELL
A Dissertation in
Industrial Engineering
by
Marc Duane Latorella
© 2017 Marc Duane Latorella
Submitted in Partial Fulfillment
of the Requirements
for the Degree of
Doctor of Philosophy
December 2017
ii
The dissertation of Marc Duane Latorella was reviewed and approved* by the following:
Soundar R. T. Kumara
Allen E. Pearce and Allen M. Pearce Professor of Industrial and Manufacturing Engineering
Dissertation Advisor
Chair of Committee
Guodong Pang
Associate Professor of Industrial and Manufacturing Engineering
Conrad Tucker
Associate Professor of Industrial and Manufacturing Engineering
and School of Engineering Design Technology
Michael Saunders
Professor Emeritus of Entomology
Janis Terpenny
Peter and Angela Dal Pezzo Professor of Industrial and Manufacturing Engineering and
Head of the Harold and Inge Marcus Department of Industrial and Manufacturing Engineering
*Signatures are on file in the Graduate School.
iii
ABSTRACT
How can effective intelligent control rules be determined for complex multi-agent
systems? Finite state automata are used to model how agents take actions to move them
between different conditional states. Agent software must effectively encode this
behavior and facilitate optimal agent collaboration. Often, complexity of the behavior and
the unforeseeable nature of emergent agent interaction makes for difficult a priori
development of intelligent controllers. An evolutionary learning method was constructed
to derive intelligent control rules for heterogeneous or homogeneous agents to
accomplish individual and system goals. These collaborative agents rely on sensor
information to inform their actions instead of relying on centralized tactical coordination
or interagent communication.
The evolutionary learning program required the instantiation of a pool of
intelligent controller solutions. A set of probabilistic vectors was selected as the format of
the intelligent controller solution. This format was chosen with an eye towards potential
continued adaptation once deployed in real-life autonomous scenarios. This solution form
also allowed for application of numerous genetic operators during the reproductive phase,
as well as reinforcement learning, during and after the simulation fitness evaluation.
Rules associated with condition states were only added as those states that were
encountered during simulation evaluation, thus maintaining compactness and tractability
of the method. Because the iterative genetic process is so closely intertwined with the
simulation process, a pre-existing simulation software package could not be used. This
required the programming of both the evolutionary method as well as the simulation
testbed, which included the modeling of all agents and the environment as well.
iv
A problem scenario was chosen wherein multiple automated guided vehicles
process parts through a manufacturing cell. Method validation was accomplished by
evolving intelligent control rules which resulted in optimal behavior of the multi-agent
system. Numerous experiments were then performed to optimally allocate controller
types to agents, or to overcome sensor or agent faults.
There is an increasing need to address real-world autonomous multi-agent
applications which require flexible controllers that can adapt to changing circumstances
in remote environments. This method was developed to be a step towards achieving that
goal.
v
TABLE OF CONTENTS
List of Tables .………………………………………………………………………………....
List of Figures .………………………………………………………………………………...
Acknowledgments .……………………………………………………………………............
Chapter 1: INTRODUCTION.…………………………………………………………………
1.1 Problem Statement …………………………………………………………………...
1.2 Problem Scenario: AGVs in a Manufacturing Cell…………………………………..
1.3 Research Background………………………………………………………………...
1.4 Research Motivation………………………………………………………………….
1.5 Discussion of the Research Problem……………………………………………........
1.6 Overview of the Solution Methodology…………………………………………..…
1.7 Contributions ……………………………………………………………………..…
1.8 Organization of the Thesis………………………………………………………..…
Chapter 2: BACKGROUND LITERATURE………………………………………...............
2.1 Collaborative Multi-Agent Systems………………………………………………...
2.2 Genetic Algorithms…………………………………………………………….........
2.3 Evolutionary Strategies……………………………………………………………..
2.4 Related Research Literature Survey ………………………………..........................
Chapter 3: METHODOLOGY..………………………………………………………………
3.1 AGVs in a Manufacturing Cell……………………………………………………..
3.1.1 AGV Problem Representation Objective ….………………………….…….….......
3.1.2 AGV Problem Control & Effectiveness Objectives………………………………...
3.1.3 AGV Problem Validation Objective …………………………………….….…........
3.1.4 AGV Problem Implementation Objective…………………………………………..
3.1.5 AGV Problem Experimentation Objective ……………………………….....….......
3.1.6 AGV Problem Instructional Objective …………………………….…….…….…....
3.2 Generalized Methodology …………………………………………………..………
3.2.1 Representation Objective …………………………….………………………..……
3.2.2 Control & Effectiveness Objective………………………………………………..…
3.2.3 Validation Objective………………………………………………..……………….
3.2.4 Implementation Objective…………………………………………………………....
3.2.5 Experimentation Objective ……………………………….....…................................
3.2.6 Instructional Objective …………………………….…….…….…............................
Chapter 4: RESULTS & EXPERIMENTS......…………………………………………..…...
4.1 Method Development Results…………………………………………..…………...
4.2 Overview of Experiments…..…………………………………………..……………
4.3 Experiment 1: Midsize / Nine-Action / Individually-Controlled………………..………
4.4 Experiment 2: Midsize / Nine-Action / Bookend-Controlled……………………..…….
4.5 Experiment 3: Midsize / Nine-Action / Universal-Controlled…………………………..
4.6 Experiment 4: Midsize / Nine-Action / Divided-Controlled……………………...…….
4.7 Experiment 5: Midsize / Five-Action / Individually-Controlled……………….……....
4.8 Experiment 6: Midsize / Five-Action / Bookend-Controlled……………………..……
4.9 Experiment 7: Midsize / Five-Action / Universal-Controlled……………………….....
4.10 Experiment 8: Midsize / Five-Action / Divided-Controlled…………………………...
4.11 Experimental Results……………………………………………………………..….
vii
viii
x
1
1
2
5
9
12
14
19
20
22
22
34
40
42
46
46
47
48
58
61
61
61
62
62
67
75
75
76
77
78
78
91
93
107
118
131
143
157
169
180
192
vi
Chapter 5: GENERALIZABILITY AND SCALABILITY………………………..…………
Chapter 6: UNIQUENESS AND CONCLUSIONS….…………………………………..….
Chapter 7: FUTURE WORK….……………………………………………………..……….
Appendix: Main Program Code………………………….………………………………….
References………………………….………………………………………………………...
198
201
204
205
229
vii
LIST OF TABLES
Table 2.1: Research Literature Survey: Evolutionary Learning of Multi-Robot/Agent Systems…….
Table 3.1: AGV Validation Problem: State Transition Rules Determined By Inspection…………....
Table 4.1: Midsize Problem Results: Nine Actions vs. Five Actions……………………………..….
43
60
194
viii
LIST OF FIGURES
Figure 1.1: AGVs in a Manufacturing Cell Problem………………………………………………….
Figure 2.1: Reactive Subsumption Agent Control Architecture……………………………………....
Figure 2.2: Markov Decision Process……………………………………………………………..…..
Figure 2.3: Q-Learning…………………………………………………………………………….….
Figure 2.4: Learning Classifier Systems and BBA Algorithm…………………………………….….
Figure 2.5: Genetic Algorithms Example………………………………………………….………….
Figure 2.6: Adaptive Landscape………………………………………………………….…………...
Figure 3.1: AGV Problem and System Characteristics………………………………….…………….
Figure 3.2: AGV Problem Condition & Action Values……………………………….………………
Figure 3.3: Initial Unbiased Automaton for AGV Grid Problem……………………….…………….
Figure 3.4: Three Initial Automaton Solutions Each With One Start State……………….………………
Figure 3.5: Solution Zero For AGV Grid Problem Executes a State Transition………….……………..
Figure 3.6: Two Automata Solutions After simulation training of visited states……….………………
Figure 3.7: Crossover point selection applied to the AGV Grid problem……………….……………
Figure 3.8: Hybridization of 1st generation automata forms a new offspring solution……….………
Figure 3.9: Small AGV Problem for Validation…..…………………………………….……………
Figure 3.10: Method to determine the next state in the automaton……………………….………..…
Figure 3.11: Automaton representation using start state and unbiased probabilities……….…………
Figure 3.12: General form of probability value and related variables…………………….………….
Figure 3.13: General form of method’s automaton solution…………………………….……………
Figure 3.14: Recombination technique applied to automaton structure……………….……………..
Figure 4.1: Generalized Representation of Collaborative Multi-AGV Problem…….……………….
Figure 4.2: Alternate Representation of Problem…………………………………….………………
Figure 4.3: Initial Configuration of System for Experiment 1……………………….………………
Figure 4.4: Evolving Fitness Scores for Experiment 1.………………………………………………
Figure 4.5: Generational Fitness for Experiment 1…………………………………………………..
Figure 4.6: Scores of Three Components of Fitness for Experiment 1………………………………
Figure 4.7: Initial Configuration of System for Experiment 2……………………….………………
Figure 4.8: Evolving Fitness Scores of Experiment 2……………………………………………….
Figure 4.9: Generational Fitness for Experiment 2………………………………………………….
Figure 4.10: Scores of Three Components of Fitness for Experiment 2…………………………….
Figure 4.11: Initial Configuration of System for Experiment 3……………………………………..
Figure 4.12: Evolving Fitness Scores for Experiment 3……………………………………………..
Figure 4.13: Generational Fitness for Experiment 3…………………………………………………
Figure 4.14: Scores of Three Components of Fitness for Experiment 3………………………..……
Figure 4.15: Initial Configuration of System for Experiment 4……………………………………...
Figure 4.16: Evolving Fitness Scores of Experiment 4………………………………………………
Figure 4.17: Generational Fitness for Experiment 4…………………………………………………
Figure 4.18: Scores of Three Components of Fitness for Experiment 4…………………………….
Figure 4.19: Initial Configuration of System for Experiment 5…………………………….………..
Figure 4.20: Evolving Fitness Scores for Experiment 5……………………………………………..
Figure 4.21: Generational Fitness for Experiment 5…………………………………………………
Figure 4.22: Scores of Three Components of Fitness for Experiment 5……………………………..
Figure 4.23: Initial Configuration of System for Experiment 6……………………………………...
Figure 4.24: Evolving Fitness Scores of Experiment 6………………………………………………
Figure 4.25: Generational Fitness for Experiment 6…………………………………………………
Figure 4.26: Scores of Three Components of Fitness for Experiment 6…………………………….
Figure 4.27: Initial Configuration of System for Experiment 7…………………………..…………
Figure 4.28: Evolving Fitness Scores for Experiment 7…………………………………………….
Figure 4.29: Generational Fitness for Experiment 7…………………………………...……………
Figure 4.30: Scores of Three Components of Fitness for Experiment 7…………………………….
4
25
30
31
32
39
41
47
47
48
50
51
54
55
56
58
65
68
70
70
72
79
80
94
104
104
106
107
116
117
118
119
128
128
129
131
140
141
142
143
155
156
157
158
167
167
169
170
178
179
180
ix
Figure 4.31: Initial Configuration of System for Experiment 8…………………………..…………
Figure 4.32: Evolving Fitness Scores of Experiment 8……………………………………………...
Figure 4.33: Generational Fitness for Experiment 8…………………………………………………
Figure 4.34: Scores of Three Components of Fitness for Experiment 8…………………………….
181
190
191
192
x
ACKNOWLEDGMENTS
I would like to thank my advisor, Dr. Soundar Kumara, for all of his help and
support over the years. His unlimited patience and dedication is a reflection of the deep
commitment he has to his students as a mentor and as a teacher. It is an example which is
inspiring and will always be remembered. I would also like to thank my other committee
members, Dr. Guodong Pang, Dr. Conrad Tucker and Dr. Michael Saunders for their
guidance in shaping the direction of this dissertation. Thanks are also due to the faculty of
the Industrial and Manufacturing Engineering Department, including Dr. M. Jeya
Chandra, Dr. Robert Voigt and Dr. Janis Terpenny, for their help facilitating my pursuit
of this degree. I must also thank Lisa Fuoss and the rest of the administrative staff for all
of their assistance. Finally, I would like to thank my parents and my sister for their
unwavering support and encouragement over the years.
During my time at Penn State, I am thankful to have received funding and
learning opportunities in the form of several teaching assistantships, a research
assistantship and a fellowship. From May 1996 to August 1998, I received a research
assistantship at the Applied Research Laboratory at Penn State, which is a division of the
Office of Naval Research. My work there partly inspired the chosen direction of my
dissertation research. I was also granted a GSRP fellowship from July 2000 to June 2002,
under NASA Research Grant NGT-1-52249, to explore advanced control strategies for
the FreeFlight autonomous aircraft program. I acknowledge that the aforementioned
federal funding was used to support some of the work described in this dissertation. The
findings and conclusions within this dissertation do not necessarily reflect the view of the
funding agencies.
Chapter One: Introduction
1.1 Problem Statement
There is a current trend toward proliferation of collaborative multi-agent systems
across numerous application areas. Agents are increasingly expected to operate
autonomously in complex and dynamic environments. The indeterminable nature of
complex system interaction prohibits derivation of effective intelligent controllers by
traditional methods. The purpose of this research is to create a unique hybrid evolutionary
learning method which uses modified probabilistic finite state automata to develop
customized intelligent controllers for complex, collaborative multi-agent systems. This
hybrid method employs both evolutionary learning and reinforcement learning
techniques. The method will be applied to a manufacturing cell problem with
collaborative automated guided vehicles (AGVs) to demonstrate its effectiveness in
deriving the intelligent control rules which lead to optimal collaborative behavior where
communication and tactical guidance is limited or even non-existent.
A number of sub-problems are associated with the derivation and application of
this method. The sub-problems are listed below (each sub-problem forms as an
objective):
1. Representation Objective devises a standard approach to reformulate any
problem in terms that will allow the method to be applied. This involves fully
understanding the conditions and actions that define the multi-agent system.
2. Control Objective constructs the method so that it can continually evolve a
pool of prospective intelligent controllers for the application problem. This
2
involves constructing an evolutionary learning process as well as a simulation
environment in which the pool of intelligent controller solutions can be
evaluated.
3. Effectiveness Objective involves the creation and fine-tuning of goal
performance functions within the simulation runs as well as the setting of
genetic and other process parameters to achieve successful evolution of an
optimal solution.
4. Validation Objective applies the method to a small problem for which the
control rules can be determined easily by inspection. These results are
matched by the control logic derived by the method thus confirming the
efficacy of the method for use in larger problems.
5. Implementation Objective entails application of the method to a realistic
multi-agent system scenario, resulting in the derivation of intelligent
controllers. The sub-problems in this research will be studied with reference to
a scenario involving automated guided vehicles in a manufacturing cell.
6. Experimentation Objective uses the method to derive different types of
controller solutions under different circumstances such as introduced sensor
faults or differently allocated controller types. Controller allocation
experiments can be performed to decide the optimal control rules for different
agents given a particular system configuration.
7. Instructional Objective enables the extraction and study of evolved system
rules so that the underlying behavioral logic evolved is better understood. The
evolved rules may serve as example strategies for other potential applications.
3
1.2 Problem Scenario: AGVs in a Manufacturing Cell
At this point, further discussion of the method would benefit from the following example
of a multi-agent system. The problem scenario of multiple AGVs in a manufacturing cell
involves a number of autonomous AGVs tasked with the goal of removing a series of
parts from a part bench and transferring them to a dump cell. The AGVs in this particular
system are capable of taking or giving parts, either from the part bench or from a
neighboring AGV. The AGVs are equipped with sensors to determine the presence of
parts. These parts may be resident to themselves, being held by neighboring agents or
waiting on the bench. The AGVs are also able to move one space at a time within the
corridor in front of the part bench providing there are no obstacles. On-board sensors are
also employed for part, neighbor and wall detection for use in obstacle avoidance. AGVs
may be equipped with different functionality in its passing or moving capabilities. They
may also be governed by different or common controller logic, depending on how
controllers are allocated to the AGVs. This AGV navigation problem is scalable in
complexity given the number of system parameters involved. The problem parameters
include the number of different AGVs, the number of parts and part types, the number
and location of deposit points, the stacking limit, the grid dimensions and shape, the
range and functionality of AGV sensors, and the means of communication (if any)
between AGVs.
4
Figure 1.1: AGVs in a Manufacturing Cell Problem
A simpler version of the AGV problem involves just two AGVs with no
communication, one type of part, one deposit cell, and a short corridor in which AGVs
pass in front of the part bench. For this scaled-down version of the problem, the effective
behavioral rules, which constitute the intelligent controller for the system, can be easily
realized through conventional methods of inspection. Alternatively, the problem can be
scaled up in complexity to be representative of a class of complex, collaborative multi-
agent systems for which derivation of the intelligent control mechanism is extremely
difficult if not impossible through conventional means. The complex interaction between
system agents cannot be predetermined simply by studying the system. The new method
developed as part of this research seeks to evolve intelligent controllers for these types of
complex systems.
The simpler AGV problem will be used to demonstrate the details of the method
as well as serve as a validation of the method. Because of the limited scope of the simpler
5
AGV navigation problem, an intelligent controller solution can be obtained through more
conventional means. This solution derived beforehand can be compared to the solution
subsequently generated by the application of the method to the problem. If the method
produces an intelligent controller that elicits system behavior comparable in performance
to the predeterminable system behavior, then the method will be validated. Though the
method will be validated through application to a small problem, the scalable nature of
the problem will point to the potential value of the method in its application to more
complex, collaborative systems.
1.3 Research Background
An agent is software or a system operating autonomously within an environment towards
a specific objective (Wooldridge 1995). Intelligent controllers for agents are often
modeled by finite state automata machines (FSA) where states are defined by the values
of system conditions and available agent actions can transition an agent to a new state
within the model. Categories of autonomous agents were first described, in descending
order of dependence on rules and environment awareness, and increasing order on self-
reliance, as (1) deterministic, (2) stochastic, and (3) adaptive (Russell 1995).
Deterministic automata carry out a pre-specified set of rules designed and developed
from the outset by programmers knowledgeable of the required actions. Such automata
are extremely reliant on external stimuli and signals. They are also dependent on the pre-
described rules and have no flexibility to adapt to unforeseen events. Stochastic automata
are based on a set of training rules derived from off-line observations and stochastically
optimized (Bellman 1961). Adaptive automata are the least dependent on environmental
6
awareness, dynamically establish their own training rules while in operation, and are the
most able to adapt to unforeseen changes and tasks.
Because of their ability to formulate new rules of behavior and functions without
external programming, and because they can grow in experience over the course of their
deployment, adaptive automata are best suited for the distributed collaborative multi-
agent applications that have grown increasingly complex in recent years. This trend will
assuredly continue into the future. Often, agent systems must operate in harsh or remote
environments, where the desired behaviors and actions of the agents cannot be known
prior to deployment. Additionally, there are dynamic environments, which present
unexpected situations that require autonomous agents to change their existing behaviors
or adopt new behaviors. Also, as the complexity of tasks relegated to agents increases, it
becomes advantageous to use multiple collaborating agents to mutually solve a common
problem or to accomplish a common task.
Multiple agents must collaborate to solve problems, often autonomously. In multi-
agent applications, the complexity of interactions among agents and between agents, and
the environment, is too great for an expert to foresee. Creating a deterministic
collaborative agent architecture is all but impossible except for the most elementary of
systems (Douglas 2013). For such complex systems, methods must be employed that
allow agents to autonomously develop their own behaviors and actions towards
completion of the collaborative goal. The agent architecture that dictates the actions of
the individual agents and the potential communications between agents must be flexible
enough to allow new behaviors to be acquired in the course of agent operations.
7
Numerous applications demand more flexible agent architecture. The increasingly
popular SWARM technology involves the use of many roving autonomous agents to
engage in collaborative problem solving often in challenging environments (Genovese et
al, 1992). NASA and other research institutions are developing teams of autonomous
rovers for planetary survey operations (Agah 1994, Miller 1990, Reid et al, 2013). The
Office of Naval Research has coordinated underwater vehicles of different academic
institutions towards complex task efforts such as autonomous mapping of the ocean
bottom (Curtin 1997). AGV work forces will be deployed to autonomously clean up
hazardous waste in environments unsafe for humans (Arkin 1989). The proliferation of
internet applications also necessitates smarter means of producing modifiable agents.
The complexity of collaborative systems brings on a whole new set of problems.
Firstly, it is not always possible to construct fully functional agent architecture from the
outset due to the complex and often indeterminate nature of agent interactions. Secondly,
a group of agents collaborating to solve a problem is collectively, a complex adaptive
system. Within the larger system, each agent is itself a complex system of functions. Just
as with agents of biological adaptive systems, complex software agents must interact,
both synergistically and competitively, within the greater environment, of which they are
actually a part. Additionally, they must interact with other components of the
environment such as other agents, time, and space. Therefore, the fitness of any given
agent toward the contribution to a desired task is highly dependent on the rest of the
system (Nowak 2006). In an optimized system, with each component residing on a local
optimization peak of a fitness landscape representing the entire system, a slight
perturbation may result in drastic system-wide changes in fitness. In collaborative
8
systems, it is system fitness, not individual fitness that determines success, yet individual
fitness is essential. However, without individual fitness, individual agents would not
survive. Various components of a system, whether sub-functions within an agent, or
agents within a system, may be in competition with each other. This is because they may
become different and mutually exclusive or competitive solutions to the same problem
with possibly equal fitness. In such cases, advancement of one agent results in a loss to
the other. They may stalemate each other (Weise 2011). Autonomous agents must be
endowed with the ability to cope with such changes, and to acquire new functions as the
need presented by a dynamic environment occurs. Since collaborative autonomous agents
have the dynamics of complex systems, they must abide by the rules governing complex
systems. Their most difficult challenge is that of co-evolution (Maes 1990).
The key to collaborative task performance is that both the individuals and the
group are subject to a fitness test, because the individuals operate as a part of a group
upon which their individual fitness is dependent. Intelligent controller architectures for
collaborative agents are often modeled by a hierarchy of interconnected finite state
automata. Each finite state automaton dictates the agent actions that should be made
under various system conditions. Some genetic algorithms successfully evolve
collaborative agent behaviors, but this evolution involves the sequencing of pre-defined
functions to constitute different simple behaviors (Maskell et al 1996). More complex
behaviors are difficult to evolve using this method. Employing fixed pre-defined
functions limits an agent’s ability to modify these elemental behaviors. Such methods are
only able to optimize behaviors composed of the given immutable functions.
9
1.4 Research Motivation
The development of intelligent controllers for collaborative autonomous agents is the
main objective of this research. As already pointed out, developing intelligent controllers
through customary means can be difficult because of the increasing complexity of multi-
agent system applications. Creating automata models for these complex collaborative
behaviors simply by inspection is often impossible. Neural networks can be trained to
produce the desired output from the given input conditions, but they do not allow
recombination of subsystem solutions which is necessary for the learning of multi-agent
behaviors. Neural networks are also limited in the ability to optimize because of the
internal competition of co-evolving solutions within its network layers to the same
problem (Iba 1996). Once neural networks are trained they are slow or impossible to
retrain (Gowda and Mayya 2014). Genetic algorithms allow for optimization by
recombination, but they lack the reinforcement training features of neural networks
(Michalewicz 1992). GAs are more reactive to new conditions and genetic operators are
able to move instruction strings from one local optimum to another without having to
transgress valleys in the fitness curve. Intelligent agent controllers must be adaptable to
new situations once the agents are deployed in a real environment. These facts drive this
research towards development of new hybrid approach drawing on the strengths of GAs
with reinforcement learning aspects.
The new algorithm will be developed to evolve a controller for problems with
complex collaborative behaviors in which pre-defined functions are too difficult to
determine from the outset. This algorithm will achieve evolution of system behavior by
employing both group and individual selection methods, and by scoring the probabilistic
10
intelligent controller solutions. Only atomic system conditions and actions will be
employed in the new method, so that complex system behavior can be evolved in a
simulation without relying on fixed pre-defined functions. It is also desired that the
method be developed with an eye towards being adaptive, allowing for the possibility of
continued modification of collaborative behavior once the agents are potentially deployed
in a real environment. Often a group of autonomous collaborative agents must
continually adapt their behaviors in the field to deal with unexpected circumstances and
dynamic environments since all such scenarios are not foreseeable in the training
simulation. The method should also be applicable to more than just networks of identical
agents. Many systems decompose the global task among a number of heteregeneous
agents. Thus, the method should be able to evolve different control schemes for non-
identical agents, which together engage in collaborative behavior. The method should
allow for evolution of behaviors from an initially untrained controller. The method
should also be able to build more complex behaviors from initial encoded intelligence
(Quinn 2000). In contrast to other methods that use fixed functions, the method must be
able to continually evolve the initial strategies instead of having them remain immutable.
This approach to adding layers of complexity to a pre-existing agent controller is critical
to the object-oriented approach and the modular construction of agents. Such an
evolutionary method could potentially allow independently formulated agents with
completely different control schemes to develop their means of interaction. This desired
feature of seeding agent behaviors prior to the simulation is already a move towards
enabling continued behavioral adaptation after the simulation when the controller is
deployed within the real environment. Ideally, the evolutionary method used within the
11
simulated environment will continue to exist in some capacity during the real-world
operation of the agents. This is the most difficult challenge since, in the real application
environment, it is much more difficult to measure individual and global performance of
agents towards continual evolution of the group strategies without the centralized means
of doing this, as is possible during the simulation training. However, limited
communication among agents and/or periodic regrouping may still allow for this
continual adjustment of strategies in the field if the controller is in a form that allows for
such adaptation (Jennings 1993). For example, an application environment, such as with
an underwater scenario, may inhibit full communication between controllers due to the
limited SONAR range. But planned rendezvous of agents may allow for information
exchange to occur now and again. As previously stated, a new evolutionary method is
required for highly complex applications where development of control schemes by
inspection is too difficult. It stands to reason that, once effective control schemes are
evolved using this method, that it would be valuable for researchers to be able to study
the evolved rules to gain insight into the complex strategies evolved for the problem.
Thus, ideally, the new method should allow the evolved control schemes to be
reconstituted into a presentable form which is intelligible to researchers for subsequent
study of complex agent behavior. Understanding the evolved strategy for a particular
agent behavior offers researchers the potential to endow this complex behavior in agents
in other collaborative task applications. The new method to be developed features some
additional challenges as well. The implementation of the method must be
computationally efficient both within the simulation training and in the subsequent
operational phase of the agent network. The method must be scalable to the size and
12
complexity of practical applications. The generalizability and scalability of the method to
application probems will be discussed in later chapters.
1.5 Discussion of the Research Problem
Problem: To develop a method which will evolve an intelligent controller which
facilitates collaborative behavior between autonomous agents in complex scenarios, with
limited communication and supervision.
Sub-problem 1: Decompose the system into component actions and conditions to
facilitate initial representation of agent controller schemes. This representation must also
allow for initial seeding of behaviors within agent controllers.
Sub-problem 2: Develop a recombination process that will yield an improved pool of
controller solutions with every new generation. This process must be flexible enough to
allow multiple effective strategies to proceed in parallel within the solution pool. Create
the simulation environment necessary to measure the performance of each potential
solution. The simulation must appropriately model the interactions of agents within the
real environment. The evolutionary process and the simulation within must be
programmed.
Sub-problem 3: Devise the means by which controller solutions are modified during the
simulation testing to reflect negative and positive reinforcement learning. This objective
includes deriving the appropriate goal performance measure functions to achieve
increasingly complex layers of behaviors within the pool of controller solutions.
Experiment with the performance measure functions as well as the numerous parameters
of the simulation and evolutionary processes in order to achieve optimal method results
13
Sub-problem 4: Apply the method to a small problem for the purposes of method
validation.
Sub-problem 5: Apply the method to a larger more realistic problem.
Sub-problem 6: Perform experiments on evolving controllers under different
circumstances such as introduction of sensor faults or changes in agent functionality or
use of alternative controller allocation approaches.
Sub-problem 7: Explore how complex strategies can be gleaned from the controller
solutions generated by this evolutionary method. The ability to extract sub-behavior
strategies from solutions will benefit study of similarly complex autonomous agent
systems.
This dissertation describes the development of the method for evolution of
collaborative autonomous agent behaviors to meet the challenges addressed above. The
method improves upon current neural network and simple genetic algorithm approaches
that fail to produce modifiable control mechanisms for complex, collaborative networks
of agents (Gowda and Mayya 2011, Weise 2011). The problem scenario of AGVs in a
manufacturing cell will be used to describe in detail each step of the devised method. The
method will be applied to a small version of this problem for which a researcher could
develop the control rules simply by inspection. This comparison will serve as a validation
of the developed method so it can be further applied to a larger problem. Once validated,
the method will be applied to a more complex problem for which derivation of the
control schemes could not have been possible through initial inspection. The resultant
control schemes could be used to operate the agents within the real environment. The
control scheme solutions of the method can also be transformed back into a more
14
conventional form allowing the evolved complex collaborative strategies to be studied.
These evolved control schemes will be in a form that could possibly allow for continued
adaptation of behaviors once deployed in a real environment.
1.6 Overview of Proposed Solution Methodology
The development of the method was motivated by the aforementioned shortcomings of
current methods in deriving intelligent controllers for complex, collaborative multi-agent
systems. The seven objectives towards developing the solution methodology are listed
below.
• Representation
• Control
• Effectiveness
• Validation
• Implementation
• Experimentation
• Instructional
These objectives will be described in terms of the AGV navigation example where
AGVs will be representative of agents in the general case. In this section, a general
description of the research methodology is undertaken. In chapter four, a detailed
explanation and mathematical formulation is given using the AGV problem as an
example.
The hierarchically structured interconnected automata customarily used to model the
intelligent controllers within complex multi-agent systems must be reconstituted in a
form that allows for evolutionary learning. Therefore, the Representation Objective is
to decompose the agent problem into component conditions and actions. For example,
15
with respect to the AGV navigation problem, this will be achieved by establishing every
possible external condition determinable by AGV or environmental sensors as well as
internal conditions that describe the internal status of the AGVs. The full range of
variables to describe AGV actuators must also be determined. Once these conditions and
actions are known, they can be used to describe a unique state within a new single
automaton to control all the behaviors for an AGV in the system. The enumeration of
states within the automata is accomplished through bitwise representation of the
conditions and actions. By absorbing the conditions and actions into the state definition,
the state to state transitions within this new automaton become conditionless and changes
in action variables that occur between successive state definitions dictate the associated
actions implemented. Because the automata arcs are conditionless, an automatic
transition will occur between states based on the implemented action. By allowing these
transitions to occur with every simulation time tick, enacting of the automata controller
within a discrete event simulation is facilitated. External or internal condition changes
can often cause a jump between states as well as the action changes of other AGVs. Thus,
collaboration between AGVs can occur through their mutual effects on the environment
or directly as a result of their actions. Augmenting the state definitions to include the
conditions and actions allows for conditionless transitions, but also vastly increases the
number of states in the automata. Few state-defining combinations of conditions and
actions actually occur; thus, the method need only deal with those states encountered.
Creating a new single automaton in this fashion allows for a probabilistic transitional arc
to be placed between every two states. This allows the method to use an altered form of a
stochastic finite state automaton as the representational data structure to facilitate
16
evolutionary learning. This is a unique approach to represent the probability values on the
transitional arcs, which allows for recombination optimization to occur. This will be
discussed in greater detail in the methodology section.
The Control Objective involves how a pool of controller solutions undergoes the
evolutionary learning process. A pool of untrained AGV controller automata is created as
the first generation of solutions. Each first-generation solution will be constructed by
choosing a random state with an equally probable chance of transitioning to every other
state. Although the method may advance with initially untrained solutions, it does allow
the possibility of initializing a pool of solution schemes with pre-existing behavioral
rules. This can be accomplished by pre-setting the automata arc probabilities to ensure
that state transitions always occur a certain way. However they may be initialized, each
of these solutions in the generational pool is evaluated by a simulation run. Within each
simulation step, an agent relies on its controller logic solution to inform its next action.
The method uses probabilities to determine the next state to visit, thereby adding this new
state to the particular solution’s controller representation. Goal performance functions are
used to assess the extent to which the transitions made using a particular solution’s
controller have resulted in successful collaborative behavior. This measure of
performance is used in both the learning process and the genetic process. In the learning
process, performance measure for a solution determines whether the path probabilities
within the automata solution should be negatively or positively reinforced. This
performance measure score is also used to determine the solution’s likelihood of mating
and passing on its rule logic to the subsequent generation of solutions. In the genetic
process, the most successful solutions in the pool are allowed to reproduce to create the
17
next generation of solutions. These solutions use various genetic operators to achieve
recombination of sub-behaviors learned by different agent automata solutions in an
attempt to build a new agent with the successful sub-behaviors of each parent solution.
Simulation is used to assess the performance of each solution within a generation. The
genetic process continues to produce generations of solutions until the best controller
solution exemplifies successful collaborative AGV behavior.
After the construction of the method’s evolutionary and simulation aspects, the
Effectiveness Objective required that significant testing must be done to determine the
desired settings for their parameters. Experimental run results are used to fine-tune the
performance measure functions used within the simulation as well as the reinforcement
learning. Further testing is used to refine the evolutionary method parameter settings
which aim to move the system towards optimal behavior.
Validation Objective can be achieved by applying the method to the simpler
form of the AGV problem for which a successful controller can be derived from the
outset by inspection. By comparing the performance of the evolved controller mechanism
with the performance of the conventionally derived controller, the effectiveness of the
method can be determined. One of the additional features of the method is that behaviors
can be modified in response to changes in the dynamic environment. This is an important
feature in that it allows for autonomous AGVs to adapt to new situations. The method can
be shown to effectively accomplish this task by altering the conditions within the
simulated environment or by altering the goal performance functions. The adaptive
method will continue to adjust towards these new conditioned or objectives.
18
The Implementation Objective involves applying the method to a multi-agent
system such as the AGVs in a Manufacturing cell. This will serve to develop intelligent
controllers for this complex, collaborative scenario. During the development of the
method, the AGV problem will be used to experiment with method parameters. It will be
shown in which ways the method is scalable and generalizable to other problems.
As part of the Experimentation Objective, once the method is developed and the
parameters are fine-tuned towards optimal performance, experiments can be conducted
on the AGV Manufacturing Cell problem. Different allocation of intelligent controller
types to the AGVs can impact collaborative performance of the system. Other
experiments may be run to explore impact of loss of agent functionality and potential
adaptability.
The Instructional Objective is to extract the evolved behavior encoded in the
controller so researchers may gain insight into the complex behavior evolved by the
method. This can be accomplished by applying some logical reduction and data mining to
the evolved state transition rules within the controller automata. This allows researchers
to implement the evolved complex strategies in other systems. For the small AGV
validation problem, nothing is to be learned by extracting evolved behavior rules since
the successful behavioral controller rules were determinable beforehand. However, for
augmented versions of the AGV problem or for more complex systems, the instructional
objective would be important and applicable. It is important to point out that the method
offers the ability to extract rule information whereas other approaches such as Neural
Nets do not because their solutions are distributed amorphously across a network.
19
1.7 Contributions
The new method to be developed will evolve more efficient and modifiable intelligent
controllers for autonomous agents in complex collaborative scenarios. This method may
be used in different application problems. It may be used to coordinate behaviors of
automated guided vehicles, autonomous underwater vehicles, swarm AGVs, or AGV
subsystems provided the method can be scaled up appropriately. Though the method will
be developed using simple atomic actions, replacing these atomic actions with procedure
calls shows how the method may be scalable and may be useable for systems with higher
level hierarchical behaviors.
The method is unique in its representation and implementation. It applies to
networks of heterogeneous agents, allows for evolution of behaviors from an initially
untrained controller and builds more complex behaviors from initial encoded intelligence.
It also offers the potential to continually evolve initial strategies instead of having them
remain immutable and potentially enables continued behavioral adaptation after
simulation training when the controller is deployed in the real environment. It also allows
evolved control schemes to be reconstituted into a intelligible form providing instruction
for researchers. It is computationally efficient within simulation training and within the
subsequent operational phase of the agent network, and is potentially scalable to size and
complexity for practical applications. The proposed method improves upon current
Neural Network and Genetic Algorithm approaches which both fail to address the
preceding desired features of an intelligent controller for complex, collaborative multi-
agent systems.
20
There are many different approaches to evolutionary learning in multi-agent
systems as will be shown in the literature survey within this dissertation. They can differ
greatly in the types of problems being solved with regards to communication level
between agents, homogeneity vs. heterogeneity, task decomposition, genetic operators
and other features. It will be shown that the developed method is unique in a number of
ways. The method is novel in the way it encodes probabilistic finite state automata in the
rules format allowing for reinforcement learning.
Furthermore, the method will only add state transition rules as they are
encountered in the simulation thus guaranteeing a very compact and tractable final
intelligent controller logic. The intelligent controller solution format is probabilistic and
uses reinforcement learning during the simulation. This approach is done with an eye
towards keeping the rules flexible so that they may facilitate adaptability if ever deployed
in realistic remote and autonomous scenarios which pose the very hard problem of
requiring agents to synthesize new behaviors to deal with unexpected circumstances. This
method will also employ more genetic processes than most similar research in an effort to
keep the gene pool open to newly evolved tactics and also to avoid local optima. This
research will also provide application of the method to demonstrate how it can be used to
determine how to effectively allocate different controller logic within a heterogeneous
agent scenario.
1.8 Organization of the Thesis
Chapter One of this proposal states that the main objective of the research is to create an
evolutionary learning method to develop modifiable intelligent controllers for complex,
21
collaborative multi-agent systems. An AGV navigation system is the sample problem to
which the method will be applied. The general problem area of intelligent control of
multi-agent systems is discussed. The motivations for development of a new method are
presented and an overview of the proposed methodology is given.
Chapter Two provides background literature on research relevant to the method
developed in this thesis. This chapter will provide the information necessary to
understand the details of the project methodology.
Chapter Three presents the methodology used to accomplish the research objectives.
The example problem is used to demonstrate the workings of the method. The difficulties
and problems encountered during development of the method discussed, as is the
evolution of the method development. Method validation is achieved in this chapter as
well.
Chapter Four presents the results of method as applied to the AGV navigation problem.
There is also discussion of the controller allocation experiments performed.
Chapter Five contains a discussion of generalizability and scalability of the method for
application problems.
Chapter Six reflects on the uniqueness and contributions of the research.
Chapter Seven discusses future work that could be done using the developed solution
methodology.
22
Chapter Two: Background Literature
This section of the proposal offers background literature on many areas relevant to the
research and the development of the method.
2.1 Collaborative Multi-Agent Systems
Agents are often characterized as a computer system autonomously interacting within an
environment to meet certain objectives. Agents endowed with the appropriate technology
can make use of sensor information to make decisions about its actions. The decision-
making capability of an agent determines the conditions under which the agent can
influence its environment through action output. The term “agent” can be used to
describe any kind of control system. In the context of this research, agents can be thought
of as the software controllers for autonomous vehicles. Agent architectures are software
architectures to facilitate the decision-making process of agents. These agent
architectures must deal with the complexity of agent operations. A number of
environmental properties add to this complexity (Russell et al 1995). Accessibility allows
an agent full knowledge of its environment. Unfortunately, most agent environments are
inaccessible and the agent must operate with partial information. Deterministic
environments are those in which agent actions have a known and predictable effect.
Agent systems are prone to failure due to the non-deterministic nature of most
environments. In an episodic environment, the behavior of agents at one time is
independent of future behavior. Most environments are non-episodic, creating complexity
for agent decision-making. A static environment experiences changes by agent actions
23
alone. Most application environments, however, are dynamic, presenting a challenge to
agent control. Discrete environments can be thought of as events occurring in steps. Most
environments are continuous, which is a more challenging scenario for agents.
Intelligence in agents is defined by the ability to engage in flexible autonomous
decision-making to deal with operations in challenging environments (Wooldridge et al,
1995). Intelligent agents are reactive, in that they timely respond to environmental
percepts. Intelligent agents are also driven by pro-active behavior in pursuit of their
goals. Another characteristic of agents is their ability to interact with other agents in order
to meet their goals. These characteristics of intelligent agents are not easily realized. A
great deal of effort is often needed for agent to achieve reactivity, pro-activeness and
social cooperation and negotiation.
Agent control architectures are used to dictate what actions to perform under
certain system conditions. Architectures are often determined by the type of agents
involved. Logical deduction software is used for simpler logic-based agents. Reactive
agents call for a controller that prescribes actions for every particular situation. Some
BDI agents employ belief, desire and intention data structures to arrive at decisions. For
more complex scenarios, layered control architectures are used. Layered architectures
involve a hierarchy of software levels each making decisions at different levels of
abstraction (Brooks 1986).
Multi-agent systems contain cooperating agents that interact or avoid each other
in the process of performing tasks. Agents may be logic-based with pre-determined sets
of rules and contingency plans for changing conditions. More sophisticated agents may
have learning ability, allowing them to develop their own rules in response to changes.
24
They may be “rational”, conducting a task based on a perceived sense acquired from
accumulated data, to choose the betters solution among several options (Russell &
Norvig 2003). Some agents employ belief, desire, and intention (BDI) for decision
making. Agent control architectures are used to dictate what actions to perform under
certain conditions. The architecture is often determined by the number of agents
involved.
Reactive architectures, as shown below in Figure 2.1, emerged to deal with agent
problems where the symbolic representation of logic-based architectures failed. Reactive
architectures introduced the ideas that an agent is not separable from its environment and
that intelligent behavior must emerge from simpler behaviors. The most popular reactive
architecture proposed by Brooks achieves decision-making through a mapping of
situations to appropriate simple task behaviors in response (Brooks 1986). This
subsumption architecture arranges decision-making capability in layers with multiple
behaviors potentially enacted simultaneously. Higher layers represent low priority but
highly abstract behaviors. Lower layers involve simpler but more important preemptive
behaviors. A reactive controller for an agent can be derived by analyzing a system, and
determining which situations call for certain actions as well as the priority for execution
of these associative rules. Reactive architectures may be computationally efficient and
robust but they have some shortcomings. Reactive architectures use only information
about their local surroundings to make decisions because they do not employ a model of
the whole environment. This partial information limits their ability to make long-term
decision planning. The reactive architecture primarily involves fixed associative rules and
does not allow for experiential learning (Schoppers 1987). Finally, the design of a
25
reactive architecture can be difficult since the desired behaviors emerge through
interaction with the environment and these interactions are often too complex to be
divined during the controller design phase. Exhaustive experimentation with simulations
must often be employed to successfully design a reactive controller.
Figure 2.1: Reactive Subsumption Agent Control Architecture (Jennings 1996)
Autonomous agents operate in an environment without real-time guidance. They
may be rule-based which requires complex rules and anticipation of contingencies.
Alternatively, they may be learning-based, their rules determined by positive and
negative feedback to neural networks. They may also be evolution-based, developing a
set of complex rules by mimicking genetic processes to select the best rules of action.
26
Autonomous, collaborating multi-agent systems may be explicitly logic-based, use a
neural network or be evolutionary in nature. Neural networks reinforce nodes that
promote favorable actions and suppress unfavorable actions. Logic-based multi-agent
systems must be supplied with extensive instructions to avoid unforeseen events and
requirements. They must be endowed with extensive and complex instructions to avoid
unforeseen events and requirements. They must react to contingencies not easily
anticipated by a programmer. They must also be able to deal with the unique problems of
multi-agent systems, competition and convergence on a task. As the number of agents
increases, the interactions increase and optimizing such a system becomes very difficult
(Weise et al 2011).
Neural networks and stochastic systems develop their own rules through feedback by
reinforcing or repressing network nodes in response to feedback. Automata with develop
stochastic rules based on inputs. Neural networks are effective in slowly changing
environments because they tend to be locked in peaks of adaptive landscapes, after long-
term training. They become slow to adapt once trained (Gowda and Mayya 2014).
Evolutionary agents develop their own rules by genetic processes, which are rules of
self-organizing complexities. They are inherently self-optimizing. They can change
rapidly inresponse to changes in the environment. They are adept, when implemented
robustly, to solve conflicts, and to prevent system oscillation and stalemating between
agents. They are scalable and limited only by computer power requirements. Robustness
is dependendent on the customary mutation, crossover recombination, back-crossing, out-
crossing, epistasis (maintaining unused solutions for potential future use) and
chromosome elongation (ability to add genes as needed).
27
Compared to some areas of agent research, there are fewer reports of successful
multi-agent, collaborative, autonomous agents. Many of the agents begin with
preordained strings or decision trees, or they are used to improve the connections
between neural networks and the environment. The research in this dissertation focuses
on a method to develop a set of rules without prior biasing. There are fewer reports of
success in this area. One research paper in the literature survey used evolutionary
algorithms to develop multi-robot teams (Quinn 2000). It addressed the problem of
overlapping and convergent solutions. They used a clonal model with two teams,
consisting of homogeneous robots to decrease the internal interagent competition. It
worked better than a non-clonal model. Their use of genetic operators was limited to
mutation, crossover and elitism (maintenance of fittest solutions), in the next generation
gene pool. They biased the evolutionary process by beginning with a best estimate
solution.
Creating a society of agents features a significant increase in the complexity of
the system and the architecture required to control each agent. Multi-agent systems
require communication protocols to facilitate message exchange and processing for an
agent. Interaction protocols formalize the message exchange events that occur between
agents. A distributed system of computational agents with decentralized decision-making
is a natural approach to solving many problems which feature a distributed, dynamic
information environment. This distributed artificial intelligence approach is effective to
creating intelligent and distributed agent modules (DAI) applied to agent control
architecture allows for cooperation to occur when solving problems in a decentralized
multi-agent environment (Jennings 1996).
28
Agents engage in communication towards achieving coordination of joint activity
within the common environment. Coordination can involve cooperation of non-
antagonistic agents or negotiation between competing agents. For complex multi-agent
tasks, each agent ideally has a model of the environment as well as a model of every
other agent in the system. However, difficult environments impose decentralized
decision-making on the agents. It is unlikely that agents will be able to obtain complete
information about the status of other agents. The partial observability of the system that
each agent has further complicates social interaction and decision-making towards a
collaborative goal. There are numerous communication protocols used to facilitate
communication between agent societies such Knowledge Query and Manipulation
Language (KQML) (Finin et al 1994).
Distributed problem solving and planning involves coherence and competence.
Coherence is the idea that agents have as part of their individual goals the desire to work
with other agents. Competence is the idea that agents need to know how to work
effectively together. Most application problems require distributed problem solving.
Distributed problem solving is favored because it introduced parallelism as a way to
reduce the amount of computational time to solve the problem (Lane and McFadzean
1994). Distributed sensor technology also points to the future of decentralized control
technology. Multi-agent problem-solving involves a number of task sharing strategies.
Supervisory agents should decompose the task into a series of subtasks for other agents.
Subtasks should be assigned to capable and available agents. Subtask decomposition
should continue recursively until an agent can do the task individually. The fourth
29
strategy is the synthesization of the subproblem solutions into a problem solution by the
initial supervisory agent.
As the complexity of multi-agent system environments increase, agents are
required to intelligently learn and adapt to dynamic conditions. Centralized learning
involves agents learning in isolation, whereas decentralized learning involves several
agents involved in the learning process. The goal of this research is to devise an
evolutionary method for decentralized learning. One characteristic of decentralized
learning is the degree of decentralization. The use of genetic operations in the proposed
method provides a distributed and parallel nature to the learning. Another characteristic
of the learning is the exact nature of the interaction that facilitates learning. Interaction
can range from observation, simple message passing or more complicated means such as
dialogue exchange and negotiation. This is an important consideration when devising the
control and learning scheme for a multi-agent system. The rate of learning is limited by
the means and frequency of interaction between agents. For the method devised in this
research, problems are addressed which require little or no communication between
agents. The agents achieve collaborative behavior through sensing the effects on the
environment of other agents in the system. However, the method is not incapable of
addressing systems with more communicative abilities. Another characteristic of agent
learning is that agents must improve on both their individual goal performance as well as
the behavior that leads to group goal achievement. There are a number of learning
methods customarily used by agents. The method in this research primarily uses a form
of learning from examples or by practice. New behavior can be learned by the
30
presentation of negative and positive examples or through practical experience in the real
or simulated environment.
In addition to evolutionary learning, the method developed in this research also
uses a variation of reinforcement learning. Reinforcement learning allows a reactive
agent to select the next action to take from a set of potential actions. Once an action is
taken, a performance measure is provided as feedback to negatively or positively
reinforce the action taken. This learning can be modeled by a discrete time, finite state
Markov decision processes as shown below in Figure 2.2 (Kaelbing et al 1996).
Figure 2.2: Markov Decision Process (Kaelbing et al 1996)
The use of the Markov decision process is an attempt to train a series of agent
state transitions to realize goal behavior. There are various forms of reinforcement
strategies of this type such as Q-Learning shown below in Figure 2.3. Note that both
Markov Decision Process
(S,A,P,r) fully describes the agent environment at any given time
S, a set of states
A, a set of actions
P, a function which provides the probability of making a state transition from a state
s1 in S to a state s2 in S via an action a1 in A
R, a scalar reward function mapping from a state s1 and taken action a1 to a real-
valued reward
P: S x S x A [0,1]
r: S x A R
H, is an agent policy which maps from a current state s to desired action(s)
RH,s,t, is a random variable associated with an agent’s reward after using policy H for
t time steps after originating from state s
VH,y, is the expect value of the discounted sum of future rewards from starting at state
s to t time steps later
y is a discount rate 0<= y <1
VH,y = E{ ytrH,s,t}
31
Markov decision process and Q-learning require prior determination of state
decomposition in the agent architecture. Both also require prior policy functions to be
conceived based on state knowledge. The method in this research does not rely on
preconceived state definitions, but operates on the level of simple conditions and actions.
However, the principal of reinforcing transitional probabilities to encourage repeated use
of successful pathways is common to these methods and the subject of this research.
Figure 2.3: Q-Learning (Watkins 1989)
Classifier systems use environmental feedback to adjust rule strengths and often
use genetic algorithms to synthesize new rules. Similar to the Q-learning method, a
simplified classifier system attempts to train associative rules where a particular
condition implies a particular action. Each time step of an interaction cycle within the
environment results in feedback reinforcement of the classifier rule that had been selected
during that step. Multiple cycles constitute a trial of the system. The Bucket Brigade
Q-Learning
Finds a policy H* to maximize VH*,y(s) as defined in figure 2.2 above
Q, is a decision policy which estimates long-term discounted rewards for each state-
action pair
Q: S x A R
QH,y(s,a) = Va,H,y(s) defines the Q values such that a and H denote choosing an
action a at the current state followed by choices dictated by policy H
Action a is chosen to maximize reward VH*,y(s) = MAX QH*,y(s,a) for all s in S
a in A
If an action a in state s makes a transition to state s’ resulting in a reinforcement of R
then the associated Q value is updated by the following equation:
Q(s,a) (1-B)*Q(s,a) + B*(R + y*MAX Q(s’,a’))
a in A
B is a small constant learning rate
32
algorithm is a classifier system which links the reinforcement between time cycles. The
notation and classifier equations are given in Figure 2.4 below.
Figure 2.4: Learning Classifier Systems and BBA Algorithm (Dorigo 1994)
These types of reinforcement techniques are useful in systems in which agents
have limited expertise and limited information about individual or collaborative goals.
Coordination of behavior among agents is difficult under these circumstances, but
reinforcement techniques are a step towards developing control schemes. One of the
central questions in agent technology is: how useful, secure, necessary and complicating
are communications between agents? Communications can be prone to failure or errors.
In some applications, such as military ones, communications may be disfavored for
security reasons. Computational complexity may also arise due to an overabundance of
communication information to process. Similarly, it may be desired to reduce the
Learning Classifier Systems
(ci,ai) is a classifier where ci is a classifier condition and ai is a classifier action
St(ci,ai) is the strength of the classifier i at time step t
Bucket Brigade Algorithm
If classifier i fires a time step t and classifier j fires at time step t+1
and R is the external feedback from classifier i firing
and f is the fractional strength removed from classifier i then
St+1(ci,ai) = (1-f) * St(ci,ai) + f * (R + St+1(cj,aj)
BBA updates the strength of the last fired classifier i by removing a fraction of its
previous strength and replacing it by the complementary factor of the reinforced
strength of currently fired classifier j
33
complexity of an agent architecture by removing the need for modeling of other agents.
Ideally, an agent can determine the status of other agents and engage in collaborative
behavior simply through interaction with the environment. A key question is: what kind
of reinforcement techniques can achieve concurrent learning for multiple agents in a
complex environment without communications? This research proposes a new method
that will attempt to solve this problem.
Concurrent Isolated Reinforcement Learning (CIRL) can be described by a
number of characteristics. Tight and loose coupling refer to the different interrelatedness
of learning between multiple agents. Any two agents in a multi-agent system may have
different types of mutualism. Relationships can be cooperative, indifferent or adversarial.
The rate of feedback learning is another important feature in CIRL. There are resource
allocation issues that arise as well, such as the optimum number of agents required to
efficiently solve a task.
A number of test domains such as block pushing, resource sharing and robot
navigation have achieved CIRL learning among both cooperative and competitive agents.
CIRL has limitations under conditions of strongly coupled systems, delayed feedback and
a reduced number of optimal behaviors to develop. More work is required in the area of
CIRL. For instance, some global behaviors may be achieved by a group of identical
autonomous agents. Other situations may benefit from agents developing different but
complimentary problem-solving abilities (Balch 1997). The method proposed in this
research will allow for both of these approaches to evolve.
A great deal of work remains to be done in the area of multi-agent learning. Much
work is left to be done to address the differences in learning between systems with
34
varying levels of communications. This research will focus more on the area of
Concurrent Isolated Reinforcement Learning. The vast number of complex real-world
environments call for more exploration of evolutionary learning techniques. This research
project will employ genetic algorithms, various evolutionary techniques and variations on
the aforementioned reinforcement learning techniques in a method to facilitate
development of intelligent and modifiable multi-agent controllers.
2.2 Genetic Algorithms
Genetic algorithms are optimization strategies basd on genetic operators found in life.
They mimic evolution in that they result in a solution to problems that are often
unpredictable and optimized. For this reason, they are ideally suited to developing rules
for agents in complex environments or performing complex tasks. A pool of
informational string structures is created and a new generation of solutions is created by
mimicking mating between the strings. This achieves a new set of solutions constructed
of parts of the parent strings from the original generation. At every generation, a fitness
selection, as judged by system and individual fitness, is used to determine which
offspring solutions will prevail to repeat the process and reproduce to make the next
generation. A degree of randomization via mutation is introduced during the mating of
strings in an attempt to generate new substrings that may be of value. The use of GA was
motivated by the desire to make use of the effectiveness of natural adaptive processes and
to design AI systems software, which maintains these natural evolutionary mechanisms
(Holland 1975).
35
Intelligent controller solutions may occupy local optima within the landscape.
Optimization requires that the controllers’ solutions move from the less optimal local
peaks to higher or even global optima. The simplest genetic operator, mutation,
introduces random changes into a pool of controller solutions. When the level of fitness is
low, mutation and selection of the most fit can result in improved fitness of controller
solutions. However, as the adaptive landscape develops, and agents become more fit,
random changes results in decreased fitness, not more (Pigliucci 2008, Wright 1932).
Collaborating multi-agent systems present a problem to genetic algorithms. They
must work towards a common goal, despite having individual requirements within a
system. They may enter into a conflict where individual fitness towards a local task and
collaborative fitness oppose each other. In such cases, a stalemate occurs. It is proposed
that both individual and group selection apply through multi-level selection (Wilson and
Sober 1994, Traulsen and Nowak 2006).
Mutation and selection are adequate in poorly evolved systems but as
sophistication increases, mutations decrease.
To achieve multi-level selection, algorithms mimicking more advanced genetic
operators than mutation alone, must be employed. More advanced genetic operators
employ the use of recombination in a number of ways. Recombination allows parts of
solutions to combine with parts of other solutions to make new and different combination
solutions. Some of these new solutions may occupy a new local optimum by jumping
from one peak to another in an adaptive landscape.
Most optimization methods employ gradient search techniques which can
sometimes result in a method getting trapped in a local optimum, unable to seek out the
36
desired global optimum. Genetic algorithms can avoid this hill-climbing problem by
introducing an element of randomization that allows a search to jump out of local
optimum without descending to a minimum first. This feature is accomplished by genetic
recombination procedures such as mating and crossover between chromosomes. GA is
valuable in that it offers a robust technique to address complex search spaces. Because
these algorithms use multiple trial solutions, there is a parallel nature to the searching.
Unlike more traditional algorithms, GAs use probabilistic transition rules instead of non-
deterministic ones. GAs are relatively easy to initialize as well given that they emphasize
coding of parameter representation instead of the parameters themselves. These
algorithms can solve complicated optimization problems without excessive information
about the rationale of an effective solution. They simply engage in a highly effective
parallel search process for the right parameter values which will result in the optimal
solution. In this sense, there is a blindness to genetic algorithms (Goldberg 1989).
The application of GAs to solve an optimization problem is a very simple process.
The parameters of the function to be optimized are represented together as a finite length
binary string. Even complex parameters can be applied because these are easily mappable
to a binary representation. A pool of random solutions is created. Successive populations
are achieved through the use of genetic operators such as crossover recomibination,
mutation, elitism, linkage and epistasis (Griffith et al 1999).
Creating the next generation pool of intelligent controller solutions requires a
devised fitness function to test each solution string in the pool. Natural selection occurs in
the next step, as the most fit solutions are allowed to procreate with higher frequency,
thereby increasing the chance that better sub-solutions are passed on to the next
37
generation. There are different heuristics used to determine which parent strings will
mate and with what frequency. One such method sums the fitness measures of the entire
pool and determines the fraction that each string contributes to the group fitness. This
number is then used as the probability that the associated string reproduces. There are
different heuristics used to determine which parent strings will mate and with what
frequency. All such heuristics favor the fitter solutions.
When crossover is employed, after two string solutions are mated, a number of
crossover points are randomly chosen. These points along the length of the strings
determine which segment of which string is carried over to the offspring solution. Note
that different offspring solutions are possible from two parents depending on which
segments are chosen during crossover. An increase in the number of crossover points can
further increase the variability among offspring strings from the same set of parents. The
next generation of solution strings can contain more offspring strings than the parent pool
contained. The fitness function can be applied again to summarily eliminate all but the
best strings in order to maintain a consistent pool size.
This process as described is limited in that solutions that can be generated are
always restricted to be a combination of the original random first-generation parents. A
higher degree of randomness is required to facilitate the emergence of new trial solutions
into the pool. This can be accomplished by the genetic operator of mutation. Mutation
can occur after crossover occurs to allow random bits of a solution string to change their
value. This operation may introduce more effective solution approaches into the pool,
which proliferate on subsequent generations. This operator is a secondary genetic
operator and is effective when used with limited frequency.
38
Ensuring that good strings can proliferate in subsequent generations can be
accomplished in a number of ways. The solutions with highest fitness can reproduce with
a greater frequency than other solutions, as shown in the example below in Figure 2.5.
Another way is to make duplicate copies of the offspring of successful parents. The
frequency of the offspring solution in the pool in comparison to other offspring is relative
to the success of the parents.
One technique that doesn’t occur in nature that is often used in genetic algorithms
is that of elitism. Elitism allows the most successful solution of a generation to live on in
the next generation along with its offspring and other offspring solutions. This technique
ensures that each generation’s best solution is at least as good as the solution of the
preceding generation. Thus, GA with elitism maintains the best solution found each
generation throughout the search process.
39
Figure 2.5: Genetic Algorithms Example
Genetic Algorithm Example
Optimize the fitness function F = 2*x – y + 3 * z
-5<=x<2, 2<y<=6, 0<=z<8, z=2*k, 0=<k<4 x,y,z,k are Integer
Therefore x=-5,-4,-3,-2,-1,0,1 y=3,4,5,6 z=0,2,4,6
Let bi = ith binary value in solution string and map the parameters to binary representation
b0 b1 b2 x value b3 b4 y value b5 b6 z value
0 0 0 -5 0 0 3 0 0 0
0 0 1 -4 0 1 4 0 1 2
0 1 0 -3 1 0 5 1 0 4
0 1 1 -2 1 1 6 1 1 6
1 0 0 -1
1 0 1 0
1 1 0 1
1 1 1 2
Solution string is of the form “b0,b1,b2,b3,b4,b5,b6”. There are 27 = 128 possible solution
strings.
Select Pool Size = 4
Random 1st generation solutions: S1=(0010100), S2=(1111001), S3=(1011000),
S4=(0101110)
Fitness evaluations are made using the function: F1=-12, F2=5, F3=-5, F4=0
Select the most fit solution to reproduce twice, with the 2nd most fit and the 3rd most fit
Select two crossover points randomly denoted by “X”
S2= 1 1 1 1 0 0 1
X X Bit mutation made: 0 to 1
S4= 0 1 0 1 1 1 0
Two potential solutions result from this crossover as shown below:
S5= 1 1 0 1 0 0 1 yields a fitness F5= 3
S6= 0 1 1 1 1 1 0 yields a fitness F6= 2
Potential solutions from mating S2 and S3 with random crossover points are as follows:
S7= 1 1 1 1 1 0 0 yields a fitness F7=-2
S8= 1 0 1 1 0 0 1 yields a fitness F8= 1
The new generation has a sum fitness of 4, an improvement over the first generations sum of –
12
Repeated generations will yield optimal solution of S*=(1110011) or (x=2,y=3,z=6,k=3) with
F*=19
40
There are many more genetic operators in nature than have been applied to GAs.
The future improvement of Genetic algorithms lies in striving to model the process as
closely to nature as possible to make use of these important additional genetic operators.
Chromosomes are complex carriers of genetic information. Each chromosome in an
organism contains many genes. Each gene codes for a specific function in the organism.
One such genetic concept that has not been implemented in standard GAs is linkage.
Closely linked functions in an organism are encoded by closely positioned genes on a
chromosome. This makes sense after review of the crossover phase of mating. As
represented by bits in the GA strings, genes during crossover are less likely to be
separated if they are next to each other. Ideally, bits coding for closely associated
primitive functions in a solution should reside near each other. In genetics, there exist
evolutionary mechanisms for experimenting with the placement of genes on a
chromosome as well as their values. This is an important feature that needs to be
incorporated into the future of genetic algorithms (Michalewicz 1992).
2.3 Evolutionary Strategies
The simplest of evolutionary strategies is mutation followed by selection. In this strategy
a pool of randomly altered agents is subjected to selection. The fittest among them
become the pool for the next round of mutation and selection. This strategy is somewhat
effective in the beginning of a selection scheme, but not at the end when the pool consists
of a high frequency of fit agents. When the frequency of fit agents is high, random
mutation are more likely to destroy fitness rather than enhance it. A more effective
41
strategy is to recombine parts of different solutions into new and different combinations
through mating and recombination. This strategy makes use of a pool of already
somewhat fit partial solutions. The added benefit of this solution is that it allows agents to
cross from one peak on a fitness curve to another without descending into a valley as they
would with mutation and selection (Wright 1932). These concepts are illustrated in
Figure 2.6 below where the X and Y axes represent the encoding of the genotype and the
height represents its fitness.
Figure 2.6: Fitness Curve
Often after repeated selection, the pool of solutions decreases in variability and
therefore no further progress can be made as a result of selection. This dilemma can be
resolved by the application of mutation but more successfully by back-crossing to a
parental generation or outcrossing to a pool never exposed to selection. In many cases
solution strings may not be valuable towards fitness in a given situation but they might
have use in future selection schemes. The phenomenon of epistasis which can silence
genes allows non-used sequences to persist in future generations without expression and
42
still be available for the application of genetic processes to solve new problems. This
research will apply mutation/selection, mating/recombination, outcrossing and back-
crossing and epistasis.
2.4 Related Research Literature Survey
A survey was undertaken of research in the area of Evolutionary Learning of Multi-
Robot/Agent Systems. Though these research projects can vary greatly in the details,
there are some prevalent attributes that merit scrutiny. The numerous attributes examined
are shown in Table 2.1 below. The contrasting and comparing of how these attributes
differ with respect to the research done in this dissertation will support the ways in which
this method is unique. Most research in the area of evolutionary learning of multi-robot
or multi-agent systems adheres to conventional string structures to encode the intelligent
control mechanism that guides the agent. This sometimes can take the form of a decision
tree which is mapped onto a string. The method developed in this dissertation uses a
different structure which attempts to capture the more complex conditional state and
action transitions of a probabilistic finite state automata. Only two other research papers
showed a similar preference on encoding the behavioral rules of the system in this way.
Furthermore, the method in this dissertation is unique in that it only stores transition rules
as they are encountered by an agent within the simulation. Thus, one uniqueness of this
research is that a very compact form of solution data structure will result. This is valuable
because as the complexity and size of the system increases, the number of rules and the
solution structure size can remain compact and thereby keep the method tractable.
43
44
Few methods reviewed employ the kind of reinforcement learning that this dissertation
method utilizes. The reinforcement learning in this method takes place both during and
after the simulation runs. The probabilistic solution vectors are updated to reflect the
positive or negative individual behavior of the agents within the simulation runs. But they
are also revised at the end of each run to reflect positive or negative collaborative
behavior. This method is also unique in the number of genetic techniques employed.
Most research papers in the survey engage in selection, crossover recombination and
mutation and sometimes elitism as well. But few if any engage in back-crossing,
outcrossing or epistasis (which is the maintenance of dormant genes for future potential
use). This reflects this dissertation method’s emphasis on fuzzy and disruptive techniques
designed to keep the intelligent control mechanism as fluid as possible during the
evolutionary learning process so that it may ensure it does not get caught in local optima.
This approach was also done with an eye to building a mechanism that could continually
adapt away from well-learned behaviors towards new ones once transferred from the
simulation to a remote and dynamic environment. The use of fuzzy probabilistic rules is
another means by which this goal is achieved which few others in the survey have
employed. The method developed in this dissertation also offers a novel way of encoding
the state transition rules of a finite state automata. This approach allows for compact and
tractable building of the intelligent controllers. This method is one of the few to attempt
to evolve a set of intelligent controllers for heterogeneous agents. It also uses no tactical
coordinator or communication between agents. Instead, agents in the system rely on
sensor information to make decisions and achieve a form of collaboration. This approach
45
is a more challenging form within the problem area. The unique aspects of this method
will be further discussed in subsequent chapters of this dissertation.
46
Chapter Three: Methodology
This methodology section will present in detail how the method achieves each of the five
design objectives discussed in Section 1.5. Each of the problems associated with
developing the methodology will be addressed first with respect to the AGV navigation
problem and then in generalized terms.
3.1 AGVs in a Manufacturing Cell
A small problem will be described that will be used to illustrate the theory of the method
previously discussed. The demonstration problem involves AGVs operating within a row
grid as previously discussed in Section 1.2. The simple collaborative task for this
problem is to have the AGVs clear the manufacturing cell of parts without colliding with
each other or a wall and to do so in minimal time. This problem was selected to facilitate
discussion of the method. Despite its simplicity, one can imagine that the problem is a
simplified, scaled-down version of the more practical problem of a group of autonomous
AGVs on a reconnaissance or survey mission.
47
Figure 3.1: AGV Problem and System Characteristics
3.1.1 AGV Problem Representation Objective
The first step is to represent the problem in a form that can be introduced to the method.
The problem must be decomposed into external and internal conditions and actions. In
this case, as in many problems, the actions are themselves changes in internal conditions.
For example, a change in the current status of the AGV direction is the effect of an AGV
rotate action coupled with environment information contained within the simulation
program. The conditions for this problem can be described as shown below in Figure 3.2.
Figure 3.2: AGV Problem Condition & Action Values
CONDITIONS:
Sh=Holding part, Sn=No part, Bh=Bench has part, Bn=Bench empty
Lh=Left is AGV with part, Ln=Left is AGV empty, Lv=Left is void, Lw=Left is wall,
Rh=Right is AGV with part, Rn=Right is AGV empty, Rv=Right is void, Rw=Right is wall
ACTIONS:
Ml=Move Left, Mr=Move Right, Dn=Do Nothing, Tb=Take from Bench, Gb=Give to Bench,
Tl=Take from Left AGV with part, Tr=Take from Right AGV with part,
Gl=Give to Left AGV, Gr=Give to Right AGV
48
When a state transition entails certain condition value changes, then the
simulation makes the appropriate actions within the environment. The effect of this
associated action is simulated on the environment. Similarly, changes enacted by other
agents or by a dynamic environment may cause sudden condition changes. These changes
result in immediate state changes within the agent’s evolving automaton.
Figure 3.3: Initial Unbiased Automaton for AGV Grid Problem
3.1.2 AGV Problem Control & Effectiveness Objectives
As discussed above, the initial controller is unbiased with an equally likely chance of any
condition value being chosen as the next value for that condition. The decision is made
by first introducing a small degree of randomization to each probability value. This
allows a choice to be chosen sometimes that is not the value with the highest probability.
This is important in allowing the automaton to explore different pathways. A condition
value is chosen in this manner for each of the system conditions. These new condition
values dictate the next state in the automaton since the state definition is determined by
n = 4, where n is the number of system conditions
M0=3 M1=4 M2=3 M3=4 M4=4 where Mc is the # value choices for condition c
A first generation automaton is initialized with one random state represented
by random condition value choices: (V0,V1,V2,V3,V4) = (0,1,2,0,0)
This state maps to a unique state index z
z = V0+M0*V1+M1*M0*V2+(M2*M1*M0)*V3 +(M3*M2*M1*M0)*V4
=0 + 3*1 + 4*3*2 + 3*4*3*0 + 3*4*3*4*0= 27
The total number of unique states for this system is s = M0*M1*M2*M3*M4 = 576
For this state, an unbiased probability vector is stored for each condition c in the system:
(0.33,0.33,0.33) for condition 0 (0.25,0.25,0.25,0.25) for condition 1
(0.33,0.33,0.33) for condition 2 (0.25,0.25,0.25,0.25) for condition 3
(0.25,0.25,0.25,0.25) for condition 4
49
the condition value settings. When a new state is encountered, a new unbiased entry is
stored in the data structure for this state. As many path transitions are made during the
simulation run, numerous states may be acquired. At the end of the simulation run-time
period, the data structure for a first-generation solution stores the probability sets for all
states visited during the run. This data is the sum knowledge of the automaton solution
thus far in the method. Paths chosen during a run are reinforced by adding to the
probabilities corresponding to values that were chosen and subtracting from probabilities
corresponding to condition values not chosen. The amount by which the pathway
transitions are negatively or positively reinforced is a dynamic parameter within the
method known as the “Reinforcement Margin”. The rate of change of this margin is to be
determined through further experimentation with the method.
For the purpose of demonstration, a trivial pool size of three is chosen. Thus, for
the first generation of solutions, two initialized automatons will be created. The first
automaton is defined as shown in the previously shown Figure 3.1. The second
automaton is randomly initialized in a similar way with condition values and condition
value probabilities defined for a single start state as shown in the next Figure 3.4.
50
Figure 3.4: Three initial automaton solutions each with one start state
Once the generation of solutions is initialized, each trial automaton can be
introduced into the simulation for a pre-specified number of simulation ticks. Solution
zero as described above in Figure 3.4 will begin the simulation in indexed state 27 in
which the condition values are (0,1,2,0,0). This represents an AGV which is sensing a
wall nearby, and is located on the grid in square 2 from Figure 3.1 and undergoing an
action of left movement. The next state of this agent in the simulation is determined by
using the probability vectors associated with the current state. In the first generation, in
the first step of the simulation, the probabilities for the condition values are all equal
across one condition. If the maximum probability were chosen, the automaton would
simply have to choose one of the equiprobable options at random. But at this point in the
Solution 0 has just one state which serves as the start state in the first simulation run:
Solution 0 start state is randomly defined as (V0,V1,V2,V3,V4) = (0,1,2,0,0) or z=27
With unbiased probability vectors:
(0.33,0.33,0.33) for condition 0 (0.25,0.25,0.25,0.25) for condition 1
(0.33,0.33,0.33) for condition 2 (0.25,0.25,0.25,0.25) for condition 3
(0.25,0.25,0.25,0.25) for condition 4
Solution 1 is similarly defined:
Solution 1 start state is randomly defined as (V0,V1,V2,V3,V4) = (2,3,1,0,0) or z=23
With unbiased probability vectors:
(0.33,0.33,0.33) for condition 0 (0.25,0.25,0.25,0.25) for condition 1
(0.33,0.33,0.33) for condition 2 (0.25,0.25,0.25,0.25) for condition 3
(0.25,0.25,0.25,0.25) for condition 4
Solution 2 is similarly defined:
Solution 2 start state is randomly defined as (V0,V1,V2,V3,V4) = (1,0,0,1,0) or z=37
With unbiased probability vectors:
(0.33,0.33,0.33) for condition 0 (0.25,0.25,0.25,0.25) for condition 1
(0.33,0.33,0.33) for condition 2 (0.25,0.25,0.25,0.25) for condition 3
(0.25,0.25,0.25,0.25) for condition 4
51
method a certain amount of randomness in introduced into each probability. Throughout
repetition of the method, this randomization will decrease to allow a particular solution to
‘settle into’ successful behaviors. The function by which this randomization decreases is
an issue for further experiment. But initially, the degree of randomization should be large
to allow new attempted pathways.
Figure 3.5: Solution zero for AGV Grid problem executes a state transition
As shown above in Figure 3.5, a random effect introduced to the probability
vectors influences the selection of the next condition value for each condition. These
selected condition values define the next state to visit within the automaton. Once a new
state is visited, the data structure expands and stores that state definition along with new
Solution 0
Solution 0 start state is randomly defined as (V0,V1,V2,V3,V4) = (0,1,2,0,0) or z=27
With unbiased probability vectors before randomization effect is introduced:
(0.33,0.33,0.33) for condition 0 (0.25,0.25,0.25,0.25) for condition 1
(0.33,0.33,0.33) for condition 2 (0.25,0.25,0.25,0.25) for condition 3
(0.25,0.25,0.25,0.25) for condition 4
At the beginning of the method, a randomization effect between –0.5 and 0.5 is introduced.
Probability vectors after the randomization effect is added:
(0.33-0.2,0.33+0.07,0.33-0.5) (0.13,0.40,-0.17)
(0.25-0.25,0.25+0.13,0.25+0.5,0.25+0) (0.0,0.38,0.75,0.25)
(0.33-0.4,0.33+0.23,0.33+0.14) (-0.07,0.56,0.47)
(0.25+0,0.25+0.45,0.25+0.13,0.25+0.24) (0.25,0.70,0.38,0.49)
(0.25+0.12,0.25-0.4,0.25+0.35,0.25-0.12) (0.37,-0.15,0.50,0.13)
Value 1 has the greatest revised probability for condition 0 so condition 0 will next be set to “1”
Value 2 has the greatest revised probability for condition 1 so condition 1 will next be set to “2”
Value 1 has the greatest revised probability for condition 2 so condition 2 will next be set to “1”
Value 1 has the greatest revised probability for condition 3 so condition 3 will next be set to “1”
Value 2 has the greatest revised probability for condition 3 so condition 3 will next be set to “2”
Therefore, the next state to be visited by this automaton is (1,2,1,1,2) or z=343
A state transition is made to this new state, with associated condition changes shown below:
Sensing Wall Sensing an AGV
Facing North Facing West
On Square #2 On Square #4
Last Action : Stay New Action: Move Ahead
52
unbiased probability vectors. Note again that only visited states need be stored by the
method thereby reducing computational storage. After the transition has occurred in the
simulation, from the start state to the new state, the probability vectors are updated for the
start state. Since this automata solution is encoding potential agent behavior, the pathway
taken during the simulation must be reinforced. Thus, the probability condition value
chosen for each condition must be increased. The probabilities for the remaining
complimentary choices must be uniformly decreased so that all probabilities still sum to
one. Note that the randomized probabilities were only temporary and were used solely to
improve the selection mechanism. It is the original probabilities to which reinforcement
learning is applied. The rate of learning increases as the number of generations increases
during the method. Thus, repeatedly reinforced pathway solutions are not overthrown by
one or two successful alternative paths that may occur. However, the recombinatory
process will ensure that highly reinforced solutions that are no longer successful will
reproduce less frequently, eventually eliminating those newly disfavored behaviors from
the gene pool. This is key to the method’s ability to adapt to new situations. The learning
parameter is another process parameter that will require further experimentation.
There are some conditions that simply cannot be changed by the agent controller.
For simplification, the previous example did not illustrate this fact. But in reality, sensor
conditions are not under the control of the automaton. The simulated environment alters
these particular condition values. A sudden change in a sensor condition can cause a
condition value change to occur on the next time tick. Thus, probability vectors really
only encode for controllable conditions (actions). In fact, in the grid problem, AGV
location is another uncontrollable condition that is determined by the simulation program.
53
The AGV performs a move action by changing the value for the last condition. Since this
is a controllable condition (an action), the simulation program must implement the effect
of the action. The effect in this problem is to move the AGV to a new square, which is
reflected by an immediate change in the location conditions. Even though some external
conditions are not controllable, these condition values are still used to define the state.
Thus, they still have an effect on the state to state transitions in the automaton. Though
the automata may begin untrained, the effects of agent actions must be properly
implemented within the simulated environment by the simulator. The emphasis of the
controller development is to evolve only the decision-making logic through which agent
behavior is accomplished.
After a number of simulation ticks have elapsed, the automata will cease to
operate within the simulated environment. At this point, for each state visited during the
run or during the runs of previous generations, a set of condition value probabilities is
stored. For those states visited during the current generation simulation run, the
probabilities have recently been reinforced. The performance of the agent behavior
during the simulation run yields individual and group performance measures. These
valuations are determined by goal performance functions. Goal performance functions
must be altered during successive generations within the method to allow training of
elemental behaviors to yield to learning of more complex behaviors. The manner in
which goal performance functions will be altered during an application of the method is
another subject of further study and experimentation.
After performance measures are determined for the first generation of solutions,
the solutions are ranked. The best solution is maintained in the subsequent generation
54
using the concept of elitism, which ensures that the best solution of each generation
always is as good as or better than the last. The other members of the next generation are
created through mating between the best performing automata of the previous generation.
The better an automaton, the better the chance of it reproducing and passing successful
sub-behaviors into the next gene pool. The manner for selection of parents and the
frequency that each is allowed to reproduce is a parameter which requires further study. It
is also possible to introduce other genetic operators such as mutation to model effective
genetic crossing of solutions. The mutation rate is another parameter to be studied. The
method allows for recombination by randomly selecting crossover points on each of the
parent solutions. For the purposes of demonstration, two trivial first-generation solutions
are shown below in Figure 3.6. Solution zero has just two states encoded; state 27 and
state 343. Note that probabilities for each have been reinforced during the learning phase
after the simulation run. Solution one has only one state stored. This sole state happens to
be state 27. This example will be used to show how two automata solutions of different
“string” lengths can be merged during mating.
Figure 3.6: Two automata solutions after simulation training of visited states
Solution 0, State 27
(V0,V1,V2,V3,V4) = (0,1,2,0,0) or z=27 With probability vectors:
(0.30,0.30,0.40) for condition 0 (0.25,0.25,0.0,0.50) for condition 1
(0.52,0.24,0.24) for condition 2 (0.07,0.07,0.07,0.79) for condition 3
(0.02,0.04,0.02,0.92) for condition 4
Solution 0, State 343
(V0,V1,V2,V3,V4) = (1,2,1,1,2) or z=343 With probability vectors:
(0.44,0.22,0.34) for condition 0 (0.12,0.12,0.23,0.53) for condition 1
(0.6,0.10,0.3) for condition 2 (0.35,0.35,0.20,0.10) for condition 3
(0.23,0.23,0.23,0.31) for condition 4
----------------------------------------------------------------------------------------------------------------
Solution 1, State 27
(V0,V1,V2,V3,V4) = (0,1,2,0,0) or z=27 With probability vectors:
(0.50,0.25,0.25) for condition 0 (0.2,0.2,0.2,0.4) for condition 1
(0.10,0.10,0.80) for condition 2 (0.3,0.3,0.26,0.14) for condition 3
(0.4,0.4,0.1,0.1) for condition 4
55
Once the parent coupling is determined, the crossover points for recombination
must be randomly chosen. Crossover allows subsystem solutions for particular actions to
from one parent to be merged with subsystem solutions from another parent automata.
Figure 3.7: Crossover point selection applied to the AGV Grid problem
Note that the condition probabilities above in Figure 3.7 show each gene as
distributed across all states as previously formalized in Figure 3.3. A method must be
used to merge the probability values for separate conditions back under the individual
states. This technique is shown below in Figure 3.8. It allows automata solutions
featuring dissimilar state sets to still merge during mating. This is achieved by filling the
missing condition probabilities of one parent with a new unbiased set of probabilities.
Crossover points randomly selected
Solution 0 Solution 0 Solution 0 Solution 0 Solution 0
Condition 0 Condition 1 Condition 2 Condition 3 Condition 4
Solution 1 Solution 1 Solution 1 Solution 1 Solution 1
Condition 0 Condition 1 Condition 2 Condition 3 Condition 4
One of the offspring solutions that could be determined from the above crossover is as follows:
Solution 0 Solution 1 Solution 1 Solution 0 Solution 0
Condition 0 Condition 1 Condition 2 Condition 3 Condition 4
56
Figure 3.8: Hybridization of 1st generation automata forms a new offspring solution
For states each parent solution have in common, such as state 27 in the example
above in Figure 3.8, the technique merges the condition value probabilities of solution
zero and solution one, dividing up the inheritance of probability vectors between the two
parents via the selected crossover points. For states that occur in only one automata
solution, the missing condition probability vectors are set as unbiased, as shown above in
Figure 3.8 for state 343. The recombination allows for individual action control strategies
to be hybridized from both parents.
Mating among the first generation of solutions will yield many more offspring
solutions than the original pool size. The number of offspring is a parameter of the
genetic process, which is subject to further experiment. Each offspring solution must be
The new child solution must merge the condition probability sets for each state from solutions 0 and 1
According to the crossover point selection, whenever possible, probabilities for conditions 0 and 4 are
inherited from solution 0 and probabilities for conditions 1,2 and 3 are inherited from solution 1
New Solution, State 27
(V0,V1,V2,V3,V4) = (0,1,2,0,0) or z=27
With probability vectors:
Inherited from solution 0:
(0.30,0.30,0.40) for condition 0 (0.02,0.04,0.02,0.92) for condition 4
Inherited from solution 1:
(0.12,0.12,0.23,0.53) for condition 1 (0.6,0.10,0.3) for condition 2
(0.35,0.35,0.20,0.10) for condition 3
New Solution, State 343
(V0,V1,V2,V3,V4) = (1,2,1,1,2) or z=343
With probability vectors:
Inherited from solution 0:
(0.44,0.22,0.34) for condition 0 (0.23,0.23,0.23,0.31) for condition 4
State does not exist in solution 1, so unbiased probabilities must be used:
(0.25,0.25,0.25,0.25) for condition 1 (0.33,0.33,0.33) for condition 2
(0.25,0.25,0.25,0.25) for condition 3
57
applied to the simulation in order to determine a performance measure for that agent
controller. The second generation of solutions is comprised of the g most successful
automata, where g is the size of the original solution pool. At this point the process
repeats with the new generation of solutions introduced into the simulation to engage in
pathway selection and probability reinforcement.
In standard genetic algorithm fashion, the genetic process continues until a pre-
specified goal is met. This criterion may be the optimization of goal performance
functions or the proliferation of a successful solution throughout the gene pool.
A benefit to this method is the ability to seed initial behaviors in the automata
solutions. This can be achieved by initializing a series of states in the automata solution
and specifying probabilities for condition changes which effectively hardwire state
transitions in the controller.
Another benefit of this method is that the use of reinforcement learning and
recombination allows automata controllers to evolve towards new successful behavior
under changing environmental conditions. What was once a well-trained solution can
easily be negatively reinforced back towards unbiased transition probabilities after
repeated failure of the controller under the new system circumstances. The genetic
process assists by eliminating poor solutions quickly through selective mating. Therefore,
this method could prove useful to collaborative agents that engage in periodic centralized
reevaluation of their respective controllers. They can learn from their respective
experiences in the real or simulated environment, then rendezvous or engage in
communication. Once in contact, the agents can apply the method to their current
controller automata. The method will produce a better controller based on the collective
58
experiences of the agents. This best automaton solution can be implemented in each agent
before they conduct another mission run. The method provides an adaptability of the
controller, which is an essential feature to remote, autonomous agents developing new
behaviors to deal with unforeseen situations.
3.1.3 AGV Problem Validation Objective
The AGV problem will be used during the method development process. A small version
of this problem will ultimately be solved by the method to demonstrate that it is capable
of evolving collaborative agent behavior.
Figure 3.9: Small AGV Problem for Validation
The method was applied to a simple two agent AGV manufacturing cell problem
where each agent is fully capable of its operations and the sensors are all working. The
agents used a common controller type so only one rule-set was evolved during the
application of the method. The set of evolved state transition rules is shown below.
59
Note that these are a minimal compact set of rules that evolved to achieve optimal
collaborative behavior in the two-agent system. These rules effectively allow the agent to
cooperate with each other and remove all parts from the bench to the dump cell. The
number of state transition rules in this rule-set is far fewer than the number of possible
states in the model of the system.
If the method is to be validated, it needs to be shown that working out the solution
by an exhaustive method of inspection yields the same set of rules. A table was made
with every possible state configuration within the finite state automata model for the
control of the agent. These state configurations are defined by the condition values as
known to the agent at any given time. Even for a small problem there are many rows in
the table representing all the states in the system. For each of these states, the correct
action was determined through inspection using knowledge of the expected logical
workings of the system behavior. The subsequent new state transition as a result of taking
that action was also determined. This exhaustive analysis was made and recorded in the
Table 3.1 below.
60
Table 3.1: AGV Validation Problem: State Transition Rules Determined by Inspection
A close comparison between the table and the method’s evolved rule-set shows
that the rules that are evolved are indeed a subset of the rules as determined by
inspection. The functionality of the evolved intelligent control rules will therefore match
the behavior dictated by the rules derived by inspection. This accomplishes validation of
the method. Note that for scaled up versions of this application or for any other larger
problem, the derivation of rules by inspection quickly becomes intractable and time-
consuming. This further illustrates the usefulness of the method.
61
3.1.4 AGV Problem Implementation Objective
As previously discussed, the method will be applied to the AGV Manufacturing Cell
Problem under various circumstances. This will be done to confirm its scalability and to
further experiment with process parameters. Much of this will be discussed in the results
section.
3.1.5 AGV Problem Experimentation Objective
Subsequent testing will involve modification of the AGV environment to confirm the
adaptive capabilities of the method. For example, if a sensor fault is introduced, will the
AGVs learn to accommodate it? What if one of the AGVs loses its ability to transfer a
part? It will be shown that the method can evolve around such problems. There will also
be experiments performed with different allocations of controller types for heterogeneous
agent situations. The results of these experiments can be found in the results section.
3.1.6 AGV Problem Instructional Objective
Behavioral rules can be extracted from the intelligent controller rules that were evolved
during application of the method to the AGV Navigation problem. There is little to learn
regarding the behavioral rules for the smaller AGV validation problem given that the
controller solution was already determined from the outset for purposes of method
validation. However, the extraction process may be used on this problem to demonstrate
the potential of acquiring evolved rules for use in other systems. Looking closer at
evolved rule-sets illuminates the strategies that evolved sometimes unexpectedly between
interacting agents towards completion of their collaborative goals.
62
3.2 Generalized Methodology
The steps necessary to successfully apply the method to a problem will be discussed.
Representation, Control, Effectiveness, Validation and Implementation research
objectives will be addressed in this section. A mathematical description of the method
heuristic will be given.
3.2.1 Representation Objective
The first objective is to reformulate a problem into a form to which the method can be
applied. As previously stated, this involves an understanding of all of the internal and
external conditions of the system as well as the actions available to the system. The range
of values possible for each variable must be finite and mappable to an integral scale. For
ease in modeling, the actions can be thought of as conditions that are controllable to the
agent. Thus, when decomposing the problem, all external, internal and action conditions
can be grouped together to be used in the augmented state definition.
Let ne = number of external conditions detectable by agent sensors
Let ni = number of internal conditions that define an agent’s current status
Let na = number of actions that an agent can implement
n = ne + ni + na = sum total number of ‘conditions’ in the system
The objective is to make use of all conditions when making any agent decisions.
The evolutionary learning process will eventually train the corresponding probabilistic
transition values to reflect those conditions which ultimately bear no impact on certain
state transitions. Since the method aims to evolve without prior embedded intelligence,
63
all conditions must be considered for every decision at the beginning of training. Each
condition must take on an integral and finite number of values. These values need not be
the actual condition values. In fact, the problem is modeled more easily if the actual
condition values are mapped to an integer scale from 0 to the maximum number of
condition values minus one.
Let Mc = maximum number of condition values for system condition c
0,1..Mc-1 is the integer range of condition values that system condition c can take on
The current state of the system, from the point of view of one agent, can be
described by a vector comprised of the current values of the system conditions.
Let Vc = the current value of system condition c
(V0,V1…Vc…Vn-1) represents the current state of the system
s = M0*M1*..*Mn-1 such that s is the total number of unique system states
If (V0,V1,…Vc,…Vn-1) represents the current state of the system then the corresponding
index, z, of this unique state can be expressed as:
z = V0+M0*V1+M1*M0*V2+..(Mc-1*Mc-2*..*M0)*Vc +…(Mn-2*Mn-3*..*M0)*Vn-1
These s unique system states defined by the values of the system conditions can
be used as the states within a new automaton for the system. This new automaton has
many states, but since each state definition is augmented by the inclusion of condition
values, the transitions between states no longer need be qualified by conditions. The
conditionless arcs in this new automaton allow for immediate transitions to occur
between states. This configuration lends itself nicely to a discrete event simulation during
which state transitions occur with every time tick. Based on the condition values
embedded in the state definition, with the next time tick, a transition is made to another
state in the system. The change in state reflects a change in condition values. Some
64
condition values changed constitute a change in a controllable agent action, other
condition changes may be imposed by other agents or by the environment, that will occur
during interactive behavior. A non-deterministic finite state automaton with probabilistic
arc values can be derived from the configuration thus far described. But instead of a
probability on the transition arcs between a state and every other state, a two-dimensional
probability vector is employed which imposes a probability on each condition value
change. Normally in an adaptive automaton, the transition taken to the next state would
be determined by applying a function to all the possible transition arc probabilities and
then choosing one arc to follow. To simply take the best arc choice thus far, the
maximum probability among the choices would be chosen. In the automaton of this
method, there are no probabilities associated with arcs. Instead there is a set of
probability vectors associated with each state. These probabilities denote confidences in
choosing the next value for each system condition. Because the states in this automaton
are defined by the system condition values, selection of the next condition values based
on the associated probabilities dictates the next state of the system. These probability
vectors are the data structures that allow for both evolutionary learning and
recombination of automata solutions.
65
Figure 3.10: Method to determine the next state in the automaton
As discussed above, the control scheme for a multi-agent system may be
expressed as single stochastic finite state automaton with many states each defined by the
values of system conditions. The transition function relies on probabilities associated
with each state for every condition value. The next state is determined by applying a
function to each probability vector to obtain the chosen next condition value. The
condition value with the maximum probability for a condition value is not necessarily the
chosen one. Annealing and randomization are applied to the probabilities to generate new
trial pathways in the automata. The probabilities for each condition value for a given
state represent the confidence the automaton solution has in choosing that value as the
next value for that condition. The next value for each condition is then used to determine
66
the next state in the automaton as shown above in Figure 3.10. The probabilities for each
potential condition value for a single condition must always sum to one. In a given state,
when a particular condition should avoid a specific new condition value then the
associated probability value for that state’s condition value is set to zero. The method
ensures that a condition value with a zero probability cannot be chosen nor can it evolve
from zero. If a probability value is zero for a state’s condition value then the other
condition value probabilities must still sum to one. Similarly, a condition value
probability set to one ensures that condition value will be chosen over any other. If a
condition value probability is one, then the other probability condition values for that
condition must sum to zero. Initialization of the method demands that condition value
probability values be made equiprobable. That is, each condition value must have an
equally likely chance of being chosen as the value for each condition. As potential
automata solutions are run within a simulation, these probability values are negatively or
positively reinforced depending on the success of the coded pathways towards
independent and global success. But unless they are intentionally set to zero or one to
reflect seeded behavior, the condition probability values always remain greater than zero
or less than one. This allows for solutions to potentially adapt out of highly reinforced
transitions if the system conditions suddenly change. This is a key feature of the
method’s ability to modify itself in response to dynamic conditions. The possibility of
making any state transition is maintained even if the associated probabilities have been
previously negatively reinforced down to near-zero values.
67
After the multi-agent system problem is decomposed into conditions and actions
and the range of values for each condition is established, the evolutionary method must
initialize the first generation of automata solutions. Though defining the states to include
every system condition greatly increases the number of states, the method itself does not
become too computationally intractable because only the states visited in the course of a
simulation run need be saved. In fact, an automaton solution can be encoded as a series of
visited state strings each containing the fixed condition values V0,V1..Vn-1 that define that
particular state (or the state index z associated with these condition values) as well as the
condition value probability vectors used to determine the next state. Unlike most genetic
algorithms, this method does not encode the entire automaton solution as a fixed length
string. The automata solution strings within one generation of the evolutionary process
may each contain a different number of visited states and therefore a different length.
These length differences need only be accommodated during the mating process. Thus,
the form of automaton solutions is as a variable length of data strings representing the
states and each probabilistic engine used to determine the subsequent state.
3.2.2 Control & Effectiveness Objectives
The evolutionary method creates a pool of ‘organisms’ where each organism is a
potential automata solution to the multi-agent system. All first-generation solutions are
randomly initialized as simply a random start state along with an unbiased set of
probability vectors to determine the next state. The form of one initial solution is shown
in the next Figure 3.11.
68
Figure 3.11: Automaton representation using start state and unbiased probabilities
Note that the sole random start state is determined by randomized values of each
condition because each state is defined by its condition values. The associated probability
vectors for the state are initially unbiased. This is reflected in the equal probability values
for each of the Mi values possible for each condition i. If there are two potential condition
values then each the initial probability vector for that condition value would be (0.5,0.5).
These probability vectors will be used to determine the next state visited. Several
parameters are used to introduce annealing and thresholding in the condition probability
selection. This technique allows new pathways to be explored within the automata. There
are g solutions created for the first-generation pool. Any genetic method must determine
the fitness of its pool of solutions. This method determines fitness measure by
introducing each automata solution into a simulation. One automaton serves to control
the behavior of a certain number of pre-specified agents within the simulation
environment. This is the strategy when attempting to evolve intelligent controllers for
identical agents. For networks of non-identical agents, multiple sets of solution automata
are incorporated into the method instead. Within the simulation, the automata solution is
used to dictate those state transitions that ultimately reflect agent behavior. As new states
Solution j: Start State Definition: (Vj,0, Vj,1, …Vj,n-1)
Probability Vectors: ((1/M0,1/M0,…1/M0), for condition 0
(1/M1,1/M1,…1/M1), for condition 1
:
(1/Mc,1/Mc,…1/Mc), for condition c
:
(1/Mn-1,1/Mn-1,…1/Mn-1)), for condition n-1
69
are visited, these entries are added to the solution ‘string’ along with a set of unbiased
probability vectors. Performance-measuring functions are used to determine the
individual and global fitness in the agent system. For the state transitions made during the
course of the simulation, the corresponding chosen condition value probabilities are
negatively or positively reinforced based on performance measures. Thus, good state
transitions are rewarded through reinforcement of the probabilities that led to those path
choices. Several parameters are used to control the rate of reinforcement. After all
automaton solutions are tested within the simulation, their respective performance
measures are compared. Genetic operators are used to mate the best solutions produced
thus far. The more successful the solution, the more likely it is to reproduce and have its
learned behavior passed on to the next generation of solutions. The parent pool size, the
mating probability and the level of random mutation are some of the genetic parameters
used to control the mating process. Elitism is another technique that is used to maintain
the best solution of the previous generation in the next pool. The principal motivation for
this method of encoding the automata solutions, rather than as a single neural network, is
that they are in a form that allows for recombinatory genetics to take place. Different
agent automaton within the pool may have successfully generated strategies to solving
different subproblems in the simulated mission. The data structure devised to encode the
automata allows for these strategies to be recombined in an attempt to develop an
offspring solution that is endowed with the best behaviors of both its parents.
70
Figure 3.12: General form of probability value and related variables
Figure 3.13: General form of method’s automaton solution
An automaton solution j:
1st state stored is defined by (Vj,0,0, Vj,0,1, …Vj,0,n-1), w/ probability vectors
((Pj,0,0,0, Pj,0,0,1…Pj,0,0,M0), for condition 0
(Pj,0,1,0, Pj,0,1,1…Pj,0,1,M1), for condition 1
:
(Pj,0,n-1,0, Pj,0,n-1,1…Pj,0,n-1,Mn-1)) for condition n-1
2nd state stored is defined by (Vj,1,0, Vj,1,1, …Vj,1,n-1), w/ probability vectors
((Pj,1,0,0, Pj,1,0,1…Pj,1,0,M0), for condition 0
(Pj,1,1,0, Pj,1,1,1…Pj,1,1,M1), for condition 1
:
(Pj,1,n-1,0, Pj,1,n-1,1…Pj,1,n-1,Mn-1)) for condition n-1
:
rth state stored is defined by (Vj,r,0, Vj,r,1, …Vj,r,n-1), w/ probability vectors
((Pj,r,0,0, Pj,r,0,1…Pj,r,0,M0), for condition 0
(Pj,r,1,0, Pj,r,1,1…Pj,r,1,M1), for condition 1
:
(Pj,r,n-1,0, Pj,r,n-1,1…Pj,r,n-1,Mn-1)) for condition n-1
:
sjth state stored is defined by (Vj,sj,0, Vj,sj,1, …Vj,sj,n-1), w/ probability vectors
((Pj,sj,0,0, Pj,sj,0,1…Pj,sj,0,M0), for condition 0
(Pj,sj,1,0, Pj,sj,1,1…Pj,sj,1,M1), for condition 1
:
(Pj,sj,n-1,0, Pj,sj,n-1,1…Pj,sj,n-1,Mn-1)) for condition n-1
A different automaton solution k:
rth state stored is defined by (Vk,r,0, Vk,r,1, …Vk,r,n-1), w/ probability vectors
((Pk,r,0,0, Pk,r,0,1…Pk,r,0,M0), for condition 0
(Pk,r,1,0, Pk,r,1,1…Pk,r,1,M1), for condition 1
:
(Pk,r,n-1,0, Pk,r,n-1,1…Pk,r,n-1,Mn-1)) for condition n-1
General form of a probability value is: Pj,x,c,i where j is the index of the automaton solution, j=1..g
where r is the index of the state storage for the jth automaton, r= 1..sj
where c is the index of the system condition, c=1..n
where i is the index of the value choice for condition c, i=1..Mc
and g is the population size of a generation
and sj is the maximum number of visited states stored for automaton j
and n is the number of system conditions
and Mc is the number of values that condition c can take on
71
During mating, the two parent automaton solutions are not likely to have visited
the exact same states during current and past simulation runs. They may not even have
the same number of states visited. When mating these two solutions, crossover occurs by
the order of conditions. That is to say, some condition probabilities are to be taken from
the first parent’s solution while others are to be taken from the second parent’s solution.
But the assortment of condition value probabilities from each parent only occurs for
common states. When a partial set of state probabilities is inherited from one parent’s
solution for a particular state that is absent from the other parent’s solution, the missing
condition value probabilities are set as unbiased. A technique is used to randomly
selected the crossover point, which determines which condition logic will be inherited
from which parent. The best use of this technique is a subject for further experimentation.
72
Figure 3.14: Recombination technique applied to automaton structure
Note that recombination on condition probabilities presumes that agent subsystem
control may be influenced by only a subset of all conditions. The method will evolve the
causal relationships between conditions and subsystem control. Recombination also
allows introduction of the concept of linkage where related conditions are closer to each
other on the solution string, thus promoting a greater chance that these two associated
condition value probability sets will be inherited together. These closely linked condition
genes reflect that an automaton has determined through experience that these conditions
are highly associated for the desired agent goals. The condition probability values in one
Condition 0
Probability
Values
Condition 1
Probability
Values
Condition 2
Probability
Values
Condition 3
Probability
Values
Condition 4
Probability
Values
Condition c
Probability
Values
Parent 1
Solution j
Condition 0
Probability
Values
Condition 1
Probability
Values
Condition 2
Probability
Values
Condition 3
Probability
Values
Condition 4
Probability
Values
Parent 2
Solution k
Randomly Selected Crossover Points
Condition 0
Probability
Values
(Parent 1)
Condition 1
Probability
Values
(Parent 2)
Condition 2
Probability
Values
(Parent 2)
Condition 3
Probability
Values
(Parent 2)
Condition 4
Probability
Values
(Parent 1)
Potential Child Solution
A “gene” exists for all n conditions in the system. A
“gene” in parent solution j contains a probability
vector for a given condition c for every one of the sj
states encountered by the parent. Let r= 0..sj visited
states for solution j, then each block for condition c
below contains the set of probabilities shown at right:
(Pj,0,c,0, Pj,0,c,1…Pj,0,c,Mc),
(Pj,1,c,0, Pj,1,c,1…Pj,1,c,Mc),
:
(Pj,sj,c,0, Pj,sj,c,1…Pj,sj,c,Mc) vectors for condition c from all visited states
Condition c
Probability
Values
73
block of a parent solution in the diagram above in Figure 3.14 represent all the
probability vectors for a particular condition across all visited states in that automata
solution. These sets of probability vectors essentially act as a gene. The path learning
steps in the method model the intragene operations and the genetic recombination steps
of the method model the intergenic operations. Together they enable the method to
engage in effective evolutionary learning.
There is a fair amount of experimentation required to determine effective values
for the annealing, reinforcement, thresholding and genetic parameters used in the method.
Determining effective goal performance functions can be difficult as well, since learning
must often occur in stages. The performance functions may need to change over time.
Initial performance measuring techniques encourage agents to learn simple individual
behaviors within the simulated environment. As these elemental behaviors are
successfully acquired, the performance measures must reward more complex
collaborative system behaviors. This approach will allow agents to successively build
more complex behaviors on top of previously learned simple ones. Goal performance
functions must also maintain the old behaviors as new ones are learned.
After the first generation of automata solutions yield corresponding performance
measures, the best solutions are allowed to reproduce via recombination and other genetic
operators as shown above in Figure 3.14. The offspring solution pool may contain child
solutions that either improve upon parent solutions or deconstruct previous behaviors
learned. The worth of each offspring solution must be determined by running each within
a simulation run. The best offspring solutions are maintained along with the best parent
solution. The mating process then repeats.
74
Within each simulation run, automata paths chosen by the agent are negatively or
positively reinforced according to the success of the agent. This learning is accomplished
by altering the condition value probabilities to favor or disfavor transitional condition
value choices previously made during the simulation run. Over several generations of the
method, pathways may be reinforced to the point where the probability of choosing one
particular binary condition value is 0.99 and the probability of choosing the alternative is
0.01. Though evolution back from these extremes is not impossible, these probabilities
reflect that this particular condition value choice in the given state is almost always going
to occur. Through the evolutionary adaptation of these successful automata state
transitions, effective agent behavior can be learned. The flexibility of the probability
vectors allows for modifiability of agent control. If the environment is dynamic or the
goal functions are altered, the probabilities will readjust for success under these new
circumstances. This method offers the potential for continued adaptation once the agents
are initially trained and deployed in the real environment.
Though much of the discussion above has centered on initially untrained automata
solutions, it is possible to seed the method with some elemental behaviors. Instead of
initializing all of the probability vectors with equiprobable values, definite state
transitions can be hardwired into each solution by setting the probability of a particular
condition choice to 1.0 and all other choices to 0.0. Thus, in a particular state, the next
value of that condition is predetermined. Entire state transitions and even pathways
within the automata can be initially fixed with this approach. Unlike some other genetic
methods used to evolve agent behaviors, this seeded information need not be immutable.
If the aforementioned probabilities are instead set to 0.99 and 0.01 then the same intended
75
behavior will occur, but with the possibility of negatively reinforcing the behavior back
towards equiprobability, in the case that the seeded behavior is antagonistic to the
individual and/or collaborative goals.
Implementing the method on a small AGV problem will be used during the
method development phase to perfect the use of process parameters and goal performance
functions.
3.2.3 Validation Objective
The devised method must finally be validated. This can be accomplished by applying the
problem to a small problem for which an intelligent controller is already available or
easily created through inspection. If the method evolves a behavioral automaton that
successfully matches or improves upon the performance of the pre-existing controller,
then the method will have shown its effectiveness at evolving collaborative performance.
As demonstrated in Section 3.13, successful validation of this method was performed
using a small version of the “AGVs in a Manufacturing Cell” problem.
3.2.4 Implementation Objective
The method will then be applied to the AGV manufacturing cell problem.
For each application of the method to a problem described above, suitable goal
performance functions must be developed. The evolutionary and simulation process
parameters may also require adjustment. Initial seeding of behaviors will be introduced
into several problems. Modifiability of trained behaviors in response to new situations
will also be explored.
76
The method and the simulation are currently both written in the Visual Basic
programming language. This was done because of the need to experiment with a platform
that offers an evolutionary method wrapped around the simulation process. Standard
software packages for MAS simulation do not offer integration with outside evolutionary
learning software. The evolutionary method and simulator were both programmed
together to allow for transfer of information between these two stages. It points to a need
for a complex evolutionary learning software that can wrap around existing simulation
packages such as Net Logo. Ideally, the method and simulation should be run so that
centralized performance measures can be applied during the training process. To
demonstrate legitimate decentralized autonomous operation, it may be necessary to
distribute the software components, especially when testing autonomous adaptive
capabilities.
The heuristics developed for the method will be included in the final dissertation
document. An effort will be made to show that the method is scalable and
computationally tractable for practical applications. The motivation for this research is to
demonstrate the potential for this method as a means of developing intelligent controllers
for complex collaborative agent networks.
3.2.5 Experimentation Objective
There are a number of experiments that can be run using the method. The method can be
applied to a number of different scenarios where controller types are allocated in varying
ways to the agents in a heterogeneous system. These experiments can yield results such
as the size of the rule base evolved or the time taken to achieve optimality. Comparing
77
these results can serve as the basis for choosing one allocation approach over another.
Further experimentation can be done to modify learned behaviors in response to
unexpected changes in the goals and/or the environment. Sensor faults or agent
functionality can be altered and experiments can be run to see how the method adapts a
different solution to overcome these deficits. Modification of behavior within the
simulation is representative of the ability of agents to use this evolutionary mechanism
for adaptability in the real-world application. The only restriction is that the performance
measures occur in a manner repeatable in the real world. That is to say, a decentralized
performance measure would likely be required. Agents deployed in real application
environments may continue to evolve without decentralized performance measure if they
are allowed to periodically communicate or rendezvous to reevaluate and modify their
respective behaviors.
3.2.6 Instructional Objective
Another objective is the ability to extract the learned behavior rules from the evolved
control automaton. The probabilities stored for each state encode these learned behavior
rules. Datamining and logic reduction techniques can be used to determine the minimal
subset of conditions that impact certain system actions. These techniques can be used to
reduce the amount of non-essential genetic material required. The effect is to discern
individual automata for subsystem controls whenever possible. By studying these
evolved strategies, researchers can gain insight into how collaborative goals were
achieved. These developed strategies may be further introduced into other collaborative
agent systems.
78
Chapter Four: Results & Experiments
4.1 Method Development Results
This section will address the completion status of each subtask for each major objective
of the research. The means of decomposing a system into component actions and
conditions is well known. The subsequent initialization of a pool of potential controller
solutions has also been accomplished. Initial seeding of behaviors can be accommodated
by the current representation but additional work is needed to easily encode abstract
behavioral rules into the solution vector representations. The evolutionary method has
been well developed including the determination of the various process parameters
involved. The reinforcement learning technique during the simulation has been
determined. The means by which the simulation is integrated with the evolutionary
process is also well known. However, further experimentation with the simulation and
evolutionary process parameters is required. In particular, the use of modifiable goal
performance functions to achieve layers of complexity in system behavior must be further
explored. Applying datamining techniques to extract evolved strategies from the
controller solutions has yet to be accomplished. The developed method will be applied to
a small AGV manufacturing cell scenario to facilitate experimentation with process
parameters. To further refine the heuristics involved and demonstrate deployability of the
adaptive method, attempts will be made to apply it to more complex manufacturing cell
problems.
As is typically the case in studies of multi-agent systems with respect to
evolutionary learning, a small simple case study will be made to demonstrate the
79
effectiveness of the methodology. The problem that will be studied in this case is that of a
small manufacturing cell of sorts. It is comprised of an AGV corridor in which a series of
AGV's are allowed to move laterally from one cell to another without passing each other.
In front of the AGV's is a part or palette bench. The AGV’s unstack a part from this
bench and hold it in its gripper assembly. In this particular problem, the AGV's are
allowed to hold only one part at a time. The objective is to have these AGV’s
collaboratively transfer all parts in the system to the farthest bench cell to the right which
is referred to as the ‘dump cell’. Note that since AGV’s may not pass each other they
must rely on a strategic series of movements and part passing in order to accomplish this
global goal. Figure 4.1 depicts a generalized description of the aforementioned problem
with four cells and three agents.
Figure 4.1: Generalized Representation of Collaborative Multi-Agent Problem
This system scenario will also be depicted by the Figure 4.2 below. The numbers
in the top row signify the cell number. The second row signifies the number of parts in
80
each bench cell. The third row denotes the agents situated in their current cell position
(sometimes agents are also referred to with consecutive numerals instead of letters). The
bottom row represents the number of parts being carried by each agent in AGV corridor.
Figure 4.2: Alternate Representation of Problem
Each AGV will be drawing from an intelligent controller mechanism which is
essentially a set of rules which prescribe an action for a given conditional state of the
agent. If the agents are to be classified into separate groups with separate functionalities
then each identical agent within a group will be controlled by the same rule-set. Thus, the
evolutionary approach seeks to evolve multiple intelligent controllers to govern the
actions of numerous identical and non-identical agents within the system. The available
actions within this sample system are shown below as excerpted from the code:
Action(1).Enabled= True: Action(1).Name= "Ml= Move to the left cell"
Action(2).Enabled= True: Action(2).Name= "Mr= Move to the right cell"
Action(3).Enabled= True: Action(3).Name= "Tb= Take part from the bench"
Action(4).Enabled= True: Action(4).Name= "Gb= Give part to the bench"
Action(5).Enabled= True: Action(5).Name= "Dn= Do nothing"
Action(6).Enabled= True: Action(6).Name= "Tl= Take part from left cell carrier"
Action(7).Enabled= True: Action(7).Name= "Tr= Take part from right cell carrier"
Action(8).Enabled= True: Action(8).Name= "Gl= Give part left cell loiterer"
Action(9).Enabled= True: Action(9).Name= "Gr= Give part right cell loiterer"
There are nine actions available to any agent providing these are all made functional
to that agent’s group type. It is assumed that each agent in the system will be making
81
consecutive, orderly ‘moves’ within the system. Thus, within a simulation of the system,
with every clock tick the ability to make a move cycles through all the agents in a system
with only one agent at a time drawing from its intelligent controller to decide an action.
An action decision begins with having an agent assess its current conditional state within
its environment. This is accomplished by providing the agent with localized sensor
capability. In this scenario, each agent can sense if there is a part or not on the bench in
front of it, if there is a neighboring loiterer (non-carrying agent), neighboring carrier
(part-carrying agent), wall or void to the immediate left or right of it, or if the agent itself
is currently carrying a part or not. These sensor conditions are further described below
within the code:
Condition(1).Name = "Self Sensor": Condition(1).ChoiceCount = 2
Condition(1).ChoiceName(1) = "holding a part"
Condition(1).ChoiceName(2) = "not holding a part"
Condition(2).Name = "Bench Sensor": Condition(2).ChoiceCount = 2
Condition(2).ChoiceName(1) = "holding a part"
Condition(2).ChoiceName(2) = "not holding a part"
Condition(3).Name = "Left Sensor": Condition(3).ChoiceCount = 4
Condition(3).ChoiceName(1) = "holding a part"
Condition(3).ChoiceName(2) = "not holding a part"
Condition(3).ChoiceName(3) = "void"
Condition(3).ChoiceName(4) = "wall"
Condition(4).Name = "Right Sensor": Condition(4).ChoiceCount = 4
Condition(4).ChoiceName(1) = "holding a part"
Condition(4).ChoiceName(2) = "not holding a part"
Condition(4).ChoiceName(3) = "void"
Condition(4).ChoiceName(4) = "wall"
As shown above, the conditional state of an agent in the system can described at
any given time by the value of four conditions: Self Sensor, Bench Sensor, Left Sensor,
and Right Sensor. Given that the number of possible settings for each condition is 2, 2, 4
and 4 respectively, there are 2*2*4*4=64 total number of possible conditional states for
an agent in the system. Note that the number of configurations for the entire system is
much larger than this since a conditional state merely expresses the localized information
82
for just one single specific agent in a system comprised of many agents. The conditional
state describing the entire system would be an amalgam of all the individual controller
states. For each conditional state of a single agent, one of the aforementioned nine actions
will be prescribed. Thus, the maximum number of rules in an intelligent controller is 64
for this problem. But since any of these nine actions may be prescribed, there are 9^64
possible intelligent controllers for an agent in the system. Not all of these controllers may
lead to collaborative goal behavior when implemented with the other controllers for the
agents in the system. It is the interaction of these controllers by which the collaborative
behavior is expressed. In many multi-agent systems, it is this interaction which is too
complex to infer from human inspection of the system. This is the motivation of this
research: to devise a method which uses evolutionary learning and simulation to evolve
these collaborative controllers.
A sample evolved intelligent control mechanism for the system is shown below.
The two rule-sets (contained within brackets) are for a system with four agents and two
agent-type groups. There are two agents within each group. Thus, the first set of rules
governs the behavior of agents of the first type and the second set of bracketed rules
governs the behavior of the agents of the second type within the system. In this case,
there are 47 rules within each rule-set, though in practice the number of rules may differ.
This is less than the number of rules to cover all 64 possible conditional states for an
agent. The reason for this reduced set is that not all agents will encounter all 64
conditional states within the system during the simulations and thus not all rules are
applicable.
83
[ShBnLvRn-Gr SnBhLhRn-Tb ShBhLwRv-Mr SnBnLvRh-Ml SnBhLvRh-Tb SnBhLvRn-Tb ShBhLnRn-Gr
ShBnLnRv-Mr SnBhLwRn-Tb ShBnLvRh-Gb SnBhLwRv-Tb SnBnLvRn-Ml SnBnLhRv-Mr ShBnLhRv-Tl
SnBhLvRv-Ml SnBhLnRv-Tb SnBnLnRv-Mr SnBnLhRn-Tl ShBnLhRn-Gr SnBnLvRv-Mr SnBhLwRh-Tb
SnBnLwRv-Mr SnBnLwRn-Dn ShBnLnRn-Gr ShBhLvRn-Gr ShBnLvRv-Mr ShBnLwRv-Mr ShBnLwRn-Gr
SnBnLnRn-Dn ShBhLwRn-Gr SnBhLnRh-Tb SnBnLnRh-Dn SnBnLhRh-Tr ShBhLnRv-Mr SnBhLhRh-Tl
ShBnLnRh-Gl ShBhLwRh-Tr ShBhLhRn-Gr SnBhLnRn-Tb ShBhLnRh-Dn ShBnLwRh-Dn ShBhLvRv-Mr
SnBhLhRv-Tb ShBhLhRv-Gb SnBnLwRh-Dn ShBhLvRh-Dn ShBnLhRh-Gl ]
[SnBhLvRn-Ml SnBhLnRv-Tb ShBnLvRw-Dn SnBhLnRh-Dn SnBhLhRn-Tb SnBhLhRv-Tl SnBhLvRw-Ml
SnBhLvRh-Tb ShBhLnRv-Mr SnBhLhRh-Tl SnBhLhRw-Tl ShBhLvRn-Gr ShBhLnRw-Gb ShBhLvRv-Mr
ShBhLnRh-Gl ShBnLhRw-Gb ShBhLvRh-Dn ShBnLhRv-Mr ShBhLvRw-Gb ShBnLnRv-Mr SnBnLvRh-Ml
SnBnLnRh-Dn SnBnLhRv-Tl SnBnLnRv-Dn ShBhLnRn-Gr ShBhLhRw-Gb ShBnLnRw-Gb SnBnLhRh-Tl
SnBnLvRn-Ml ShBnLnRh-Dn ShBnLnRn-Gr SnBhLnRn-Tb SnBhLvRv-Ml ShBnLvRn-Ml ShBnLvRv-Ml
SnBnLhRn-Tl ShBnLhRh-Dn ShBhLhRn-Gr SnBnLvRv-Dn SnBhLnRw-Dn ShBhLhRh-Gb ShBhLhRv-Mr
SnBnLnRn-Dn ShBnLhRn-Gr ShBnLvRh-Gb SnBnLhRw-Dn SnBnLvRw-Ml ]
The rule “ShBnLvRn-Gr” signifies that when the conditions are such that, Sh (the
agent itself is holding a part) and Bn (the bench in front of it is empty) and Lv (there is an
empty void cell to the left of the agent) and Rn (there is a non-carrying agent to the right),
then the agent should implement the action Gr (Give the part to the agent on the right).
This is a sound rule given that the objective is to transfer all parts in the system to the
dump cell on the right. By giving the part to the agent on its right this action frees itself to
move left in search of more parts to transfer right. Meanwhile, the newly part-acquired
agent on the right can query its controller with its new conditional state and determine
what action it should now take. Cycling through the agents in this manner, allowing each
to take a single consecutive action, is how the collaborative goal of transferring all parts
to the dump cell can be achieved.
Note that although we are dealing with a simple system with elemental atomic
actions, each simple action could in theory be replaced with a call to a procedure which
could enact a more complicated form of behavior. In this sense, the approach is scalable
to larger more complicated problems. Although this simple problem has 64 conditional
states (and therefore maximally 64 rules in a rule-set), it is easy to see how an increase in
84
the number of sensor conditions as well as an increase in the range of possibilities for
each condition would explode the number of rules that could exist in the system thereby
making this approach intractable for larger problems. However, the evolutionary
approach devised and shown below alleviates this problem by only evolving the rules
necessary to govern the individual agent behavior in the system. Additionally, the number
of rules necessary to govern behavior is a smaller subset of the number of all possible
rules.
The method devised must also be robust with respect to changes in the initial
system configuration and also with respect to overcoming sensor failures. The program
written accommodates both eventualities. It is also possible to seed the agent controllers
with certain rules and allow a new controller to adapt from these starting rules. This is a
useful technique towards layering complexity on top of an already evolved system when
a new function or goal is introduced.
As discussed above, a rule consists of a conditional state for an agent as well as a
prescribed action. In the example above, there are 64 possible rules per controller. How is
one to evolve the multiple collaborative controllers that govern the behavior in a system?
How are these rules to be encoded in such a way that an evolutionary process may
operate on them? How can this method maintain scalability and tractability?
Approach I
The first approach considered was to use conventional genetic algorithms where a string
encodes all 64 rules of each controller in the system. The rules would be ordinally
arranged within each rule-set string so that the function of the rule and its position within
85
the genetic string would always be correlated. This in theory would allow for
conventional genetic algorithms to operate on the string. The fitness measure would be
determined by implementing these rules within a system simulation for a certain number
of time ticks and finding some measure of fitness. The best solution strings would
reproduce and form a new generation of solution strings. This approach would in theory
work, but as previously mentioned, the string size vastly expands with larger systems and
the number of generations required to accomplish evolution would be computationally
intractable.
Approach II
An alternate approach attempted was a decision tree model but this too had similar
problems. In the decision tree approach, a single condition such as the Bench-sensor
condition was placed at the root of the tree. A ‘holding’ value for this condition sent the
decision-making process down the left branch of the sub-tree while a ‘non-holding’ value
of this condition sent the decision-making process down the right-hand side of the sub-
tree. Similar simple conditions existed at other branches of the tree. Ideally, by the time a
leaf node was reached the system had processed all the simple condition testing necessary
to decide on the execution of a particular action stored at the leaf node. The drawback to
this approach was that number of branches could grow immeasurably without restriction
and conditions were often unnecessarily repeated along a branch. The number of
generations necessary to evolve a tree which contained the minimal number of necessary
conditional state tests proved to be equally difficult as the first approach.
86
Approach III: Best/Selected
It was decided that a controller data structure would be used which would store rules
associated with only those conditional states which were encountered by the agent within
the simulation run. There was no need to learn and store rules for states that the agent
would never or could never reach throughout the course of its operation. Thus, a learning
string structure was devised that expands as is necessary to control the agent’s operations.
This approach is well-suited for larger problems which, due to the increased number of
conditions, may have an exceedingly high number of possible conditional states but never
actually experience the majority of them during operations.
The program posed many challenges such as the development of the data structure
for the controller, the numerous genetic parameters, the simulation, the fitness functions,
the accommodation of sensor failure, of controller seeding, etc.
The method for evolving controllers for each of the agents in the system is as
follows. Create an initial genepool with many solutions (“organisms”), where each
solution is a set of blank rule-sets (“chromosomes”), one for each agent type. Each
solution in the gene pool must be tested within the simulation to obtain a fitness measure.
The first time tick of the simulation prompts the first agent to evaluate its conditional
sensor state within the system. A corresponding rule is associated with this determined
conditional state. It then queries its associated controller string for the prescribed action
to this rule. If this rule is present within the rule-set then the associated action is taken. If
this rule is not present in the string then a random action is taken and the new rule is
87
stored within the controller string. The simulation then enacts the agent’s action and the
effects of the action on the simulation agents and environment are realized. The
simulation then prompts the next agent to make its move in a similar manner. The
simulation cycles through all the agents in this manner and then repeats the process for a
pre-specified number of cycles within the simulation. This process slowly builds each
controller string as each of the agents in the system associated with the controller is
exposed to more and more conditional states. Not all implemented actions will lead to
acceptable behavior within the system. Most random actions lead to a failure, such as
taking a part from the bench when it isn’t there, or moving left into a wall, or carrying
more than one part, etc. The program is able to detect failure states and the rules leading
to system failures are excised from the controller strings as they occur. Throughout the
simulation process, failed moves do not stop the system but rather the effects are undone,
the failed action is counted as a non-move and the agent loses its chance to make a
successful action at that step in the simulation. This approach allows the simulation phase
to keep a count of the number of failed moves during a run for each solution string-tuple.
After a simulation run, the fitness test is measured by a combination of three factors: the
number of legal steps taken, the value of a scoring function which weighs the part-flow in
the system, and a measure of how quickly the goal state was reached. These three
objectives provide a composite fitness measure for each prospective solution in the gene
pool. This value is a measure of how well the evolved controllers representing a single
solution work together towards achieving the collaborative goal. Once a fitness measure
is obtained for each solution in the genepool, it is used during the genetic recombination
phase of the method to produce the next generation of solutions. Genetic operators such
88
as mutation, inter-chromosome mating and intra-chromosome mating are utilized. These
operators had to be tailored to fit the unusual format of the solution strings given that
substring position and substring function were no longer correlated as they are in
standard genetic algorithms.
The number of actions available to an agent-type group can be adjusted:
ActionCount = 9
Each action can be enabled or disabled thus showing how the method evolves for
different functionalities:
Action(i).Enabled = True
The initial system configuration is specified as follows:
1. standard problem:
InitBenchString = "302110"
InitAgentString = "012340"
InitCarryString = "010010"
2. harder problem:
InitBenchString = "0211002100"
InitAgentString = "1020340567"
InitCarryString = "0010000010"
The program uses the above initial strings to determine the cell count, agent
count, part count, agent positions and agent carry status.
The assignment of agents into agent-type groups dictates the number of
controllers to evolve. The bulk of the experiments run so far demonstrate the differences
in rule-sets and rule-set sizes that arise from different agent group designations. Four
specific controller designations were studied for the above problem: ‘Universal’,
‘Divided’, ‘Bookend’, and ‘Individual’. The ‘Universal plan’ has all agents in the system
89
being designated as from the same agent group. Thus, each agent is governed by the same
controller and in turn, this controller evolves from the experiences of every agent in the
system. Therefore, a universal controller is likely to increase in the number of rules
during the learning phase faster than the other approaches since it is exposed to the
conditional states of all agents within the system simulation. The ‘Divided plan’ splits the
example AGV system in half with agents A and B governed by one controller and agents
C and D governed by a second rule-set. Thus, there are really only two controllers
interacting within the simulation, each governing two agents. The ‘Bookend plan’ has
three controllers including a unique controller for each side agent as well as a single
controller for all AGV agents in the middle. The bookend approach would seem to work
well for this example four-agent problem because unlike agents A and D, agents B and C
never need to encounter a wall thus all rules including wall-sensing are unnecessary for
these controllers. It makes sense to convolve the control schemes for agents B and C into
the same controller since there functionality within the system is identical. The
‘Individual plan’ provides a separate controller assigned to each agent in the system. In
this scenario, an agent cannot benefit from the learning experienced by the other agents in
the system because their respective rule-sets evolve based only on their own experiences
in the simulation.
The number of rules and the nature of the rules evolved by this evolutionary
method for each of the above controller designations are of principal interest in this
research project. The results of the simulations for different controller plans could be
useful in dictating the ratio of agent type designations to implement within a system.
90
Specific sensor equipment failure may be implemented to see how the
evolutionary process circumvents it:
AgentOneSenseRightFaulty = False
In this case, the right cell sensor for agent A is made faulty. Thus, during the
simulation runs, it cannot reliably return the proper conditional assessment of what is to
the right of agent A. This ability to implement failures allows for the study of how the
process evolves different rule-sets to circumvent failures.
Simulation parameters:
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
Genetic parameters:
GenerationCount = 500
SolutionCount = 500 '1000
Fitness parameters:
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0 'BenchScoreBase * CellCount
InitConfigScore = 0
For i = 1 To CellCount
InitConfigScore = InitConfigScore + Mid(InitBenchString, i, 1) * BenchScoreBase ^ i
InitConfigScore = InitConfigScore + Mid(InitCarryString, i, 1) * CarryScoreBase ^ i
Next i
InitConfigScore = InitConfigScore + Right(InitBenchString, 1) ^DumpScoreCoeff
MaxConfigScore = MaxDumpCount * (BenchScoreBase ^ CellCount + DumpScoreCoeff)
Reproduction Parameters:
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossCount = 2 * Int(RouletteCrossPercentage * SolutionCount / 2)
OutCrossCount = 2 * Int(OutCrossPercentage * SolutionCount / 2)
RandomCrossCount = 2 * Int(RandomCrossPercentage * SolutionCount / 2)
EliteChoiceCount = Int(EliteChoicePercentage * SolutionCount)
91
WorstChoiceCount = Int(WorstChoicePercentage * SolutionCount)
RandomChoiceCount = ToolMax(SolutionCount – EliteChoiceCount
WorstChoiceCount - RandomCrossCount - RouletteCrossCount
OutCrossCount - RandomEliteChoiceCount - RandomWorstChoiceCount, 0)
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1
RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
4.2 Overview of Experiments
The first set of experiments are homogeneous in nature with respect to sensing capability
and available actions. They have all nine actions enabled with four different controller
designation plans implemented. These experiments will illustrate the effect of assigning
different controller designations on overall efficiency of the agent system.
The second set of experiments are also homogeneous but have differing number of the
nine actions enabled. These experiments will demonstrate how the controllers evolve
different schemes when their atomic actions are altered and how this ultimately affects
system efficiency. This could, perhaps, be used to show the minimal subset of atomic
actions necessary for a multi-agent system to successfully accomplish goal behavior.
The third set of experiments are heterogeneous in nature.
Each experimental section is divided into a number of sections. A diagram of the
same standard AGV corridor problem is cited with its initial configuration reflected. A
settings section follows which include the various value settings for the method for that
experiment. The next section shows the ‘generational process’ of the method application.
92
Each line (example: 1 S= 800 D= 4 PGS= 800 LS= 679 CFG= 3681 SIM= 4360 LG)
shows the fitness attributes of the top solution within the genepool for that generation.
The variables are as follows:
S=simulation steps,
PGS=steps taken before goal reached,
LS=legal steps out of the 800 maximum,
CFG=weighted fitness measure reflecting part transfer progress,
SIM=CFG+LS+(S-PGS)= overall fitness measure to be maximized.
The last variable, SIM, represents the results of the last generation of the method
and the final fitness it produced.
The next section shows the simulation output for the final optimal controller that
was evolved. Simulation output is reflected in the status of the bench, the AGV corridor
and the AGV carry status at each time tick in the simulation as demonstrated below. The
trace also shows the agent number and the last agent action performed at any given time
tick. For example, the fourth block represents the state of the system after the 2nd agent
(or agent B) finished performing a “Gr” action in which it passed its part to neighboring
agent #3. This trace output continues for all steps in the simulation until the rightmost cell
of the bench is filled with all system parts which signifies goal completion.
TRACE: ORIG 1Gr 2Gr 3Gb 4Mr 1Ml 2Tb 3Dn …
BENCH: 302110 302110 302110 302210 302210 302210 301210 301210 …
AGENT: 012340 012340 012340 012340 012304 102304 102304 102304 …
CARRY: 010010 001010 000110 000010 000001 000001 001001 001001 …
93
The subsequent section discusses these results and details the controller rule-sets
that were evolved. Each rule-set is in the form:
[SnBnLvRn-Ml SnBhLvRn-Tb SnBhLwRv-Tb ShBnLvRn-Gr ShBnLvRh-Gb …………].
Within the graph analysis section of these results there will be three figures. The
first figure will reflect the fitness scores of every fifth solution within an ordered
genepool for every generation up until the last generation. The second figure will reflect
only the fitness of the top solution in every genepool. This is also known as the
generational fitness. The third figure will show the fitness function components and
reflects how fitness is being achieved simultaneously with respect to legality, quickness
and part-transfer.
Following the sections detailing the experiments, a section for discussion of the
experimental results is included.
4.3 Experiment 1: Midsize / Nine-Action / Individually-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has all 9 actions enabled. Additionally, this experiment has each
agent relying on its own set of controller rules. Thus, each agent’s rule-set is evolved
pursuant to the particular agent’s experiences within the simulation training. This is the
‘individual’ controller designation plan.
94
Figure 4.3: Initial Configuration of System for Experiment 1
Experiment 1: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = True: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = True: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = True: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = True: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = "Individual"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
95
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 1: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 4 PGS= 800 LS= 679 CFG= 3681 SIM= 4360 --
2 S= 800 D= 5 PGS= 800 LS= 794 CFG= 3897 SIM= 4691 --
3 S= 800 D= 5 PGS= 800 LS= 798 CFG= 3897 SIM= 4695 --
4 S= 800 D= 6 PGS= 800 LS= 762 CFG= 4485 SIM= 5247 --
5 S= 800 D= 6 PGS= 800 LS= 784 CFG= 5187 SIM= 5971 --
6 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3897 SIM= 4697 L-
7 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3897 SIM= 4697 L-
8 S= 800 D= 7 PGS= 800 LS= 793 CFG= 5355 SIM= 6148 --
9 S= 800 D= 7 PGS= 800 LS= 769 CFG= 5139 SIM= 5908 --
10 S= 800 D= 8 PGS= 800 LS= 782 CFG= 6075 SIM= 6857 --
11 S= 800 D= 9 PGS= 799 LS= 795 CFG= 6561 SIM= 7357 --
12 S= 800 D= 8 PGS= 800 LS= 791 CFG= 6561 SIM= 7352 --
13 S= 800 D= 9 PGS= 799 LS= 800 CFG= 6561 SIM= 7362 LG
14 S= 800 D= 9 PGS= 799 LS= 800 CFG= 6561 SIM= 7362 LG
15 S= 800 D= 9 PGS= 799 LS= 800 CFG= 6561 SIM= 7362 LG
16 S= 800 D= 9 PGS= 799 LS= 800 CFG= 6561 SIM= 7362 LG
17 S= 800 D= 9 PGS= 131 LS= 791 CFG= 6561 SIM= 8021 --
18 S= 800 D= 9 PGS= 131 LS= 793 CFG= 6561 SIM= 8023 --
19 S= 800 D= 9 PGS= 131 LS= 793 CFG= 6561 SIM= 8023 --
20 S= 800 D= 9 PGS= 107 LS= 799 CFG= 6561 SIM= 8053 --
21 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
22 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
23 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
24 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
25 S= 800 D= 9 PGS= 91 LS= 799 CFG= 6561 SIM= 8069 --
26 S= 800 D= 9 PGS= 87 LS= 800 CFG= 6561 SIM= 8074 LG
96
27 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
28 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
29 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
30 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
31 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
32 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
33 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
34 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
35 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
36 S= 800 D= 9 PGS= 75 LS= 797 CFG= 6561 SIM= 8083 --
37 S= 800 D= 9 PGS= 75 LS= 799 CFG= 6561 SIM= 8085 --
38 S= 800 D= 9 PGS= 75 LS= 799 CFG= 6561 SIM= 8085 --
39 S= 800 D= 9 PGS= 71 LS= 799 CFG= 6561 SIM= 8089 --
40 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
41 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
42 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
43 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
44 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
45 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
46 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
47 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
48 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
49 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
50 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
51 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
52 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
53 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
54 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
55 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
56 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
57 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
58 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
59 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
60 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
61 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
62 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
63 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
64 S= 800 D= 9 PGS= 55 LS= 800 CFG= 6561 SIM= 8106 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 8106 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 55 legal steps within the simulation using the final evolved rule-sets
shown below. 55 steps is one step shy of 14 cycles within the simulation. This solution
was found in the 64th generation. The four agents each take a consecutive step to
complete one cycle. Note how the fitness measure allows for multi-criteria evolution to
97
occur. At times the evolutionary process will lean towards transferring more parts to the
dump cell even if it uses more illegal steps to accomplish this. Other times the
evolutionary process will seek to make all steps legal at the expense of goal progress.
Over time, the effect is to get better at each objective and finally to settle down into an
optimal, legal solution.
Experiment 1: Final Results G: 64 SIMSCORE= 8106 CFGSCORE= 6561 GOALS= 9
LEGALSTEPS= 800 PostGoalSTEPS= 745 STEPS= 800
An output listing below shows the trace for each of the 55 steps. It shows the step
action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
TRACE: ORIG 1Gr 2Gr 3Gb 4Mr 1Ml 2Tb 3Dn 4Gb 1Tb 2Gr 3Mr 4Tl 1Mr 2Tl 3Ml
4Gb 1Ml 2Gr 3Mr 4Tl 1Tb 2Tb 3Ml 4Gb 1Mr 2Gr 3Mr 4Tl 1Gr 2Mr 3Tl 4Gb 1Ml
2Tb 3Gr 4Gb 1Tb 2Gr 3Gr 4Gb 1Mr 2Tb 3Tl 4Tl 1Mr 2Dn 3Tb 4Gb 1Gr 2Gl 3Gr
4Gb 1Gr 2Gr 3Gr 4Gb 1Ml 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml
2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn
4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr
2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn
4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml
2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
98
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn
4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn
1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn
3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn
1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn 3Dn 4Dn 1Mr 2Dn 3Dn 4Dn 1Ml 2Dn
3Dn 4Dn
BENCH: 302110 302110 302110 302210 302210 302210 301210 301210 301211 201211 201211 201211 201211
201211 201211 201211 201212 201212 201212 201212 201212 101212 100212 100212 100213 100213 100213
100213 100213 100213 100213 100213 100214 100214 100114 100114 100115 000115 000115 000115 000116
000116 000016 000016 000016 000016 000016 000006 000007 000007 000007 000007 000008 000008 000008
000008 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
99
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009
AGENT: 012340 012340 012340 012340 012304 102304 102304 102304 102304 102304 102304 102034 102034
012034 012034 012304 012304 102304 102304 102034 102034 102034 102034 102304 102304 012304 012304
012034 012034 012034 010234 010234 010234 100234 100234 100234 100234 100234 100234 100234 100234
010234 010234 010234 010234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100
100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234
100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234 010234 010234
010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234 100234 100234
010234 010234 010234 010234 100234 100234 100234 100234 010234 010234 010234 010234 100234 100234
100234 100234
CARRY: 010010 001010 000110 000010 000001 000001 001001 001001 001000 101000 100100 100010 100001
010001 001001 001001 001000 001000 000100 000010 000001 100001 101001 101001 101000 011000 010100
010010 010001 001001 000101 000011 000010 000010 000110 000101 000100 100100 100010 100001 100000
010000 010100 010010 010001 001001 001001 001011 001010 000110 001010 001001 001000 000100 000010
000001 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
101
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
Experiment 1: Evolved Rule-Sets
Below are shown the evolved rule-sets for the system. Since the individual controller plan
was used, a distinct rule-set for each agent was evolved based on the experiences of that
agent in the simulations. Each rule-set below is preceded by a number reflecting the
number of rules that were evolved within that rule-set. Each rule is of the format
‘C1C2C3C4-Action’ where a series of condition settings have an associated subscribed
action. Note that the total number of possible rules is equal to the total number of
possible condition states which in this problem is 64. The evolution of a reduced number
102
of rules is a significant improvement over trying to pre-determine all 64 rules for each
agent outside of an evolutionary learning process. Note that Agent A’s rule-set never has
a rule with a “Rw: wall to the right” condition since its leftmost position in the corridor
prevents it from ever encountering the right wall during the course of the simulation. This
is due to agents being unable to pass each other in the corridor. Each agent is therefore
limited in its scope of experience and this in turn influences which rules evolve for it. It
will be shown that in subsequent experiments in which an alternative controller plan is
used, the experiences of multiple agents sourced by a common controller are shared and
are used towards the evolutionary shaping of that rule-set. Note also that the disabling of
side-to-side taking and giving actions will force agents to rely on the more awkward and
laborious bench passing and cell shifting in order to accomplish part transfer in the
system. The rule-sets below are a reduced set of rules necessary to elicit optimal
collaborative performance within the simulation. Operating the agents individually by
these rules does not result in optimal system performance. It is the necessary inter-
dependence that is captured by these evolved rule-sets which leads to collaboration.
The four controllers were evolved in 64 generations. These controllers
operating together took 55 steps in the simulation to reach the collaborative goal.
The number of rules evolved across the four controllers 120.
# 24 [SnBnLvRn-Ml SnBhLvRn-Tb SnBhLwRv-Tb ShBnLvRn-Gr ShBnLvRh-Gb SnBhLwRh-Tr
SnBnLvRv-Ml SnBhLvRv-Tb ShBhLvRn-Mr ShBhLvRv-Mr ShBnLwRv-Mr ShBnLwRn-Gr SnBnLwRn-
Dn ShBnLwRh-Dn ShBnLvRv-Mr SnBhLwRn-Dn ShBhLwRn-Gb SnBnLvRh-Ml ShBhLwRv-Mr
SnBnLwRv-Mr ShBhLvRh-Tl SnBhLvRh-Tb ShBhLwRh-Dn SnBnLwRh-Dn ]
# 36 [SnBhLhRh-Tl SnBhLvRh-Tb SnBhLhRn-Tb ShBnLhRv-Mr SnBhLnRh-Gr ShBhLvRn-Gr
SnBnLvRh-Ml ShBnLvRn-Dn SnBhLnRn-Tb SnBhLvRv-Tb SnBnLhRn-Dn SnBnLvRn-Dn ShBhLvRh-
Dn ShBnLnRn-Gr SnBnLnRn-Dn SnBnLnRv-Dn ShBhLnRh-Gb SnBnLnRh-Tr SnBnLhRv-Mr
SnBhLvRn-Tb SnBnLvRv-Mr ShBhLvRv-Mr ShBhLhRn-Dn ShBhLnRn-Gr SnBnLhRh-Dn ShBnLhRn-
Gr ShBhLnRv-Dn SnBhLnRv-Tl ShBnLnRh-Gl SnBhLhRv-Tl ShBnLvRv-Ml ShBhLhRv-Gb ShBnLvRh-
Gb ShBnLhRh-Gb ShBhLhRh-Gb ShBnLnRv-Mr ]
103
# 36 [SnBhLhRn-Tl SnBhLhRh-Tl SnBhLhRv-Dn ShBhLvRh-Ml SnBhLnRv-Tb ShBhLnRh-Gb
ShBhLnRv-Mr ShBhLnRn-Gr SnBhLnRn-Tb SnBnLnRv-Mr ShBnLnRv-Dn SnBnLvRn-Ml ShBhLhRv-
Mr SnBnLhRh-Tl ShBnLhRv-Dn SnBnLvRh-Dn SnBhLvRh-Ml ShBnLvRn-Gb ShBnLnRn-Gr
SnBnLhRn-Tl SnBnLnRh-Dn SnBhLvRv-Tb SnBnLhRv-Ml ShBhLvRv-Gb ShBnLnRh-Gb ShBnLvRv-
Dn ShBnLhRn-Gr SnBnLvRv-Ml SnBhLnRh-Tb ShBhLvRn-Ml ShBnLvRh-Dn SnBnLnRn-Dn
SnBhLvRn-Tb ShBhLhRn-Gr ShBnLhRh-Dn ShBhLhRh-Dn ]
# 24 [ShBnLvRw-Gb SnBhLhRv-Tb SnBhLvRw-Ml ShBhLnRv-Mr ShBhLhRw-Gb ShBhLvRw-Gb
SnBnLhRv-Tl SnBhLnRw-Dn ShBnLhRv-Mr ShBnLnRv-Mr SnBhLvRv-Ml SnBnLnRv-Dn SnBnLvRw-
Ml ShBhLvRv-Tr SnBnLvRv-Tb ShBhLnRw-Gb SnBhLhRw-Tl SnBnLhRw-Tl ShBhLhRv-Mr
ShBnLhRw-Gb ShBnLvRv-Gb ShBnLnRw-Tr SnBhLnRv-Tb SnBnLnRw-Gr ]
Note that this is the result of just one application of the method. No two runs are
alike. Subsequent runs would evolve slightly different solutions to the problem although
the main strategy would be the same. Efficiency and size of the rulesets would also be
approximately the same providing all initial conditions remain the same. These evolved
rulesets depend greatly on the giving and taking of parts between agents as signified by
the presence of many “Gr”, “Gl”, “Tl” and “Tr” actions in the rulesets. In the 5-action
experiments detailed later, these agent-passing actions are disabled which forces the
system to evolve a different means of achieving the collaborative goal behavior.
Experiment 1: Graph Analysis
Figure 4.4 below shows the fitness score of every fifth solution ranked low to high within
the 500-count solution gene pool in every generation leading up to the final 64th
generation. This graph reflects the changing nature of the gene pool throughout the
evolutionary process.
104
Figure 4.4: Evolving Fitness Scores for Experiment 1 (Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.5: Generational Fitness for Experiment 1
(Y-axis: Fitness Measure, X-axis: Generation)
1 7
13
19
25
31
37
43
49
55
61
S1
S19
S37
S55
S73
S91
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64
105
Figure 4.6 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state with all parts transferred to the dump
cell. The blue line denotes the maximum number of steps in the simulation (800) minus
the pre-goal steps. It is desired that this quantity be maximized. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a failure
(i.e. are legal). The third component is the configscore (cfg) which for this problem is
maximally 6561, a weighted value which reflects all parts in the system having been
transferred to the dump cell. This value is denoted by the yellow line. These three values
are divided by their maximum values and the ratios are plotted below. Note that not all
solutions leading to a goal are achieved through legal steps (yellow optimal but pink
suboptimal). Note also that in early generations, a solution can be comprised entirely of
legal steps and yet not achieve goal behavior (pink optimal but yellow suboptimal). Note
that it isn’t until generation 15 that the process is able to achieve goal behavior (blue line
goes non-zero). Note also that after a solution is found which is both legal and goal-
reaching (pink and yellow optimal), that the evolutionary method continues to evolve
solutions that achieve this goal in fewer and fewer steps (the blue line subtly increases).
As the method attempts to adjust its collaborative behavior towards reaching the goal in
fewer steps, note that it sometimes results in a solution that lapses into illegality (roughly
generation #35 below where pink goes suboptimal). This lack of fitness in illegality is
made up for temporarily by the forming of new rules which reduce the number of steps
needed to accomplish the system goal. Note that subsequent to this generation, the
106
process goes back to restoring legality while maintaining the improved rules towards
fewer steps to the goal. The method successfully and autonomously negotiates these three
fitness measure components so that eventually a legal, optimal solution finds the goal in a
minimal number of steps. Note that every solution from generation ~18 onwards (except
for generation 35) has a non-zero blue line and optimal pink and yellow lines. These
solutions are all acceptable solutions to the problem. They merely implement a different
set of rules perhaps in a different order in order to achieve the collaborative goal
behavior. Allowing the process to continue results in a more effective rule-set that finds
the solution more quickly.
Figure 4.6: Scores of Three Components of Fitness for Experiment 1 (Y-axis: Coefficient of Optimality, X-axis: Generation Index)
0
0.25
0.5
0.75
1
1 21 41 61
(max-pgs)/max (ls/max) (cfg/max)
107
4.4 Experiment 2: Midsize / Nine-Action / Bookend-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the experiment settings
section below. This experiment has all 9 actions enabled.
The agents in this experiment are not all governed by their own individual
controller as they were in the previous experiment. In this ‘bookend’ approach, the end
agents A and D each get their own controller. The middle agents B and C will be
sourcing a common controller. This is a reasonable approach given that B and C are
exposed to similar conditional states whereas A and D experience conditional states
unique to each other as well as to the central agents. Thus, there are three controllers to
be evolved in this system. The second controller draws from the experiences of both
agent B and C. The ability of the method to specify the sourcing controller for each agent
in the system is useful for systems in which collaboration between two or more distinct
agent groups is desired.
Figure 4.7: Initial Configuration of System for Experiment 2
108
Experiment 2: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = True: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = True: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = True: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = True: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = “Bookend"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1
RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
109
Experiment 2: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 6 PGS= 800 LS= 733 CFG= 4383 SIM= 5116 --
2 S= 800 D= 4 PGS= 800 LS= 771 CFG= 3417 SIM= 4188 --
3 S= 800 D= 5 PGS= 800 LS= 787 CFG= 3735 SIM= 4522 --
4 S= 800 D= 5 PGS= 800 LS= 778 CFG= 4413 SIM= 5191 --
5 S= 800 D= 6 PGS= 800 LS= 791 CFG= 4419 SIM= 5210 --
6 S= 800 D= 6 PGS= 800 LS= 767 CFG= 4455 SIM= 5222 --
7 S= 800 D= 9 PGS= 227 LS= 774 CFG= 6561 SIM= 7908 --
8 S= 800 D= 9 PGS= 135 LS= 796 CFG= 6561 SIM= 8022 --
9 S= 800 D= 9 PGS= 135 LS= 796 CFG= 6561 SIM= 8022 --
10 S= 800 D= 9 PGS= 135 LS= 797 CFG= 6561 SIM= 8023 --
11 S= 800 D= 9 PGS= 135 LS= 798 CFG= 6561 SIM= 8024 --
12 S= 800 D= 9 PGS= 119 LS= 783 CFG= 6561 SIM= 8025 --
13 S= 800 D= 9 PGS= 175 LS= 783 CFG= 6561 SIM= 7969 --
14 S= 800 D= 9 PGS= 95 LS= 794 CFG= 6561 SIM= 8060 --
15 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
16 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
17 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
18 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
19 S= 800 D= 9 PGS= 83 LS= 796 CFG= 6561 SIM= 8074 --
20 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
21 S= 800 D= 9 PGS= 79 LS= 800 CFG= 6561 SIM= 8082 LG
22 S= 800 D= 9 PGS= 75 LS= 797 CFG= 6561 SIM= 8083 --
23 S= 800 D= 9 PGS= 75 LS= 797 CFG= 6561 SIM= 8083 --
24 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
25 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
26 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
27 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
28 S= 800 D= 9 PGS= 71 LS= 798 CFG= 6561 SIM= 8088 --
29 S= 800 D= 9 PGS= 71 LS= 799 CFG= 6561 SIM= 8089 --
30 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
31 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
32 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
33 S= 800 D= 9 PGS= 63 LS= 793 CFG= 6561 SIM= 8091 --
34 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
35 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
36 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
37 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
38 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
39 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
40 S= 800 D= 9 PGS= 59 LS= 799 CFG= 6561 SIM= 8101 --
110
41 S= 800 D= 9 PGS= 59 LS= 799 CFG= 6561 SIM= 8101 --
42 S= 800 D= 9 PGS= 63 LS= 800 CFG= 6561 SIM= 8098 LG
43 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
44 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
45 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
46 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
47 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
48 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
49 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
50 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
51 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
52 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
53 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
54 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
55 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
56 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
57 S= 800 D= 9 PGS= 55 LS= 799 CFG= 6561 SIM= 8105 --
58 S= 800 D= 9 PGS= 55 LS= 799 CFG= 6561 SIM= 8105 --
59 S= 800 D= 9 PGS= 55 LS= 800 CFG= 6561 SIM= 8106 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 8106 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 55 legal steps within the simulation using the final evolved rule-sets
shown below. 55 steps is one step shy of 14 cycles within the simulation. The four agents
each take a consecutive step to complete one cycle. Note how the fitness measure allows
for multi-criteria evolution to occur. At times the evolutionary process will lean towards
transferring more parts to the dump cell even if it uses more illegal steps to accomplish
this. Other times the evolutionary process will seek to make all steps legal at the expense
of goal progress. Over time, the effect is to get better at each objective and finally to
settle down into the optimal, legal solution.
Experiment 2: Final Results
G: 67 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 55 POSTGOALSTEPS= 745
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 8106 LG
111
The output below shows the trace for each of the 55 steps. It shows the step
action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
TRACE: ORIG 1Gr 2Gr 3Gb 4Mr 1Dn 2Tb 3Dn 4Gb 1Ml 2Gr 3Mr 4Tl 1Tb 2Ml 3Tb
4Gb 1Gr 2Mr 3Gr 4Gb 1Tb 2Dn 3Ml 4Ml 1Mr 2Gr 3Gr 4Mr 1Gr 2Gr 3Mr 4Gb 1Dn
2Tb 3Gr 4Gb 1Ml 2Mr 3Tl 4Tl 1Tb 2Tb 3Tl 4Gb 1Mr 2Tb 3Gr 4Gb 1Mr 2Gr 3Gr
4Gb 1Gr 2Gr 3Gr 4Gb 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn
112
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
BENCH: 302110 302110 302110 302210 302210 302210 301210 301210 301211 301211 301211 301211 301211
201211 201211 201201 201202 201202 201202 201202 201203 101203 101203 101203 101203 101203 101203
101203 101203 101203 101203 101203 101204 101204 100204 100204 100205 100205 100205 100205 100205
000205 000105 000105 000106 000106 000006 000006 000007 000007 000007 000007 000008 000008 000008
000008 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
113
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
AGENT: 012340 012340 012340 012340 012304 012304 012304 012304 012304 102304 102304 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 102034 102304 102340 012340 012340
012340 012304 012304 012304 012034 012034 012034 012034 012034 012034 102034 100234 100234 100234
100234 100234 100234 100234 010234 010234 010234 010234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
114
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 001234 001234
CARRY: 010010 001010 000110 000010 000001 000001 001001 001001 001000 001000 000100 000010 000001
100001 100001 100011 100010 010010 001010 001001 001000 101000 101000 101000 101000 011000 010100
010010 010001 001001 000101 000011 000010 000010 001010 001001 001000 001000 000100 000010 000001
100001 100101 100011 100010 010010 010110 010101 010100 001100 001010 001001 001000 000100 000010
000001 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
115
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
Experiment 2: Evolved Rule-Sets
The three controllers were evolved in only 59 generations. These controllers operating
together took 55 steps in the simulation to reach the collaborative goal. The number of
rules evolved for all controllers is 84. Note that the agents central B and C are governed
by the single middle controller. The rules in the second controller are sufficient for the
control of actions of both B and C agents within the simulation.
# 24 [ShBhLwRv-Mr SnBnLvRh-Ml SnBhLwRv-Tb SnBhLvRn-Tb ShBnLvRv-Mr SnBhLvRh-Ml
SnBhLwRn-Tb ShBnLvRn-Gr SnBnLwRn-Dn SnBnLwRv-Mr SnBnLvRv-Ml ShBnLwRn-Dn
SnBhLvRv-Dn SnBhLwRh-Tb ShBhLvRn-Gr ShBnLwRv-Mr ShBnLvRh-Dn ShBhLvRh-Ml ShBhLwRh-
Dn ShBhLvRv-Dn ShBnLwRh-Gb SnBnLwRh-Dn ShBhLwRn-Gr SnBnLvRn-Dn ]
# 36 [ShBhLnRn-Gr SnBhLhRh-Tb SnBhLhRn-Tl SnBhLvRn-Tb SnBnLhRn-Tl ShBhLnRh-Gb
ShBnLhRv-Mr ShBnLvRn-Gr ShBhLhRn-Gr ShBnLnRn-Gr SnBnLnRv-Mr SnBnLnRn-Dn SnBhLvRv-
Ml SnBhLnRn-Tb SnBnLvRh-Dn ShBnLvRh-Dn ShBhLvRv-Dn SnBnLvRn-Ml SnBnLvRv-Ml
ShBnLvRv-Mr ShBhLnRv-Mr SnBhLhRv-Dn SnBnLhRh-Tl ShBnLnRh-Dn SnBhLnRv-Tb ShBnLhRn-
Gr SnBnLhRv-Dn ShBnLhRh-Gb ShBhLvRn-Gr SnBnLnRh-Dn SnBhLnRh-Dn SnBhLvRh-Tb
ShBnLnRv-Mr ShBhLvRh-Ml ShBhLhRv-Mr ShBhLhRh-Dn ]
# 24 [ShBhLnRv-Mr ShBnLvRw-Gb SnBhLhRv-Tb SnBhLhRw-Tl ShBhLhRw-Gb ShBhLnRw-Gb
ShBhLvRv-Gb ShBnLhRv-Mr SnBnLhRv-Tl ShBhLvRw-Gb SnBnLnRv-Mr SnBnLvRv-Mr ShBnLvRv-
Mr ShBnLnRv-Mr SnBhLnRv-Mr ShBnLhRw-Gb ShBnLnRw-Gb SnBnLhRw-Tl SnBhLvRw-Ml
SnBnLvRw-Dn SnBhLvRv-Tb SnBnLnRw-Dn SnBhLnRw-Dn ShBhLhRv-Mr ]
116
Experiment 2: Graph Analysis
Figure 4.8 shows the fitness score of every fifth solution ranked low to high within the
gene pool in every generation leading up to the final 59th generation. This graph reflects
the changing nature of the gene pool throughout the evolutionary process.
Figure 4.8: Evolving Fitness Scores of Experiment 2
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.9 below reflects the fitness measure of the best solution in each gene
pool from generation 1 to generation 59. Note that although the system is able to transfer
all parts to the dump cell in earlier generations, this solution may include illegal and/or
superfluous steps. It is not until generation 59 that the evolutionary process stabilizes to
an optimal solution achieved in 55 legal steps.
1 8
15
22
29
36
43
50
57
64
71
78
85
92
99
S1
S17
S33
0
1000
2000
3000
4000
5000
6000
7000
8000
117
Figure 4.9: Generational Fitness for Experiment 2
(Y-axis: Fitness Measure, X-axis: Generation)
Figure 4.10 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
maximum values and the ratios are plotted below in Figure 4.10. Note that not all
solutions leading to a goal are achieved through legal steps (yellow optimal but pink
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45
118
suboptimal). Note also that in early generations, a solution can be comprised entirely of
legal steps and yet not achieve goal behavior (pink optimal but yellow suboptimal).
Figure 4.10: Scores of Three Components of Fitness for Experiment 2
(Y-axis: Coefficient of Optimality, X-axis: Generation)
4.5 Experiment 3: Midsize / Nine-Action / Universal-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has the Tr, Tl, Gr and Gl actions disabled, thus the agents can
only collaboratively transport parts to the dump cell by using the Ml, Mr, Tl, Tb and Dn
actions. This limitation makes it much harder for the collaborative agents to show
progress and more steps will be required in the simulation to move all 9 parts to the dump
cell.
0
0.25
0.5
0.75
1
1 21 41
(max-pgs)/max (ls/max) (cfg/max)
119
The agents in this experiment are not all governed by their own individual
controller, nor is this a ‘bookend’ approach. In this ‘universal’ approach, all agents are
governed by the same controller. Thus, there is only one controller to be evolved in this
system and it will draw from the experiences of all agents within the system during the
evolutionary process.
Figure 4.11: Initial Configuration of System for Experiment 3
Experiment 3: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = True: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = True: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = True: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = True: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = “Universal"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
120
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 3: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 3 PGS= 800 LS= 743 CFG= 2763 SIM= 3506 --
2 S= 800 D= 2 PGS= 800 LS= 749 CFG= 2727 SIM= 3476 --
3 S= 800 D= 5 PGS= 800 LS= 742 CFG= 3735 SIM= 4477 --
4 S= 800 D= 8 PGS= 800 LS= 760 CFG= 5859 SIM= 6619 --
5 S= 800 D= 5 PGS= 800 LS= 795 CFG= 4431 SIM= 5226 --
6 S= 800 D= 9 PGS= 123 LS= 798 CFG= 6561 SIM= 8036 --
7 S= 800 D= 9 PGS= 123 LS= 798 CFG= 6561 SIM= 8036 --
8 S= 800 D= 9 PGS= 123 LS= 798 CFG= 6561 SIM= 8036 --
9 S= 800 D= 9 PGS= 107 LS= 799 CFG= 6561 SIM= 8053 --
10 S= 800 D= 9 PGS= 107 LS= 799 CFG= 6561 SIM= 8053 --
11 S= 800 D= 9 PGS= 107 LS= 799 CFG= 6561 SIM= 8053 --
12 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
121
13 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
14 S= 800 D= 9 PGS= 107 LS= 800 CFG= 6561 SIM= 8054 LG
15 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
16 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
17 S= 800 D= 9 PGS= 95 LS= 800 CFG= 6561 SIM= 8066 LG
18 S= 800 D= 9 PGS= 83 LS= 799 CFG= 6561 SIM= 8077 --
19 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
20 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
21 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
22 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
23 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
24 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
25 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
26 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
27 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
28 S= 800 D= 9 PGS= 83 LS= 800 CFG= 6561 SIM= 8078 LG
29 S= 800 D= 9 PGS= 79 LS= 799 CFG= 6561 SIM= 8081 --
30 S= 800 D= 9 PGS= 79 LS= 800 CFG= 6561 SIM= 8082 LG
31 S= 800 D= 9 PGS= 75 LS= 799 CFG= 6561 SIM= 8085 --
32 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
33 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
34 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
35 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
36 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
37 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
38 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
39 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
40 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
41 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
42 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
43 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
44 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
45 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
46 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
47 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
48 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
49 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
50 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
51 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
52 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
53 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
54 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
55 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
56 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
57 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
58 S= 800 D= 9 PGS= 63 LS= 799 CFG= 6561 SIM= 8097 --
59 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
60 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
61 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
62 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
63 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
64 S= 800 D= 9 PGS= 63 LS= 799 CFG= 6561 SIM= 8097 --
65 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
66 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
67 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
68 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
122
69 S= 800 D= 9 PGS= 59 LS= 799 CFG= 6561 SIM= 8101 --
70 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
71 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
72 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
73 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 8102 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 59 legal steps within the simulation using the final evolved rule-sets
shown below. 59 steps is one step shy of 15 cycles within the simulation. The four agents
each take a consecutive step to complete one cycle. Note how the fitness measure allows
for multi-criteria evolution to occur. At times the evolutionary process will lean towards
transferring more parts to the dump cell even if it uses more illegal steps to accomplish
this. Other times the evolutionary process will seek to make all steps legal at the expense
of goal progress. Over time, the effect is to get better at each objective and finally to
settle down into the optimal, legal solution.
Experiment 3: Final Results
G: 73 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 59 POSTGOALSTEPS= 741
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 8102 LG
An output listing below shows the trace for each of the 59 steps. It shows the step
action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
123
TRACE: ORIG 1Gr 2Dn 3Tl 4Mr 1Ml 2Ml 3Mr 4Gb 1Tb 2Tl 3Gr 4Gb 1Dn 2Mr 3Tb 4Tl
1Tb 2Mr 3Dn 4Gb 1Mr 2Gr 3Gr 4Gb 1Mr 2Tb 3Tl 4Tl 1Gr 2Gr 3Gl 4Gb 1Ml 2Gr
3Gr 4Gb 1Dn 2Ml 3Ml 4Ml 1Ml 2Tb 3Tl 4Tl 1Tb 2Tb 3Dn 4Mr 1Mr 2Gr 3Mr 4Gb
1Gr 2Mr 3Gr 4Gb 1Dn 2Gr 3Gr 4Gb 1Dn 2Ml 3Ml 4Ml 1Ml 2Ml 3Ml 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
BENCH: 302110 302110 302110 302110 302110 302110 302110 302110 302111 202111 202111 202111 202112
202112 202112 202102 202102 102102 102102 102102 102103 102103 102103 102103 102104 102104 102004
102004 102004 102004 102004 102004 102005 102005 102005 102005 102006 102006 102006 102006 102006
102006 101006 101006 101006 001006 000006 000006 000006 000006 000006 000006 000007 000007 000007
000007 000008 000008 000008 000008 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
124
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
AGENT: 012340 012340 012340 012340 012304 102304 120304 120034 120034 120034 120034 120034 120034
120034 102034 102034 102034 102034 100234 100234 100234 010234 010234 010234 010234 001234 001234
001234 001234 001234 001234 001234 001234 010234 010234 010234 010234 010234 012034 012304 012340
102340 102340 102340 102340 102340 102340 102340 102304 012304 012304 012034 012034 012034 010234
010234 010234 010234 010234 010234 010234 010234 012034 012304 012340 102340 120340 123040 123040
125
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
CARRY: 010010 001010 001010 000110 000101 000101 000101 000011 000010 100010 010010 010001 010000
010000 001000 001010 001001 101001 100101 100101 100100 010100 010010 010001 010000 001000 001100
001010 001001 000101 000011 000101 000100 000100 000010 000001 000000 000000 000000 000000 000000
000000 001000 000100 000010 100010 101010 101010 101001 011001 010101 010011 010010 001010 000110
000101 000100 000100 000010 000001 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
126
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
127
Experiment 3: Evolved Rule-Sets
The single controller was evolved in 73 generations. This controller took 59 steps in the
simulation to reach the collaborative goal. The number of rules evolved for all controllers
is 57 out of a possible 64. Thus, some states were never and could never be visited.
# 57 [ShBnLvRn-Gr SnBhLvRn-Tb SnBhLwRn-Tb ShBhLnRv-Gl ShBnLvRw-Gb SnBhLvRh-Ml SnBhLnRv-
Tb SnBhLvRv-Dn ShBhLvRv-Mr ShBhLvRn-Gr SnBhLwRv-Tb ShBhLwRv-Mr ShBnLhRv-Mr SnBhLhRh-Tl
ShBhLnRh-Dn SnBnLvRv-Dn ShBnLnRv-Mr SnBhLnRn-Dn ShBhLhRv-Mr ShBhLnRn-Dn ShBnLnRh-Gl
ShBnLvRv-Mr ShBhLvRh-Dn SnBhLvRw-Ml ShBnLnRn-Gr SnBnLhRn-Tl SnBhLhRv-Tl SnBhLhRn-Tb
SnBnLvRh-Ml SnBnLvRn-Ml SnBnLhRv-Tl ShBhLwRn-Dn SnBnLnRv-Dn ShBnLhRw-Gb SnBhLhRw-Tl
ShBnLhRh-Dn SnBhLwRh-Dn ShBnLvRh-Ml ShBhLvRw-Gb ShBhLhRw-Gb SnBnLwRv-Mr SnBnLnRn-Dn
ShBhLnRw-Gb SnBnLhRh-Dn ShBnLhRn-Gr ShBhLhRn-Gr ShBhLhRh-Dn ShBnLwRn-Gr SnBnLhRw-Tl
SnBhLnRw-Dn ShBnLwRv-Mr ShBnLnRw-Gb SnBnLwRh-Dn SnBnLwRn-Dn SnBnLnRh-Dn SnBhLnRh-Tb
ShBhLwRh-Dn ]
Experiment 3: Graph Analysis
Figure 4.12 below shows the fitness score of every fifth solution ranked low to high
within the gene pool in every generation leading up to the final 73rd generation. This
graph reflects the changing nature of the gene pool throughout the evolutionary process.
128
Figure 4.12: Evolving Fitness Scores of Experiment 3
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.13 below reflects the fitness measure of the best solution in each gene
pool from generation 1 to generation 27.
1 9
17
25
33
41
49
57
65
73
81
89
97
S1
S21
S41
S610
1000
2000
3000
4000
5000
6000
7000
8000
9000
129
Figure 4.13: Generational Fitness for Experiment 3
(Y-axis: Fitness Measure, X-axis: Generation)
Figure 4.14 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
maximum values and the ratios are plotted below in Figure 4.14. Note that not all
solutions leading to a goal are achieved through legal steps (yellow optimal but pink
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73
130
suboptimal). Note also that in early generations, a solution can be comprised entirely of
legal steps and yet not achieve goal behavior (pink optimal but yellow suboptimal).
Figure 4.14: Scores of Three Components of Fitness for Experiment 3.
(Y-axis: Coefficient of Optimality, X-axis: Generation)
0
0.25
0.5
0.75
1
1 21 41 61
(max-pgs)/max (ls/max) (cfg/max)
131
4.6 Experiment 4: Midsize / Nine-Action / Divided-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has the Tr, Tl, Gr and Gl actions disabled, thus the agents can
only collaboratively transport parts to the dump cell by using the Ml, Mr, Tl, Tb and Dn
actions. This limitation makes it much harder for the collaborative agents to show
progress and more steps will be required in the simulation to move all 9 parts to the dump
cell. The agents in this experiment are not all governed by their own individual controller,
nor is this a ‘bookend’ approach. In this ‘Divided’ approach, the first two agents are
governed by the one controller while the last two agents in the system are governed by a
second controller.
Figure 4.15: Initial Configuration of System for Experiment 4
Experiment 4: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = True: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = True: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = True: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = True: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
132
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = “Divided"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 4: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
133
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 7 PGS= 800 LS= 693 CFG= 5859 SIM= 6552 --
2 S= 800 D= 5 PGS= 800 LS= 770 CFG= 4383 SIM= 5153 --
3 S= 800 D= 5 PGS= 800 LS= 745 CFG= 4383 SIM= 5128 --
4 S= 800 D= 5 PGS= 800 LS= 757 CFG= 3735 SIM= 4492 --
5 S= 800 D= 6 PGS= 800 LS= 733 CFG= 4383 SIM= 5116 --
6 S= 800 D= 7 PGS= 800 LS= 792 CFG= 5115 SIM= 5907 --
7 S= 800 D= 7 PGS= 800 LS= 764 CFG= 5157 SIM= 5921 --
8 S= 800 D= 7 PGS= 800 LS= 797 CFG= 5157 SIM= 5954 --
9 S= 800 D= 7 PGS= 800 LS= 789 CFG= 5913 SIM= 6702 --
10 S= 800 D= 7 PGS= 800 LS= 763 CFG= 5835 SIM= 6598 --
11 S= 800 D= 8 PGS= 800 LS= 762 CFG= 5841 SIM= 6603 --
12 S= 800 D= 6 PGS= 800 LS= 796 CFG= 5373 SIM= 6169 --
13 S= 800 D= 9 PGS= 223 LS= 750 CFG= 6561 SIM= 7888 --
14 S= 800 D= 9 PGS= 143 LS= 794 CFG= 6561 SIM= 8012 --
15 S= 800 D= 9 PGS= 131 LS= 799 CFG= 6561 SIM= 8029 --
16 S= 800 D= 9 PGS= 131 LS= 799 CFG= 6561 SIM= 8029 --
17 S= 800 D= 9 PGS= 107 LS= 794 CFG= 6561 SIM= 8048 --
18 S= 800 D= 9 PGS= 115 LS= 790 CFG= 6561 SIM= 8036 --
19 S= 800 D= 9 PGS= 115 LS= 800 CFG= 6561 SIM= 8046 LG
20 S= 800 D= 9 PGS= 103 LS= 797 CFG= 6561 SIM= 8055 --
21 S= 800 D= 9 PGS= 103 LS= 800 CFG= 6561 SIM= 8058 LG
22 S= 800 D= 9 PGS= 103 LS= 800 CFG= 6561 SIM= 8058 LG
23 S= 800 D= 9 PGS= 103 LS= 800 CFG= 6561 SIM= 8058 LG
24 S= 800 D= 9 PGS= 103 LS= 800 CFG= 6561 SIM= 8058 LG
25 S= 800 D= 9 PGS= 103 LS= 800 CFG= 6561 SIM= 8058 LG
26 S= 800 D= 9 PGS= 103 LS= 800 CFG= 6561 SIM= 8058 LG
27 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
28 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
29 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
30 S= 800 D= 9 PGS= 75 LS= 800 CFG= 6561 SIM= 8086 LG
31 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
32 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
33 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
34 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
35 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
36 S= 800 D= 9 PGS= 71 LS= 800 CFG= 6561 SIM= 8090 LG
37 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
38 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
39 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
40 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
41 S= 800 D= 9 PGS= 67 LS= 800 CFG= 6561 SIM= 8094 LG
42 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
43 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
44 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
45 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
46 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
47 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
48 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
49 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
50 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
51 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
52 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
53 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
54 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
134
55 S= 800 D= 9 PGS= 59 LS= 800 CFG= 6561 SIM= 8102 LG
56 S= 800 D= 9 PGS= 55 LS= 800 CFG= 6561 SIM= 8106 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 8106 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 55 legal steps within the simulation using the final evolved rule-sets
shown below. 55 steps is one step shy of 14 cycles within the simulation. The four agents
each take a consecutive step to complete one cycle. Note how the fitness measure allows
for multi-criteria evolution to occur. At times the evolutionary process will lean towards
transferring more parts to the dump cell even if it uses more illegal steps to accomplish
this. Other times the evolutionary process will seek to make all steps legal at the expense
of goal progress. Over time, the effect is to get better at each objective and finally to
settle down into the optimal, legal solution.
Experiment 4: Final Results G: 56 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 55 POSTGOALSTEPS= 745
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 8106 LG
An output listing below shows the trace for each of the 55 steps. It shows the step
action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
TRACE: ORIG 1Gr 2Gr 3Gl 4Mr 1Ml 2Gr 3Mr 4Gb 1Tb 2Ml 3Gr 4Gb 1Gr 2Mr 3Ml
4Ml 1Tb 2Gr 3Gr 4Mr 1Mr 2Tb 3Tl 4Gb 1Gr 2Dn 3Mr 4Tl 1Ml 2Mr 3Tl 4Gb 1Tb
2Tb 3Gr 4Gb 1Mr 2Gr 3Gr 4Gb 1Mr 2Tl 3Tb 4Tl 1Tb 2Gr 3Dn 4Gb 1Gr 2Gl 3Gr
4Gb 1Gr 2Gr 3Gr 4Gb 1Ml 2Ml 3Ml 4Ml 1Ml 2Ml 3Ml 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn
135
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
BENCH: 302110 302110 302110 302110 302110 302110 302110 302110 302111 202111 202111 202111 202112
202112 202112 202112 202112 102112 102112 102112 102112 102112 101112 101112 101113 101113 101113
101113 101113 101113 101113 101113 101114 001114 001014 001014 001015 001015 001015 001015 001016
001016 001016 001006 001006 000006 000006 000006 000007 000007 000007 000007 000008 000008 000008
000008 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
136
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
AGENT: 012340 012340 012340 012340 012304 102304 102304 102034 102034 102034 120034 120034 120034
120034 102034 102304 102340 102340 102340 102340 102304 012304 012304 012304 012304 012304 012304
012034 012034 102034 100234 100234 100234 100234 100234 100234 100234 010234 010234 010234 010234
001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234 001234
001234 001234 010234 012034 012304 012340 102340 120340 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040
137
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040 123040
123040 123040
CARRY: 010010 001010 000110 001010 001001 001001 000101 000011 000010 100010 100010 100001 100000
010000 001000 001000 001000 101000 100100 100010 100001 010001 011001 010101 010100 001100 001100
001010 001001 001001 000101 000011 000010 100010 100110 100101 100100 010100 010010 010001 010000
001000 000100 000110 000101 001101 001011 001011 001010 000110 001010 001001 001000 000100 000010
000001 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
138
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
139
Experiment 4: Evolved Rule-Sets
Single controller was evolved in 56 generations. This controller took 55 steps in the
simulation to reach the collaborative goal. The number of rules evolved for each
controller is 47 and 47 respectively for the two controllers out of a possible 64. Thus,
some states were never and could never be visited. Note that the rule-sets are not the
same.
# 47 [ShBnLvRn-Gr SnBhLhRn-Tb ShBhLwRv-Mr SnBnLvRh-Ml SnBhLvRh-Tb SnBhLvRn-Tb
ShBhLnRn-Gr ShBnLnRv-Mr SnBhLwRn-Tb ShBnLvRh-Gb SnBhLwRv-Tb SnBnLvRn-Ml SnBnLhRv-
Mr ShBnLhRv-Tl SnBhLvRv-Ml SnBhLnRv-Tb SnBnLnRv-Mr SnBnLhRn-Tl ShBnLhRn-Gr SnBnLvRv-
Mr SnBhLwRh-Tb SnBnLwRv-Mr SnBnLwRn-Dn ShBnLnRn-Gr ShBhLvRn-Gr ShBnLvRv-Mr
ShBnLwRv-Mr ShBnLwRn-Gr SnBnLnRn-Dn ShBhLwRn-Gr SnBhLnRh-Tb SnBnLnRh-Dn SnBnLhRh-
Tr ShBhLnRv-Mr SnBhLhRh-Tl ShBnLnRh-Gl ShBhLwRh-Tr ShBhLhRn-Gr SnBhLnRn-Tb ShBhLnRh-
Dn ShBnLwRh-Dn ShBhLvRv-Mr SnBhLhRv-Tb ShBhLhRv-Gb SnBnLwRh-Dn ShBhLvRh-Dn
ShBnLhRh-Gl ]
# 47 [SnBhLvRn-Ml SnBhLnRv-Tb ShBnLvRw-Dn SnBhLnRh-Dn SnBhLhRn-Tb SnBhLhRv-Tl
SnBhLvRw-Ml SnBhLvRh-Tb ShBhLnRv-Mr SnBhLhRh-Tl SnBhLhRw-Tl ShBhLvRn-Gr ShBhLnRw-
Gb ShBhLvRv-Mr ShBhLnRh-Gl ShBnLhRw-Gb ShBhLvRh-Dn ShBnLhRv-Mr ShBhLvRw-Gb
ShBnLnRv-Mr SnBnLvRh-Ml SnBnLnRh-Dn SnBnLhRv-Tl SnBnLnRv-Dn ShBhLnRn-Gr ShBhLhRw-
Gb ShBnLnRw-Gb SnBnLhRh-Tl SnBnLvRn-Ml ShBnLnRh-Dn ShBnLnRn-Gr SnBhLnRn-Tb
SnBhLvRv-Ml ShBnLvRn-Ml ShBnLvRv-Ml SnBnLhRn-Tl ShBnLhRh-Dn ShBhLhRn-Gr SnBnLvRv-
Dn SnBhLnRw-Dn ShBhLhRh-Gb ShBhLhRv-Mr SnBnLnRn-Dn ShBnLhRn-Gr ShBnLvRh-Gb
SnBnLhRw-Dn SnBnLvRw-Ml ]
Experiment 4: Graph Analysis
Figure 4.16 below shows the fitness score of every fifth solution ranked low to
high within the gene pool in every generation leading up to the final 56th generation. This
graph reflects the changing nature of the gene pool throughout the evolutionary process.
140
Figure 4.16: Evolving Fitness Scores of Experiment 4
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.17 below reflects the fitness measure of the best solution in each gene
pool from generation 1 to generation 56.
1 8
15
22
29
36
43
50
57
64
71
78
85
92
99
S1
S19
S37
S550
1000
2000
3000
4000
5000
6000
7000
8000
9000
141
Figure 4.17: Generational Fitness for Experiment 4
(Y-axis: Fitness Measure, X-axis: Generation)
Figure 4.18 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
maximum values and the ratios are plotted below. Note that not all solutions leading to a
goal are achieved through legal steps (yellow optimal but pink suboptimal). Note also
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55
142
that in early generations, a solution can be comprised entirely of legal steps and yet not
achieve goal behavior (pink optimal but yellow suboptimal).
Figure 4.18: Scores of Three Components of Fitness for Experiment 4
(Y-axis: Coefficient of Optimality, X-axis: Generation)
0
0.25
0.5
0.75
1
1 21 41
(max-pgs)/max (ls/max) (cfg/max)
143
4.7 Experiment 5: Midsize / Five-Action / Individually-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has the Tr, Tl, Gr and Gl actions disabled, thus the agents can
only collaboratively transport parts to the dump cell by using the Ml, Mr, Tl, Tb and Dn
actions. This limitation makes it much harder for the collaborative agents to show
progress and more steps will be required in the simulation to move all 9 parts to the
dump cell. Additionally, this experiment has each agent relying on its own set of
controller rules. Thus, each agent’s rule-set is evolved pursuant to the particular agent’s
experiences within the simulation training. This is the ‘individual’ controller plan
designation.
Figure 4.19: Initial Configuration of System for Experiment 5
Experiment 5: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = False: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = False: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = False: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = False: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
144
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = "Individual"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 5: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
145
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 2 PGS= 800 LS= 731 CFG= 1845 SIM= 2576 --
2 S= 800 D= 2 PGS= 800 LS= 793 CFG= 1767 SIM= 2560 --
3 S= 800 D= 1 PGS= 800 LS= 759 CFG= 1827 SIM= 2586 --
4 S= 800 D= 3 PGS= 800 LS= 701 CFG= 2691 SIM= 3392 --
5 S= 800 D= 3 PGS= 800 LS= 782 CFG= 2691 SIM= 3473 --
6 S= 800 D= 3 PGS= 800 LS= 793 CFG= 2691 SIM= 3484 --
7 S= 800 D= 3 PGS= 800 LS= 798 CFG= 2691 SIM= 3489 --
8 S= 800 D= 3 PGS= 800 LS= 799 CFG= 2691 SIM= 3490 --
9 S= 800 D= 3 PGS= 800 LS= 799 CFG= 2691 SIM= 3490 --
10 S= 800 D= 3 PGS= 800 LS= 799 CFG= 2295 SIM= 3094 --
11 S= 800 D= 3 PGS= 800 LS= 799 CFG= 2295 SIM= 3094 --
12 S= 800 D= 4 PGS= 800 LS= 746 CFG= 3249 SIM= 3995 --
13 S= 800 D= 3 PGS= 800 LS= 786 CFG= 3249 SIM= 4035 --
14 S= 800 D= 3 PGS= 800 LS= 796 CFG= 3249 SIM= 4045 --
15 S= 800 D= 4 PGS= 800 LS= 760 CFG= 3663 SIM= 4423 --
16 S= 800 D= 3 PGS= 800 LS= 798 CFG= 3249 SIM= 4047 --
17 S= 800 D= 3 PGS= 800 LS= 798 CFG= 3249 SIM= 4047 --
18 S= 800 D= 3 PGS= 800 LS= 798 CFG= 3249 SIM= 4047 --
19 S= 800 D= 3 PGS= 800 LS= 798 CFG= 3249 SIM= 4047 --
20 S= 800 D= 3 PGS= 800 LS= 799 CFG= 3249 SIM= 4048 --
21 S= 800 D= 5 PGS= 800 LS= 766 CFG= 3735 SIM= 4501 --
22 S= 800 D= 6 PGS= 800 LS= 781 CFG= 4383 SIM= 5164 --
23 S= 800 D= 6 PGS= 800 LS= 768 CFG= 4383 SIM= 5151 --
24 S= 800 D= 5 PGS= 800 LS= 792 CFG= 3663 SIM= 4455 --
25 S= 800 D= 5 PGS= 800 LS= 793 CFG= 3663 SIM= 4456 --
26 S= 800 D= 5 PGS= 800 LS= 799 CFG= 3663 SIM= 4462 --
27 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3663 SIM= 4463 L-
28 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3663 SIM= 4463 L-
29 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3663 SIM= 4463 L-
30 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3663 SIM= 4463 L-
31 S= 800 D= 5 PGS= 800 LS= 800 CFG= 3663 SIM= 4463 L-
32 S= 800 D= 6 PGS= 800 LS= 786 CFG= 4383 SIM= 5169 --
33 S= 800 D= 6 PGS= 800 LS= 794 CFG= 4383 SIM= 5177 --
34 S= 800 D= 6 PGS= 800 LS= 795 CFG= 4383 SIM= 5178 --
35 S= 800 D= 6 PGS= 800 LS= 796 CFG= 4383 SIM= 5179 --
36 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4383 SIM= 5183 L-
37 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4383 SIM= 5183 L-
38 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4383 SIM= 5183 L-
39 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4383 SIM= 5183 L-
40 S= 800 D= 6 PGS= 800 LS= 796 CFG= 4389 SIM= 5185 --
41 S= 800 D= 6 PGS= 800 LS= 799 CFG= 4389 SIM= 5188 --
42 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4389 SIM= 5189 L-
43 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4389 SIM= 5189 L-
44 S= 800 D= 6 PGS= 800 LS= 762 CFG= 4779 SIM= 5541 --
45 S= 800 D= 8 PGS= 800 LS= 795 CFG= 5835 SIM= 6630 --
46 S= 800 D= 8 PGS= 800 LS= 797 CFG= 5835 SIM= 6632 --
47 S= 800 D= 8 PGS= 800 LS= 799 CFG= 5835 SIM= 6634 --
48 S= 800 D= 8 PGS= 800 LS= 799 CFG= 5835 SIM= 6634 --
49 S= 800 D= 8 PGS= 800 LS= 799 CFG= 5835 SIM= 6634 --
50 S= 800 D= 8 PGS= 800 LS= 799 CFG= 5835 SIM= 6634 --
51 S= 800 D= 8 PGS= 800 LS= 799 CFG= 5835 SIM= 6634 --
52 S= 800 D= 8 PGS= 800 LS= 799 CFG= 5835 SIM= 6634 --
53 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5835 SIM= 6635 L-
54 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
146
55 S= 800 D= 8 PGS= 800 LS= 779 CFG= 5913 SIM= 6692 --
56 S= 800 D= 8 PGS= 800 LS= 794 CFG= 5913 SIM= 6707 --
57 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
58 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
59 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
60 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
61 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
62 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
63 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
64 S= 800 D= 9 PGS= 315 LS= 776 CFG= 6561 SIM= 7822 --
65 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
66 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
67 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
68 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
69 S= 800 D= 9 PGS= 395 LS= 764 CFG= 6561 SIM= 7730 --
70 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
71 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
72 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
73 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
74 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
75 S= 800 D= 8 PGS= 800 LS= 800 CFG= 5841 SIM= 6641 L-
76 S= 800 D= 9 PGS= 399 LS= 760 CFG= 6561 SIM= 7722 --
77 S= 800 D= 9 PGS= 371 LS= 799 CFG= 6561 SIM= 7789 --
78 S= 800 D= 9 PGS= 371 LS= 799 CFG= 6561 SIM= 7789 --
79 S= 800 D= 9 PGS= 371 LS= 799 CFG= 6561 SIM= 7789 --
80 S= 800 D= 9 PGS= 371 LS= 800 CFG= 6561 SIM= 7790 LG
81 S= 800 D= 9 PGS= 371 LS= 800 CFG= 6561 SIM= 7790 LG
82 S= 800 D= 9 PGS= 371 LS= 800 CFG= 6561 SIM= 7790 LG
83 S= 800 D= 9 PGS= 371 LS= 800 CFG= 6561 SIM= 7790 LG
84 S= 800 D= 9 PGS= 371 LS= 800 CFG= 6561 SIM= 7790 LG
85 S= 800 D= 9 PGS= 299 LS= 767 CFG= 6561 SIM= 7829 --
86 S= 800 D= 9 PGS= 371 LS= 800 CFG= 6561 SIM= 7790 LG
87 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
88 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
89 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
90 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
91 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
92 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
93 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
94 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
95 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
96 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
97 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
98 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
99 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
100 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
101 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
102 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
103 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
104 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
105 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
106 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
107 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
108 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
109 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
110 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
147
111 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
112 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
113 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
114 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
115 S= 800 D= 9 PGS= 239 LS= 785 CFG= 6561 SIM= 7907 --
116 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
117 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
118 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
119 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
120 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
121 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
122 S= 800 D= 9 PGS= 363 LS= 800 CFG= 6561 SIM= 7798 LG
123 S= 800 D= 9 PGS= 215 LS= 789 CFG= 6561 SIM= 7935 --
124 S= 800 D= 9 PGS= 211 LS= 799 CFG= 6561 SIM= 7949 --
125 S= 800 D= 9 PGS= 211 LS= 799 CFG= 6561 SIM= 7949 --
126 S= 800 D= 9 PGS= 211 LS= 799 CFG= 6561 SIM= 7949 --
127 S= 800 D= 9 PGS= 211 LS= 799 CFG= 6561 SIM= 7949 --
128 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
129 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
130 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
131 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
132 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
133 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
134 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
135 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
136 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
137 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
138 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
139 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
140 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
141 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
142 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
143 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
144 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
145 S= 800 D= 9 PGS= 207 LS= 800 CFG= 6561 SIM= 7954 LG
146 S= 800 D= 9 PGS= 199 LS= 798 CFG= 6561 SIM= 7960 --
147 S= 800 D= 9 PGS= 203 LS= 800 CFG= 6561 SIM= 7958 LG
148 S= 800 D= 9 PGS= 203 LS= 800 CFG= 6561 SIM= 7958 LG
149 S= 800 D= 9 PGS= 199 LS= 799 CFG= 6561 SIM= 7961 --
150 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
151 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
152 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
153 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
154 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
155 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
156 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
157 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
158 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
159 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
160 S= 800 D= 9 PGS= 191 LS= 796 CFG= 6561 SIM= 7966 --
161 S= 800 D= 9 PGS= 191 LS= 796 CFG= 6561 SIM= 7966 --
162 S= 800 D= 9 PGS= 191 LS= 796 CFG= 6561 SIM= 7966 --
163 S= 800 D= 9 PGS= 191 LS= 796 CFG= 6561 SIM= 7966 --
164 S= 800 D= 9 PGS= 191 LS= 799 CFG= 6561 SIM= 7969 --
165 S= 800 D= 9 PGS= 191 LS= 799 CFG= 6561 SIM= 7969 --
166 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
148
167 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
168 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
169 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
170 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
171 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
172 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
173 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
174 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
175 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
176 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
177 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
178 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
179 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
180 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
181 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
182 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
183 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
184 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
185 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
186 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
187 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
188 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
189 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
190 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
191 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
192 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
193 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
194 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
195 S= 800 D= 9 PGS= 187 LS= 800 CFG= 6561 SIM= 7974 LG
196 S= 800 D= 9 PGS= 183 LS= 800 CFG= 6561 SIM= 7978 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 7978 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 183 legal steps within the simulation using the final evolved rule-sets
shown below. 183 steps is one step shy of 46 cycles within the simulation. This solution
was found in the 196th generation. The four agents each take a consecutive step to
complete one cycle. Note how the fitness measure allows for multi-criteria evolution to
occur. At times the evolutionary process will lean towards transferring more parts to the
dump cell even if it uses more illegal steps to accomplish this. Other times the
evolutionary process will seek to make all steps legal at the expense of goal progress.
149
Over time, the effect is to get better at each objective and finally to settle down into the
optimal, legal solution.
Experiment 5: Final Results G: 295 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 183 POSTGOALSTEPS= 617
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 7978 LG
An output listing below shows the trace for each of the 183 steps. It shows the
step action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
TRACE: ORIG 1Dn 2Dn 3Tb 4Mr 1Dn 2Tb 3Mr 4Gb 1Gb 2Mr 3Gb 4Dn 1Ml 2Gb 3Dn
4Dn 1Tb 2Ml 3Ml 4Ml 1Mr 2Dn 3Dn 4Tb 1Gb 2Tb 3Tb 4Mr 1Dn 2Gb 3Mr 4Gb 1Ml
2Tb 3Gb 4Dn 1Tb 2Gb 3Ml 4Ml 1Mr 2Dn 3Dn 4Tb 1Gb 2Tb 3Dn 4Mr 1Dn 2Gb 3Dn
4Gb 1Ml 2Ml 3Ml 4Ml 1Tb 2Dn 3Tb 4Tb 1Dn 2Tb 3Mr 4Mr 1Dn 2Mr 3Mr 4Gb 1Mr
2Mr 3Dn 4Dn 1Dn 2Gb 3Gb 4Dn 1Dn 2Ml 3Ml 4Ml 1Gb 2Dn 3Dn 4Tb 1Ml 2Ml 3Tb
4Mr 1Dn 2Tb 3Mr 4Gb 1Dn 2Mr 3Gb 4Dn 1Dn 2Gb 3Ml 4Ml 1Dn 2Ml 3Ml 4Tb 1Dn
2Tb 3Tb 4Mr 1Dn 2Gb 3Mr 4Gb 1Dn 2Tb 3Mr 4Dn 1Dn 2Mr 3Gb 4Dn 1Dn 2Gb 3Ml
4Ml 1Dn 2Ml 3Ml 4Tb 1Dn 2Tb 3Tb 4Mr 1Dn 2Gb 3Mr 4Gb 1Dn 2Tb 3Mr 4Dn 1Dn
2Mr 3Gb 4Dn
1Dn 2Gb 3Ml 4Ml 1Dn 2Ml 3Ml 4Tb 1Dn 2Dn 3Tb 4Mr 1Dn 2Dn 3Mr 4Gb 1Dn 2Dn
3Mr 4Dn 1Dn 2Dn 3Gb 4Dn 1Dn 2Dn 3Ml 4Ml 1Dn 2Dn 3Ml 4Tb 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn 3Dn 4Gb 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn
3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn
3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml
1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn
3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn
3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml
1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn
3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn
3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml
1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn
3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn
3Dn 4Ml
1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn
3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn
3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml
1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn
3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn
3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml
150
1Dn 2Dn 3Dn
4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn
2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn
4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn
2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn
4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn
2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn
4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn
2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn
4Mr 1Dn
2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn
4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn
2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn
4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Mr
BENCH: 302110 302110 302110 302010 302010 302010 301010 301010 301011 311011 311011 311021 311021
311021 311121 311121 311121 211121 211121 211121 211121 211121 211121 211121 211111 221111 220111
220011 220011 220011 221011 221011 221012 221012 220012 220022 220022 120022 121022 121022 121022
121022 121022 121022 121012 131012 130012 130012 130012 130012 131012 131012 131013 131013 131013
131013 131013 031013 031013 030013 030003 030003 020003 020003 020003 020003 020003 020003 020004
020004 020004 020004 020004 020004 020104 020114 020114 020114 020114 020114 020114 030114 030114
030114 030104 030104 030104 030004 030004 030004 020004 020004 020005 020005 020005 020015 020015
020015 021015 021015 021015 021015 021015 021015 021005 021005 011005 010005 010005 010005 020005
020005 020006 020006 010006 010006 010006 010006 010006 010016 010016 010016 011016 011016 011016
011016 011016 011016 011006 011006 001006 000006 000006 000006 010006 010006 010007 010007 000007
000007 000007 000007 000007 000017 000017
000017 001017 001017 001017 001017 001017 001017 001007 001007 001007 000007 000007 000007 000007
000007 000008 000008 000008 000008 000008 000008 000008 000018 000018 000018 000018 000018 000018
000018 000018 000018 000008 000008 000008 000008 000008 000008 000008 000008 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
151
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
AGENT: 012340 012340 012340 012340 012304 012304 012304 012034 012034 012034 010234 010234 010234
100234 100234 100234 100234 100234 102034 102304 102340 012340 012340 012340 012340 012340 012340
012340 012304 012304 012304 012034 012034 102034 102034 102034 102034 102034 102034 102304 102340
012340 012340 012340 012340 012340 012340 012340 012304 012304 012304 012304 012304 102304 120304
123004 123040 123040 123040 123040 123040 123040 123040 120340 120304 120304 102304 102034 102034
012034 010234 010234 010234 010234 010234 010234 010234 010234 012034 012304 012340 012340 012340
012340 012340 102340 120340 120340 120304 120304 120304 120034 120034 120034 102034 102034 102034
102034 102034 102304 102340 102340 120340 123040 123040 123040 123040 123040 123004 123004 123004
120304 120304 120304 120304 120034 120034 120034 102034 102034 102034 102034 102034 102304 102340
102340 120340 123040 123040 123040 123040 123040 123004 123004 123004 120304 120304 120304 120304
120034 120034 120034 102034 102034 102034
102034 102034 102304 102340 102340 120340 123040 123040 123040 123040 123040 123004 123004 123004
120304 120304 120304 120304 120034 120034 120034 120034 120034 120034 120034 120034 120304 120340
120340 120340 123040 123040 123040 123040 123040 123004 123004 123004 123004 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
152
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040
123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004 123004 123004
123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040 123040 123004
123004 123004 123004 123040 123040 123040 123040 123004 123004 123004 123004 123040 123040 123040
123040 123004
CARRY: 010010 010010 010010 010110 010101 010101 011101 011011 011010 001010 000110 000100 000100
000100 000000 000000 000000 100000 100000 100000 100000 010000 010000 010000 010010 000010 001010
001110 001101 001101 000101 000011 000010 000010 001010 001000 001000 101000 100000 100000 100000
010000 010000 010000 010010 000010 001010 001010 001001 001001 000001 000001 000000 000000 000000
000000 000000 100000 100000 101000 101010 101010 111010 110110 110101 110101 101101 101011 101010
011010 010110 010110 010110 010110 010010 010000 010000 010000 010000 010000 010000 000000 000000
000000 000010 000010 000010 000110 000101 000101 010101 010011 010010 010010 001010 001000 001000
001000 000000 000000 000000 000000 000000 000000 000010 000010 010010 011010 011001 011001 001001
000101 000100 000100 010100 010010 010010 010010 001010 001000 001000 001000 000000 000000 000000
000000 000000 000000 000010 000010 010010 011010 011001 011001 001001 000101 000100 000100 010100
010010 010010 010010 001010 001000 001000
001000 000000 000000 000000 000000 000000 000000 000010 000010 000010 001010 001001 001001 001001
000101 000100 000100 000100 000010 000010 000010 000010 000000 000000 000000 000000 000000 000000
000000 000000 000000 000010 000010 000010 000010 000001 000001 000001 000001 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
153
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
Experiment 5: Evolved Rule-Sets
Below are shown the evolved rule-sets for the system. Since the individual controller plan
was used, a distinct rule-set for each agent was evolved based on the experiences of that
agent in the simulations. Each rule-set below is preceded by a number reflecting the
number of rules that were evolved within the rule-set. Each rule is of the format
‘C1C2C3C4-Action’ where a series of condition settings have an associated subscribed
action. Note that the total number of possible rules is equal to the total number of
possible condition states which in this problem is 64. The evolution of a reduced number
of rules is a significant improvement over trying to pre-determine all 64 rules for each
agent outside of an evolutionary learning process. Note that Agent A’s rule-set never has
154
a rule with “wall to the right” condition since this is never encountered by the agent in the
course of the simulation. This is due to agents being unable to pass each other in the
corridor. Each agent is therefore limited in its scope of experience and this in turn
influences which rules evolve for it. It will be shown that in subsequent experiments in
which an alternative controller plan is used, the experiences of multiple agents sourced by
a common controller are shared and are used towards the evolutionary shaping of that
rule-set. Note also that the disabling of side-to-side taking and giving actions has forced
the agents to rely on the more awkward and laborious bench passing and cell shifting
in order to accomplish part transfer in the system. The rule-sets below are a reduced set
of rules necessary to elicit optimal collaborative performance within the simulation.
Operating the agents individually by these rules does not result in optimal system
performance. It is the necessary inter-dependence that is captured by these evolved rule-
sets. The four controllers were evolved in 196 generations. These controllers
operating together took 183 steps in the simulation to reach the collaborative goal.
The # rules evolved across the four controllers 118.
# 22 [ShBnLvRn-Dn SnBhLvRn-Ml ShBnLvRh-Gb SnBhLvRv-Ml ShBhLvRv-Dn ShBhLvRn-Gb
SnBhLwRv-Tb ShBhLwRv-Mr ShBnLwRv-Mr SnBnLwRv-Dn SnBnLwRn-Dn SnBhLwRn-Tb
SnBhLvRh-Dn SnBnLvRv-Dn SnBnLwRh-Dn ShBnLwRh-Dn ShBnLwRn-Dn SnBhLwRh-Gl
SnBnLvRn-Ml ShBhLwRh-Dn ShBnLvRv-Mr ShBhLvRh-Gb ]
# 36 [ShBhLhRn-Dn ShBnLvRh-Gb SnBhLvRn-Ml ShBhLvRn-Gb SnBnLvRn-Ml SnBhLhRh-Tb
ShBnLvRn-Gb SnBhLvRv-Tb SnBhLhRn-Dn SnBnLhRv-Dn ShBhLhRv-Mr SnBnLnRn-Dn ShBnLnRn-
Gb ShBhLvRv-Mr ShBnLhRv-Mr SnBnLnRv-Dn SnBhLnRv-Tb ShBnLvRv-Gb SnBhLnRn-Tb
ShBnLhRn-Gb SnBnLvRh-Dn ShBnLhRh-Tl SnBnLvRv-Tb ShBhLnRv-Mr ShBhLvRh-Dn ShBhLnRn-
Dn SnBnLhRh-Dn SnBhLnRh-Tb SnBhLvRh-Dn ShBhLnRh-Gb SnBhLhRv-Tb SnBnLnRh-Dn
ShBnLnRv-Mr SnBnLhRn-Dn ShBnLnRh-Gb ShBhLhRh-Dn ]
# 36 [ShBnLnRv-Mr ShBnLhRv-Mr SnBnLnRv-Dn SnBhLnRv-Tb ShBhLnRn-Gb SnBnLnRn-Dn
SnBhLvRn-Ml ShBnLvRn-Gb ShBhLvRn-Gb SnBnLvRv-Ml SnBhLvRv-Tb ShBnLvRv-Mr ShBhLvRh-
Tb SnBhLhRv-Tb ShBnLnRh-Dn SnBhLhRn-Tb SnBhLnRn-Dn SnBhLnRh-Tb ShBhLhRn-Gb
ShBhLvRv-Mr SnBhLvRh-Tb SnBnLhRv-Mr ShBhLhRv-Mr SnBnLhRn-Dn ShBhLnRv-Mr SnBnLhRh-
Dn SnBnLvRn-Ml SnBnLnRh-Dn SnBhLhRh-Tb ShBnLhRn-Dn ShBhLnRh-Tb ShBnLvRh-Tl
ShBnLnRn-Gb SnBnLvRh-Ml ShBhLhRh-Tl ShBnLhRh-Dn ]
155
# 24 [ShBhLnRv-Mr SnBhLvRw-Ml ShBhLvRw-Gb SnBhLhRv-Tb SnBhLvRv-Tb SnBnLvRv-Mr
SnBnLnRv-Dn ShBhLhRv-Mr SnBhLhRw-Dn ShBhLvRv-Mr ShBnLnRw-Gb SnBnLhRw-Dn
ShBhLhRw-Gb SnBnLvRw-Tr ShBnLnRv-Mr SnBnLnRw-Dn ShBnLhRv-Mr SnBhLnRw-Dn
SnBhLnRv-Tb ShBhLnRw-Tb ShBnLvRv-Mr ShBnLvRw-Gb SnBnLhRv-Mr ShBnLhRw-Gb ]
Experiment 5: Graph Analysis
Figure 4.20 below shows the fitness score of every fifth solution ranked low to high
within the gene pool in every generation leading up to the final 196th generation. This
graph reflects the changing nature of the gene pool throughout the evolutionary process.
Figure 4.20: Evolving Fitness Scores of Experiment 5
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.21 below reflects the fitness measure of the best solution in each gene
pool from generation 1 to generation 196. Note that although the system is able to
1
14
27
40
53
66
79
92
10
5
11
8
13
1
14
4
15
7
17
0
18
3
19
6
S1
S35
S69
0
1000
2000
3000
4000
5000
6000
7000
8000
156
transfer all parts to the dump cell as early as generation 123, this solution may include
illegal and/or superfluous steps. It is not until generation 196 that the evolutionary
process stabilizes to an optimal solution achieved in 183 legal steps.
Figure 4.21: Generational Fitness for Experiment 5
(Y-axis: Fitness Measure, X-axis: Generation)
Figure 4.22 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177 188
157
maximum values and the ratios are plotted below in Figure 4.22. Note that not all
solutions leading to a goal are achieved through legal steps (yellow optimal but pink
suboptimal). Note also that in early generations, a solution can be comprised entirely of
legal steps and yet not achieve goal behavior (pink optimal but yellow suboptimal).
Figure 4.22: Scores of Three Components of Fitness for Experiment 5
(Y-axis: Coefficient of Optimality, X-axis: Generation)
4.8 Experiment 6: Midsize / Five-Action / Bookend-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has the Tr, Tl, Gr and Gl actions disabled, thus the agents can
only collaboratively transport parts to the dump cell by using the Ml, Mr, Tl, Tb and Dn
actions. This limitation makes it much harder for the collaborative agents to show
0
0.25
0.5
0.75
1
1 21 41 61 81 101 121 141 161 181
(max-pgs)/max (ls/max) (cfg/max)
158
progress and more steps will be required in the simulation to move all 9 parts to the dump
cell.
The agents in this experiment are not all governed by their own individual
controller. In this ‘bookend’ approach, the end agents A and D each get their own
controller. The middle agents B and C will be sourcing a common controller. This is a
reasonable approach given that B and C are exposed to similar conditional states whereas
A and D experience conditional states unique to each other as well as to the central
agents. Thus, there are three controllers to be evolved in this system. The second
controller draws from the experiences of both agent B and C. The ability of the method to
specify the sourcing controller for each agent in the system is useful for systems in which
collaboration between two or more distinct agent groups is desired. It is expected that use
of the ‘bookend’ controller plan also decreases the number of steps required to find the
optimal collaborative goal. The number of rules in the three controllers will also be
altered from the prior four rule-sets evolved under the individual controller plan.
Figure 4.23: Initial Configuration of System for Experiment 6
Experiment 6: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
159
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = False: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = False: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = False: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = False: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = "Bookend"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 6: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
160
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 1 PGS= 800 LS= 732 CFG= 1773 SIM= 2505 --
2 S= 800 D= 1 PGS= 800 LS= 793 CFG= 1773 SIM= 2566 --
3 S= 800 D= 3 PGS= 800 LS= 747 CFG= 2703 SIM= 3450 --
4 S= 800 D= 3 PGS= 800 LS= 776 CFG= 2259 SIM= 3035 --
5 S= 800 D= 3 PGS= 800 LS= 771 CFG= 2601 SIM= 3372 --
6 S= 800 D= 3 PGS= 800 LS= 798 CFG= 2601 SIM= 3399 --
7 S= 800 D= 3 PGS= 800 LS= 798 CFG= 2601 SIM= 3399 --
8 S= 800 D= 3 PGS= 800 LS= 798 CFG= 2601 SIM= 3399 --
9 S= 800 D= 4 PGS= 800 LS= 799 CFG= 3171 SIM= 3970 --
10 S= 800 D= 4 PGS= 800 LS= 772 CFG= 3249 SIM= 4021 --
11 S= 800 D= 4 PGS= 800 LS= 800 CFG= 3171 SIM= 3971 L-
12 S= 800 D= 4 PGS= 800 LS= 800 CFG= 3171 SIM= 3971 L-
13 S= 800 D= 4 PGS= 800 LS= 800 CFG= 3171 SIM= 3971 L-
14 S= 800 D= 5 PGS= 800 LS= 796 CFG= 3663 SIM= 4459 --
15 S= 800 D= 5 PGS= 800 LS= 774 CFG= 3921 SIM= 4695 --
16 S= 800 D= 5 PGS= 800 LS= 797 CFG= 3663 SIM= 4460 --
17 S= 800 D= 5 PGS= 800 LS= 785 CFG= 3897 SIM= 4682 --
18 S= 800 D= 7 PGS= 800 LS= 799 CFG= 5109 SIM= 5908 --
19 S= 800 D= 7 PGS= 800 LS= 800 CFG= 5109 SIM= 5909 L-
20 S= 800 D= 7 PGS= 800 LS= 793 CFG= 5349 SIM= 6142 --
21 S= 800 D= 7 PGS= 800 LS= 800 CFG= 5109 SIM= 5909 L-
22 S= 800 D= 8 PGS= 800 LS= 772 CFG= 6075 SIM= 6847 --
23 S= 800 D= 7 PGS= 800 LS= 800 CFG= 5115 SIM= 5915 L-
24 S= 800 D= 7 PGS= 800 LS= 800 CFG= 5115 SIM= 5915 L-
25 S= 800 D= 7 PGS= 800 LS= 784 CFG= 5133 SIM= 5917 --
26 S= 800 D= 7 PGS= 800 LS= 799 CFG= 5133 SIM= 5932 --
27 S= 800 D= 7 PGS= 800 LS= 800 CFG= 5133 SIM= 5933 L-
28 S= 800 D= 8 PGS= 800 LS= 795 CFG= 5835 SIM= 6630 --
29 S= 800 D= 8 PGS= 800 LS= 795 CFG= 5835 SIM= 6630 --
30 S= 800 D= 8 PGS= 800 LS= 793 CFG= 6075 SIM= 6868 --
31 S= 800 D= 9 PGS= 215 LS= 795 CFG= 6561 SIM= 7941 --
32 S= 800 D= 9 PGS= 215 LS= 799 CFG= 6561 SIM= 7945 --
33 S= 800 D= 9 PGS= 215 LS= 799 CFG= 6561 SIM= 7945 --
34 S= 800 D= 9 PGS= 215 LS= 799 CFG= 6561 SIM= 7945 --
35 S= 800 D= 9 PGS= 215 LS= 799 CFG= 6561 SIM= 7945 --
36 S= 800 D= 9 PGS= 199 LS= 799 CFG= 6561 SIM= 7961 --
37 S= 800 D= 9 PGS= 199 LS= 799 CFG= 6561 SIM= 7961 --
38 S= 800 D= 9 PGS= 199 LS= 799 CFG= 6561 SIM= 7961 --
39 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
40 S= 800 D= 9 PGS= 191 LS= 797 CFG= 6561 SIM= 7967 --
41 S= 800 D= 9 PGS= 191 LS= 799 CFG= 6561 SIM= 7969 --
42 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
43 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
44 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
45 S= 800 D= 9 PGS= 179 LS= 800 CFG= 6561 SIM= 7982 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
161
The method found an optimal solution score of 7982 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 179 legal steps within the simulation using the final evolved rule-sets
shown below. 179 steps is one step shy of 45 cycles within the simulation. The four
agents each take a consecutive step to complete one cycle. Note how the fitness measure
allows for multi-criteria evolution to occur. At times the evolutionary process will lean
towards transferring more parts to the dump cell even if it uses more illegal steps to
accomplish this. Other times the evolutionary process will seek to make all steps legal at
the expense of goal progress. Over time, the effect is to get better at each objective and
finally to settle down into the optimal, legal solution.
Experiment 6: Final Results
G: 45 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 179 POSTGOALSTEPS= 621
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 7982 LG
An output listing below shows the trace for each of the 179 steps. It shows the
step action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
TRACE: ORIG 1Gb 2Dn 3Tb 4Mr 1Ml 2Tb 3Mr 4Gb 1Dn 2Mr 3Gb 4Dn 1Dn 2Gb 3Dn
4Dn 1Dn 2Ml 3Ml 4Ml 1Dn 2Ml 3Ml 4Ml 1Dn 2Dn 3Dn 4Tb 1Dn 2Dn 3Tb 4Mr 1Dn
2Tb 3Mr 4Mr 1Tb 2Mr 3Mr 4Gb 1Mr 2Mr 3Gb 4Dn 1Dn 2Gb 3Dn 4Dn 1Dn 2Ml 3Ml
4Ml 1Gb 2Dn 3Dn 4Tb 1Ml 2Dn 3Tb 4Mr 1Dn 2Ml 3Mr 4Gb 1Dn 2Tb 3Gb 4Dn 1Tb
2Mr 3Ml 4Ml 1Mr 2Gb 3Dn 4Tb 1Gb 2Dn 3Dn 4Mr 1Ml 2Ml 3Ml 4Gb 1Dn 2Dn 3Tb
4Ml 1Dn 2Tb 3Mr 4Tb 1Tb 2Mr 3Dn 4Mr 1Mr 2Dn 3Mr 4Gb 1Dn 2Mr 3Gb 4Dn 1Dn
2Gb 3Dn 4Dn 1Dn 2Ml 3Ml 4Ml 1Gb 2Dn 3Dn 4Tb 1Ml 2Dn 3Tb 4Mr 1Dn 2Ml 3Mr
4Gb 1Dn 2Tb 3Gb 4Dn 1Dn 2Mr 3Ml 4Ml 1Dn 2Gb 3Dn 4Tb 1Dn 2Ml 3Ml 4Mr 1Dn
2Dn 3Tb 4Gb
1Dn 2Dn 3Mr 4Ml 1Dn 2Mr 3Gb 4Tb 1Dn 2Dn 3Tb 4Mr 1Dn 2Ml 3Mr 4Gb 1Dn 2Mr
3Gb 4Dn 1Dn 2Ml 3Ml 4Ml 1Dn 2Mr 3Dn 4Tb 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Mr 4Gb
1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml
162
3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr
3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn
1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml
3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
1Dn 2Mr
3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn
1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml
3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr
3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn
1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml
3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr
3Dn 4Dn
1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml
3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr
3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn
1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml
3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr
3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn
1Dn 2Ml 3Dn
4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn
2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn
4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn
2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn
4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn
2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn
4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn
2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn
4Dn 1Dn
2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn
4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn
2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn
4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn 1Dn 2Mr 3Dn 4Dn 1Dn 2Ml 3Dn 4Dn
BENCH: 302110 312110 312110 312010 312010 312010 311010 311010 311011 311011 311011 311021 311021
311021 311121 311121 311121 311121 311121 311121 311121 311121 311121 311121 311121 311121 311121
311121 311021 311021 311021 310021 310021 310021 300021 300021 300021 200021 200021 200021 200022
200022 200022 200032 200032 200032 200132 200132 200132 200132 200132 200132 200132 210132 210132
210132 210122 210122 210122 210022 210022 210022 210022 210022 210023 210023 200023 200033 200033
100033 100033 100033 100033 100033 101033 101033 101023 111023 111023 111023 111023 111023 111023
111023 111024 111024 111024 110024 110024 110024 100024 100024 100014 000014 000014 000014 000014
000014 000014 000014 000015 000015 000015 000025 000025 000025 000125 000125 000125 000125 000125
000125 000125 010125 010125 010125 010115 010115 010115 010015 010015 010015 010015 010015 010016
010016 000016 000026 000026 000026 000026 000026 000026 000026 001026 001026 001016 001016 001016
001016 001016 001016 001016 000016 000017
000017 000017 000017 000017 000017 000017 000117 000107 000107 000107 000007 000007 000007 000007
000007 000008 000008 000008 000018 000018 000018 000018 000018 000018 000018 000018 000018 000008
000008 000008 000008 000008 000008 000008 000008 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
163
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
AGENT: 012340 012340 012340 012340 012304 102304 102304 102034 102034 102034 100234 100234 100234
100234 100234 100234 100234 100234 102034 102304 102340 102340 120340 123040 123400 123400 123400
123400 123400 123400 123400 123400 123040 123040 123040 120340 120304 120304 102304 102034 102034
012034 010234 010234 010234 010234 010234 010234 010234 010234 012034 012304 012340 012340 012340
012340 012340 102340 102340 102340 102304 102304 120304 120034 120034 120034 120034 120034 120034
120034 102034 102304 102340 012340 012340 012340 012340 012340 012340 012340 012304 102304 120304
123004 123004 123004 123004 123004 123040 123040 123040 120340 120340 120340 102340 102340 102304
012304 012304 012034 012034 012034 010234 010234 010234 010234 010234 010234 010234 010234 012034
012304 012340 012340 012340 012340 012340 102340 102340 102340 102304 102304 120304 120034 120034
120034 120034 120034 120034 120034 102034 102304 102340 102340 102340 102340 102340 102340 120340
123040 123004 123004 123004 123004 123004
123004 123004 120304 120340 120340 102340 102340 102340 102340 102340 102340 102304 102304 120304
120034 120034 120034 102034 102034 102034 102034 120034 120304 120340 120340 102340 102340 102340
102340 102340 102340 102304 102304 102304 102034 102034 102034 120034 120034 120034 120034 102034
164
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034
102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034 120034 102034
102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034 120034 120034
120034 102034 102034 102034 102034 120034 120034 120034 120034 102034 102034 102034 102034 120034
120034 120034
CARRY: 010010 000010 000010 000110 000101 000101 001101 001011 001010 001010 000110 000100 000100
000100 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000100 000100 000100 001100 001010 001010 011010 010110 010101 110101 101101 101011 101010
011010 010110 010100 010100 010100 010000 010000 010000 010000 010000 010000 010000 000000 000000
000000 000010 000010 000010 000110 000101 000101 000101 000011 000010 000010 010010 010000 010000
110000 101000 101000 101000 011000 010000 010000 010010 000010 000010 000010 000001 000001 000001
000001 000000 000000 000000 001000 001000 001000 011000 010100 010110 110110 101110 101110 101101
011101 011101 011011 011010 011010 010110 010100 010100 010100 010000 010000 010000 010000 010000
010000 010000 000000 000000 000000 000010 000010 000010 000110 000101 000101 000101 000011 000010
000010 010010 010000 010000 010000 001000 001000 001000 001000 000000 000000 000010 000010 000010
165
000010 000001 000001 000001 001001 001000
001000 001000 000100 000100 000100 000100 000000 000010 000010 000010 000110 000101 000101 000101
000011 000010 000010 000010 000000 000000 000000 000000 000000 000000 000000 000000 000000 000010
000010 000010 000010 000001 000001 000001 000001 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
166
Experiment 6: Evolved Rule-Sets
Below are shown the evolved rule-sets for the system. The three controllers were
evolved in only 45 generations. These controllers operating together took 179 steps
in the simulation to reach the collaborative goal. The number of rules evolved for all
controllers is 81. Note that the agents central B and C are governed by the single middle
controller. The rules in the second controller are sufficient for the control of actions of
both B and C agents within the simulation.
# 24 [SnBhLvRn-Ml SnBhLvRh-Dn ShBnLvRh-Dn SnBhLwRv-Dn SnBhLvRv-Tb ShBnLvRn-Gb
ShBhLwRv-Mr ShBnLvRv-Dn ShBhLvRn-Ml ShBhLvRh-Tl SnBnLvRn-Dn ShBhLwRn-Dn SnBhLwRn-
Dn SnBnLvRv-Ml ShBhLwRh-Gb SnBnLvRh-Ml ShBnLwRv-Mr SnBnLwRv-Dn SnBnLwRn-Dn
ShBnLwRn-Dn SnBnLwRh-Dn ShBnLwRh-Dn ShBhLvRv-Dn SnBhLwRh-Tb ]
# 36 [SnBhLvRn-Ml SnBhLnRh-Tb SnBhLnRn-Dn ShBhLvRh-Gb ShBhLhRv-Dn SnBhLhRn-Tb
SnBhLhRv-Dn SnBhLnRv-Tb ShBnLhRv-Mr ShBnLvRh-Dn ShBhLvRv-Mr ShBhLvRn-Gb ShBhLnRv-
Dn ShBhLhRn-Gb ShBnLnRv-Mr ShBhLhRh-Gr SnBhLvRv-Dn ShBnLnRn-Gb ShBnLhRn-Gb
SnBnLnRv-Mr ShBnLvRn-Gb ShBhLnRh-Dn SnBnLvRn-Dn SnBnLnRn-Dn ShBhLnRn-Gb ShBnLvRv-
Mr ShBnLnRh-Gb SnBnLvRh-Ml SnBnLnRh-Dn SnBnLhRn-Dn SnBnLhRv-Dn SnBnLvRv-Ml
SnBnLhRh-Dn ShBnLhRh-Dn SnBhLvRh-Tb SnBhLhRh-Tb ]
# 21 [ShBhLhRv-Mr ShBnLnRw-Gb SnBhLhRw-Dn SnBhLvRw-Ml ShBnLhRw-Gb SnBhLnRw-Dn
SnBhLnRv-Tb ShBhLvRw-Gb ShBnLhRv-Mr ShBhLnRw-Gb ShBhLnRv-Mr SnBhLvRv-Ml SnBnLnRv-
Dn ShBnLvRw-Gb ShBnLvRv-Gb ShBnLnRv-Mr SnBnLvRv-Dn ShBhLvRv-Mr ShBhLhRw-Gb
SnBhLhRv-Tb SnBnLhRv-Mr ]
Experiment 6: Graph Analysis
Figure 4.24 below shows the fitness score of every fifth solution ranked low to high
within the gene pool in every generation leading up to the final 45th generation. This
graph reflects the changing nature of the gene pool throughout the evolutionary process.
167
Figure 4.24: Evolving Fitness Scores of Experiment 6
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.25: Generational Fitness for Experiment 6
(Y-axis: Fitness Measure, X-axis: Generation)
1 8
15
22
29
36
43
50
57
64
71
78
85
92
99
S1
S17
S33
0
1000
2000
3000
4000
5000
6000
7000
8000
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45
168
Figure 4.26 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
maximum values and the ratios are plotted below. Note that not all solutions leading to a
goal are achieved through legal steps (yellow optimal but pink suboptimal). Note also
that in early generations, a solution can be comprised entirely of legal steps and yet not
achieve goal behavior (pink optimal but yellow suboptimal).
169
Figure 4.26: Scores of Three Components of Fitness for Experiment 6
(Y-axis: Coefficient of Optimality, X-axis: Generation)
4.9 Experiment 7: Midsize / Five-Action / Universal-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has the Tr, Tl, Gr and Gl actions disabled, thus the agents can
only collaboratively transport parts to the dump cell by using the Ml, Mr, Tl, Tb and Dn
actions. This limitation makes it much harder for the collaborative agents to show
progress and more steps will be required in the simulation to move all 9 parts to the dump
cell.
The agents in this experiment are not all governed by their own individual
controller, nor is this a ‘bookend’ approach. In this ‘universal’ approach, all agents are
governed by the same controller. Thus, there is only one controller to be evolved in this
0
0.25
0.5
0.75
1
1 21 41
(max-pgs)/max (ls/max) (cfg/max)
170
system and it will draw from the experiences of all agents within the system during the
evolutionary process.
Figure 4.27: Initial Configuration of System for Experiment 7
Experiment 7: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = False: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = False: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = False: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = False: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = "Universal"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
171
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 7: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 1 PGS= 800 LS= 711 CFG= 2199 SIM= 2910 --
2 S= 800 D= 0 PGS= 800 LS= 759 CFG= 2223 SIM= 2982 --
3 S= 800 D= 0 PGS= 800 LS= 799 CFG= 2223 SIM= 3022 --
4 S= 800 D= 0 PGS= 800 LS= 800 CFG= 2223 SIM= 3023 L-
5 S= 800 D= 1 PGS= 800 LS= 736 CFG= 2673 SIM= 3409 --
6 S= 800 D= 3 PGS= 800 LS= 788 CFG= 3513 SIM= 4301 --
7 S= 800 D= 3 PGS= 800 LS= 798 CFG= 3513 SIM= 4311 --
8 S= 800 D= 6 PGS= 800 LS= 746 CFG= 4383 SIM= 5129 --
9 S= 800 D= 5 PGS= 800 LS= 763 CFG= 4383 SIM= 5146 --
10 S= 800 D= 5 PGS= 800 LS= 792 CFG= 4701 SIM= 5493 --
11 S= 800 D= 9 PGS= 259 LS= 723 CFG= 6561 SIM= 7825 --
12 S= 800 D= 8 PGS= 800 LS= 800 CFG= 6075 SIM= 6875 L-
13 S= 800 D= 8 PGS= 800 LS= 782 CFG= 6561 SIM= 7343 --
14 S= 800 D= 9 PGS= 287 LS= 775 CFG= 6561 SIM= 7849 --
15 S= 800 D= 8 PGS= 800 LS= 798 CFG= 6561 SIM= 7359 --
16 S= 800 D= 9 PGS= 231 LS= 782 CFG= 6561 SIM= 7912 --
17 S= 800 D= 9 PGS= 239 LS= 798 CFG= 6561 SIM= 7920 --
18 S= 800 D= 9 PGS= 235 LS= 796 CFG= 6561 SIM= 7922 --
19 S= 800 D= 9 PGS= 211 LS= 800 CFG= 6561 SIM= 7950 LG
20 S= 800 D= 9 PGS= 211 LS= 800 CFG= 6561 SIM= 7950 LG
172
21 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
22 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
23 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
24 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
25 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
26 S= 800 D= 9 PGS= 199 LS= 800 CFG= 6561 SIM= 7962 LG
27 S= 800 D= 9 PGS= 195 LS= 800 CFG= 6561 SIM= 7966 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 7966 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 195 legal steps within the simulation using the final evolved rule-sets
shown below. 195 steps is one step shy of 49 cycles within the simulation. The four
agents each take a consecutive step to complete one cycle. Note how the fitness measure
allows for multi-criteria evolution to occur. At times the evolutionary process will lean
towards transferring more parts to the dump cell even if it uses more illegal steps to
accomplish this. Other times the evolutionary process will seek to make all steps legal at
the expense of goal progress. Over time, the effect is to get better at each objective and
finally to settle down into the optimal, legal solution.
Experiment 7: Final Results
G: 27 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 195 POSTGOALSTEPS= 605
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 7966 LG
An output listing below shows the trace for each of the 195 steps. It shows the
step action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
173
TRACE: ORIG 1Gb 2Dn 3Tb 4Mr 1Ml 2Ml 3Mr 4Gb 1Dn 2Tb 3Ml 4Ml 1Tb 2Mr 3Gb
4Tb 1Mr 2Gb 3Tb 4Mr 1Gb 2Tb 3Mr 4Gb 1Ml 2Gb 3Gb 4Dn 1Tb 2Tb 3Ml 4Ml 1Mr
2Gb 3Dn 4Tb 1Ml 2Ml 3Ml 4Mr 1Gb 2Dn 3Tb 4Gb 1Dn 2Tb 3Mr 4Ml 1Tb 2Mr 3Gb
4Dn 1Mr 2Gb 3Dn 4Dn 1Gb 2Dn 3Dn 4Dn 1Ml 2Ml 3Ml 4Ml 1Dn 2Dn 3Dn 4Tb 1Dn
2Dn 3Tb 4Mr 1Dn 2Tb 3Mr 4Mr 1Tb 2Mr 3Mr 4Gb 1Mr 2Mr 3Gb 4Dn 1Mr 2Gb 3Dn
4Dn 1Ml 2Ml 3Ml 4Ml 1Gb 2Dn 3Dn 4Tb 1Ml 2Ml 3Ml 4Mr 1Dn 2Dn 3Tb 4Gb 1Dn
2Tb 3Mr 4Ml 1Dn 2Mr 3Gb 4Dn 1Dn 2Ml 3Ml 4Ml 1Dn 2Dn 3Dn 4Tb 1Dn 2Dn 3Tb
4Mr 1Dn 2Dn 3Mr 4Mr 1Dn 2Mr 3Mr 4Gb 1Dn 2Mr 3Gb 4Dn 1Dn 2Ml 3Ml 4Ml 1Dn
2Gb 3Dn 4Tb
1Dn 2Ml 3Ml 4Mr 1Dn 2Dn 3Tb 4Gb 1Dn 2Dn 3Mr 4Ml 1Dn 2Dn 3Ml 4Ml 1Dn 2Dn
3Dn 4Tb 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Mr 4Mr 1Dn 2Dn 3Mr 4Gb 1Dn 2Dn 3Gb 4Dn
1Dn 2Dn 3Ml 4Ml 1Dn 2Dn 3Ml 4Tb 1Dn 2Dn 3Dn 4Mr 1Dn 2Dn 3Dn 4Gb 1Dn 2Dn
3Dn 4Ml 1Dn 2Dn 3Dn 4Ml 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
BENCH: 302110 312110 312110 312010 312010 312010 312010 312010 312011 312011 302011 302011 302011
202011 202011 202111 202101 202101 203101 203001 203001 213001 212001 212001 212002 212002 213002
213012 213012 113012 112012 112012 112012 112012 113012 113012 113002 113002 113002 113002 113002
213002 213002 212002 212003 212003 202003 202003 202003 102003 102003 102103 102103 102103 103103
103103 103103 113103 113103 113103 113103 113103 113103 113103 113103 113103 113103 113103 113003
113003 113003 112003 112003 112003 102003 102003 102003 002003 002003 002003 002004 002004 002004
174
002014 002014 002014 002114 002114 002114 002114 002114 002114 002114 012114 012114 012114 012104
012104 012104 012104 012104 012104 012104 011104 011105 011105 001105 001105 001105 001105 001105
001205 001205 001205 001205 001205 001205 001205 001205 001205 001105 001105 001105 000105 000105
000105 000105 000105 000105 000105 000105 000105 000106 000106 000106 000116 000116 000116 000116
000116 000116 000116 001116 001116 001106
001106 001106 001106 001106 001106 001106 000106 000107 000107 000107 000107 000107 000107 000107
000107 000107 000107 000107 000107 000007 000007 000007 000007 000007 000007 000007 000007 000007
000007 000007 000007 000008 000008 000008 000018 000018 000018 000018 000018 000018 000018 000018
000018 000008 000008 000008 000008 000008 000008 000008 000008 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
175
AGENT: 012340 012340 012340 012340 012304 102304 120304 120034 120034 120034 120034 120304 120340
120340 102340 102340 102340 012340 012340 012340 012304 012304 012304 012034 012034 102034 102034
102034 102034 102034 102034 102304 102340 012340 012340 012340 012340 102340 120340 123040 123004
123004 123004 123004 123004 123004 123004 120304 120340 120340 102340 102340 102340 012340 012340
012340 012340 012340 012340 012340 012340 102340 120340 123040 123400 123400 123400 123400 123400
123400 123400 123400 123040 123040 123040 120340 120304 120304 102304 102034 102034 012034 010234
010234 010234 001234 001234 001234 001234 010234 012034 012304 012340 012340 012340 012340 012340
102340 120340 123040 123004 123004 123004 123004 123004 123004 123004 120304 120340 120340 102340
102340 102340 102340 120340 123040 123400 123400 123400 123400 123400 123400 123400 123400 123040
123040 123040 120340 120304 120304 102304 102034 102034 102034 100234 100234 100234 100234 102034
102304 102340 102340 102340 102340 102340
102340 120340 123040 123004 123004 123004 123004 123004 123004 123004 120304 120340 120340 120340
123040 123400 123400 123400 123400 123400 123400 123400 123400 123040 123040 123040 120340 120304
120304 120304 120034 120034 120034 120034 120034 120034 120034 120034 120304 120340 120340 120340
123040 123040 123040 123040 123040 123004 123004 123004 123004 123004 123004 123004 123004 123040
123040 123040 123040 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
176
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400 123400
123400 123400
CARRY: 010010 000010 000010 000110 000101 000101 000101 000011 000010 000010 010010 010100 010100
110100 101100 101000 101010 011010 010010 010110 010101 000101 001101 001011 001010 001010 000010
000000 000000 100000 101000 101000 101000 011000 010000 010000 010010 100010 100010 100010 100001
000001 000001 001001 001000 001000 011000 010100 010100 110100 101100 101000 101000 011000 010000
010000 010000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000100
000100 000100 001100 001010 001010 011010 010110 010101 110101 101101 101011 101010 011010 010110
010100 010100 001100 001000 001000 001000 010000 010000 010000 010000 000000 000000 000000 000010
000010 000010 000010 000001 000001 000001 001001 001000 001000 011000 010100 010100 010100 001100
001000 001000 001000 010000 010000 010000 010000 010000 010000 010100 010100 010100 011100 011010
011010 011010 010110 010101 010101 001101 001011 001010 001010 000110 000100 000100 000100 001000
001000 001000 001000 000000 000000 000010
000010 000010 000010 000001 000001 000001 001001 001000 001000 001000 000100 000100 000100 000100
001000 001000 001000 001000 001000 001100 001100 001100 001100 001010 001010 001010 000110 000101
000101 000101 000011 000010 000010 000010 000000 000000 000000 000000 000000 000000 000000 000000
000000 000010 000010 000010 000010 000001 000001 000001 000001 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
177
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
Experiment 7: Evolved Rule-Sets
Below are shown the evolved rule-sets for the system. Since the universal controller plan
was used, a distinct rule-set for each agent was evolved based on the experiences of that
agent in the simulations. Each rule-set below is preceded by a number reflecting the
number of rules that were evolved within the rule-set. Each rule is of the format
‘C1C2C3C4-Action’ where a series of condition settings have an associated subscribed
action. Note that the total number of possible rules is equal to the total number of
possible condition states which in this problem is 64. The evolution of a reduced number
of rules is a significant improvement over trying to pre-determine all 64 rules for each
agent outside of an evolutionary learning process. Keep in mind also that for every
condition state there are nine possible actions. Note also that the disabling of side-to-side
taking and giving actions has forced the agents to rely on the more awkward and
laborious bench passing and cell shifting in order to accomplish part transfer in the
system. The rule-set below is a reduced set of rules necessary to elicit optimal
collaborative performance within the simulation. Operating the agents by this common
rule-set achieves the necessary inter-dependence that leads to collaborative performance.
The single controller was evolved in only 27 generations. This controller took 195
steps in the simulation to reach the collaborative goal. The number of rules evolved
for all controllers is 59 out of a possible 64. Thus some states were never and could
never be visited.
178
# 59 [ShBnLvRn-Gb ShBhLnRv-Mr SnBhLvRh-Ml SnBhLvRn-Ml SnBhLhRv-Tb ShBnLvRw-Gb
ShBhLvRh-Dn ShBhLhRh-Gb SnBhLvRv-Tb ShBhLhRv-Mr ShBnLvRv-Mr ShBnLhRw-Gb ShBhLvRn-
Ml ShBhLnRn-Gb ShBnLvRh-Gb SnBhLhRn-Dn SnBhLhRh-Tb SnBhLwRv-Tb ShBhLwRv-Mr
SnBhLvRw-Ml ShBnLnRv-Mr ShBnLnRw-Gb SnBhLwRh-Tb SnBnLvRv-Ml ShBnLhRv-Mr ShBhLnRh-
Gb SnBhLnRw-Dn SnBhLnRn-Dn SnBhLhRw-Dn SnBnLnRv-Dn SnBnLvRn-Ml ShBhLvRw-Gb
SnBnLhRv-Mr SnBnLvRh-Ml ShBnLhRh-Gb SnBhLwRn-Dn ShBhLnRw-Gb SnBhLnRh-Tb ShBnLnRh-
Dn ShBhLwRh-Gb ShBhLhRw-Gb SnBnLnRn-Dn SnBnLnRh-Dn SnBhLnRv-Tb ShBnLwRn-Dn
ShBnLwRv-Mr ShBhLwRn-Gb SnBnLhRn-Dn ShBhLvRv-Gb SnBnLwRv-Dn SnBnLwRn-Dn
SnBnLwRh-Dn SnBnLhRw-Tr ShBnLwRh-Gb SnBnLhRh-Dn ShBnLhRn-Gb ShBhLhRn-Gb ShBnLnRn-
Dn SnBnLvRw-Tr ]
Experiment 7: Graph Analysis
Figure 4.28 below shows the fitness score of every fifth solution ranked low to high
within the genepool in every generation leading up to the final 27th generation. This graph
reflects the changing nature of the genepool throughout the evolutionary process.
Figure 4.28: Evolving Fitness Scores of Experiment 7
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
1 7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
S1
S16
0
1000
2000
3000
4000
5000
6000
7000
8000
179
Figure 4.29 below reflects the fitness measure of the best solution in each gene
pool from generation 1 to generation 27.
Figure 4.29: Generational Fitness for Experiment 7
(Y-axis: Fitness Measure, X-axis: Generation)
Figure 4.30 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 3 5 7 9 11 13 15 17 19 21 23 25 27
180
maximum values and the ratios are plotted below. Note that not all solutions leading to a
goal are achieved through legal steps (yellow optimal but pink suboptimal). Note also
that in early generations, a solution can be comprised entirely of legal steps and yet not
achieve goal behavior (pink optimal but yellow suboptimal).
Figure 4.30: Scores of Three Components of Fitness for Experiment 7
(Y-axis: Coefficient of Optimality, X-axis: Generation)
4.10 Experiment 8: Midsize / Five-Action / Divided-Controlled
The experiment was performed on a midsize problem with 6 cells, 4 agents and a total of
9 parts in the system. The other system parameters are given in the settings section
below. This experiment has the Tr, Tl, Gr and Gl actions disabled, thus the agents can
only collaboratively transport parts to the dump cell by using the Ml, Mr, Tl, Tb and Dn
actions. This limitation makes it much harder for the collaborative agents to show
0
0.25
0.5
0.75
1
1 21
(max-pgs)/max (ls/max) (cfg/max)
181
progress and more steps will be required in the simulation to move all 9 parts to the dump
cell.
The agents in this experiment are not all governed by their own individual controller, nor
is this a ‘bookend’ approach. In this ‘Divided’ approach, the first two agents are governed
by the one controller while the last two agents in the system are governed by a second
controller.
Figure 4.31: Initial Configuration of System for Experiment 8
Experiment 8: Settings
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = False: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = False: Action(7).Name = "Tr = Take part from the carrier in the right cell"
Action(8).Enabled = False: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = False: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
CellCount = 6
AgentCount = 4
MaxDumpCount = 9
StateCount = 64
ControllerPlan = "Divided"
AgentOneSenseRightFaulty = False
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
GenerationCount = 500
182
SolutionCount = 500
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
RouletteCrossParentFraction = 1
RouletteCrossParentMax = Int(RouletteCrossParentFraction * SolutionCount)
OutCrossParentFraction = 0.25
OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1
RandomCrossParentMax = Int(RandomCrossParentFraction * SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
Experiment 8: Generational Progress
G = Generation Number
S = Number of Steps to be performed (= # Agents * # Cycles)
D = Number of Parts moved to Dump Cell
PGS = Pre-Goal Steps achieved
LS = Legal Steps achieved out of S Steps
CFG: A weighted sum of the contents of the bench and carry-status of each agent
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
SIM: The true fitness measure of the system at the end of the simulation (maximization desired)
SimScore = ConfigScore + PostGoalSteps + LegalStepCount
“L” - All Steps Legal “G” - All Parts Transferred
1 S= 800 D= 1 PGS= 800 LS= 752 CFG= 1773 SIM= 2525 --
2 S= 800 D= 2 PGS= 800 LS= 790 CFG= 1827 SIM= 2617 --
3 S= 800 D= 2 PGS= 800 LS= 794 CFG= 1821 SIM= 2615 --
4 S= 800 D= 2 PGS= 800 LS= 779 CFG= 1833 SIM= 2612 --
5 S= 800 D= 1 PGS= 800 LS= 769 CFG= 1875 SIM= 2644 --
6 S= 800 D= 1 PGS= 800 LS= 766 CFG= 2199 SIM= 2965 --
7 S= 800 D= 3 PGS= 800 LS= 779 CFG= 2241 SIM= 3020 --
8 S= 800 D= 2 PGS= 800 LS= 787 CFG= 2451 SIM= 3238 --
9 S= 800 D= 2 PGS= 800 LS= 796 CFG= 2451 SIM= 3247 --
10 S= 800 D= 3 PGS= 800 LS= 776 CFG= 2685 SIM= 3461 --
11 S= 800 D= 3 PGS= 800 LS= 797 CFG= 2685 SIM= 3482 --
183
12 S= 800 D= 3 PGS= 800 LS= 798 CFG= 2577 SIM= 3375 --
13 S= 800 D= 3 PGS= 800 LS= 762 CFG= 2961 SIM= 3723 --
14 S= 800 D= 3 PGS= 800 LS= 794 CFG= 2961 SIM= 3755 --
15 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4707 SIM= 5507 L-
16 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4707 SIM= 5507 L-
17 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4707 SIM= 5507 L-
18 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4707 SIM= 5507 L-
19 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4707 SIM= 5507 L-
20 S= 800 D= 8 PGS= 800 LS= 791 CFG= 6075 SIM= 6866 --
21 S= 800 D= 8 PGS= 800 LS= 792 CFG= 6075 SIM= 6867 --
22 S= 800 D= 8 PGS= 800 LS= 796 CFG= 6075 SIM= 6871 --
23 S= 800 D= 6 PGS= 800 LS= 800 CFG= 4707 SIM= 5507 L-
24 S= 800 D= 7 PGS= 800 LS= 783 CFG= 5115 SIM= 5898 --
25 S= 800 D= 9 PGS= 299 LS= 748 CFG= 6561 SIM= 7810 --
26 S= 800 D= 9 PGS= 279 LS= 796 CFG= 6561 SIM= 7878 --
27 S= 800 D= 9 PGS= 235 LS= 800 CFG= 6561 SIM= 7926 LG
28 S= 800 D= 9 PGS= 235 LS= 800 CFG= 6561 SIM= 7926 LG
29 S= 800 D= 9 PGS= 235 LS= 800 CFG= 6561 SIM= 7926 LG
30 S= 800 D= 9 PGS= 235 LS= 800 CFG= 6561 SIM= 7926 LG
31 S= 800 D= 9 PGS= 235 LS= 800 CFG= 6561 SIM= 7926 LG
32 S= 800 D= 9 PGS= 231 LS= 798 CFG= 6561 SIM= 7928 --
33 S= 800 D= 9 PGS= 231 LS= 798 CFG= 6561 SIM= 7928 --
34 S= 800 D= 9 PGS= 223 LS= 799 CFG= 6561 SIM= 7937 --
35 S= 800 D= 9 PGS= 223 LS= 800 CFG= 6561 SIM= 7938 LG
36 S= 800 D= 9 PGS= 191 LS= 799 CFG= 6561 SIM= 7969 --
37 S= 800 D= 9 PGS= 191 LS= 799 CFG= 6561 SIM= 7969 --
38 S= 800 D= 9 PGS= 191 LS= 800 CFG= 6561 SIM= 7970 LG
SIMULATION STOPPED AFTER 100 MORE STATIC GENERATIONS
The method found an optimal solution score of 7970 signifying that the
collaborative agents successfully transferred all 9 parts to the dump cell. This goal was
accomplished in 191 legal steps within the simulation using the final evolved rule-sets
shown below. 191 steps is one step shy of 48 cycles within the simulation. The four
agents each take a consecutive step to complete one cycle. Note how the fitness measure
allows for multi-criteria evolution to occur. At times the evolutionary process will lean
towards transferring more parts to the dump cell even if it uses more illegal steps to
accomplish this. Other times the evolutionary process will seek to make all steps legal at
the expense of goal progress. Over time, the effect is to get better at each objective and
finally to settle down into the optimal, legal solution.
184
Experiment 8: Final Results
G: 38 SSTEPS= 800 DUMP= 9 PREGOALSTEPS= 191 POSTGOALSTEPS= 609
LEGALSTEPS= 800 CONFIG= 6561 SIMSCORE= 7970 LG
An output listing below shows the trace for each of the 191 steps. It shows the
step action, bench status, agent position status and agent carry status at each step. It is
essentially a simulation trace for the best solution in the last gene pool of the evolutionary
process.
TRACE: ORIG 1Gb 2Dn 3Dn 4Mr 1Ml 2Ml 3Ml 4Dn 1Tb 2Tb 3Tb 4Dn 1Gb 2Dn 3Mr
4Dn 1Tb 2Mr 3Mr 4Dn 1Mr 2Mr 3Dn 4Dn 1Dn 2Gb 3Gb 4Gb 1Dn 2Ml 3Ml 4Ml 1Gb
2Dn 3Dn 4Tb 1Ml 2Ml 3Tb 4Mr 1Tb 2Tb 3Mr 4Dn 1Gb 2Mr 3Gb 4Gb 1Tb 2Dn 3Ml
4Ml 1Mr 2Gb 3Dn 4Tb 1Gb 2Dn 3Dn 4Mr 1Ml 2Ml 3Ml 4Gb 1Tb 2Tb 3Tb 4Ml 1Gb
2Dn 3Mr 4Tb 1Tb 2Mr 3Dn 4Mr 1Mr 2Dn 3Mr 4Dn 1Gb 2Mr 3Dn 4Dn 1Mr 2Gb 3Gb
4Gb 1Ml 2Ml 3Ml 4Ml 1Ml 2Ml 3Ml 4Ml 1Dn 2Dn 3Dn 4Tb 1Dn 2Dn 3Dn 4Mr 1Dn
2Dn 3Tb 4Mr 1Dn 2Tb 3Mr 4Gb 1Dn 2Mr 3Mr 4Dn 1Mr 2Mr 3Gb 4Dn 1Dn 2Ml 3Ml
4Ml 1Dn 2Gb 3Dn 4Tb 1Ml 2Ml 3Tb 4Mr 1Dn 2Dn 3Mr 4Dn 1Dn 2Dn 3Gb 4Gb 1Dn
2Dn 3Ml 4Ml
1Dn 2Dn 3Dn 4Tb 1Dn 2Dn 3Ml 4Mr 1Dn 2Dn 3Tb 4Gb 1Dn 2Dn 3Mr 4Ml 1Dn 2Dn
3Gb 4Tb 1Dn 2Dn 3Tb 4Mr 1Dn 2Dn 3Mr 4Dn 1Dn 2Dn 3Gb 4Gb 1Dn 2Dn 3Ml 4Ml
1Dn 2Dn 3Dn 4Tb 1Dn 2Dn 3Ml 4Mr 1Dn 2Dn 3Mr 4Gb 1Dn 2Dn 3Mr 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn
3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
185
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn
2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn
4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn 1Dn 2Dn 3Dn 4Dn
BENCH: 302110 312110 312110 312110 312110 312110 312110 312110 312110 212110 202110 201110 201110
301110 301110 301110 301110 201110 201110 201110 201110 201110 201110 201110 201110 201110 201210
201220 201221 201221 201221 201221 201221 211221 211221 211221 211211 211211 211211 211111 211111
111111 101111 101111 101111 201111 201111 201121 201122 101122 101122 101122 101122 101122 102122
102122 102112 112112 112112 112112 112112 112112 112112 112112 112113 012113 002113 001113 001113
101113 101113 101113 101103 001103 001103 001103 001103 001103 001103 001103 001103 011103 011103
011103 011103 011103 011203 011213 011214 011214 011214 011214 011214 011214 011214 011214 011214
011214 011214 011214 011114 011114 011114 011114 011114 011114 011114 010114 010114 010114 000114
000114 000115 000115 000115 000115 000115 000115 000115 000125 000125 000125 000125 000125 000125
000125 001125 001125 001115 001115 001115 001015 001015 001015 001015 001015 001015 001015 001015
001025 001026 001026 001026 001026 001026
001026 001026 001026 001016 001016 001016 001016 001016 001016 001016 000016 000017 000017 000017
000017 000017 000017 000017 000117 000107 000107 000107 000007 000007 000007 000007 000007 000007
000007 000007 000017 000018 000018 000018 000018 000018 000018 000018 000018 000008 000008 000008
000008 000008 000008 000008 000008 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
186
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009 000009
000009 000009
AGENT: 012340 012340 012340 012340 012304 102304 120304 123004 123004 123004 123004 123004 123004
123004 123004 120304 120304 120304 102304 102034 102034 012034 010234 010234 010234 010234 010234
010234 010234 010234 012034 012304 012340 012340 012340 012340 012340 102340 120340 120340 120304
120304 120304 120034 120034 120034 102034 102034 102034 102034 102034 102304 102340 012340 012340
012340 012340 012340 012340 012340 012304 102304 120304 123004 123004 123004 123004 123004 123040
123040 123040 120340 120340 120340 102340 102340 102304 012304 012304 012034 012034 012034 010234
010234 010234 001234 001234 001234 001234 010234 012034 012304 012340 102340 120340 123040 123400
123400 123400 123400 123400 123400 123400 123400 123040 123040 123040 123040 123004 123004 123004
120304 120304 120304 102304 102034 102034 012034 010234 010234 010234 010234 012034 012304 012340
012340 012340 012340 012340 102340 120340 120340 120304 120304 120304 120034 120034 120034 120034
120034 120034 120034 120034 120304 120340
120340 120340 120340 120340 120340 120340 123040 123004 123004 123004 123004 123004 123004 123004
120304 120340 120340 120340 120340 120340 120340 120340 120340 120304 120304 120304 120034 120034
120034 120034 120034 120034 120034 120034 120304 120340 120340 120340 120340 120340 120340 120340
123040 123004 123004 123004 120304 120304 120304 120304 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
187
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034 120034
120034 120034
CARRY: 010010 000010 000010 000010 000001 000001 000001 000001 000001 100001 110001 111001 111001
011001 011001 010101 010101 110101 101101 101011 101011 011011 010111 010111 010111 010111 010011
010001 010000 010000 010000 010000 010000 000000 000000 000000 000010 000010 000010 000110 000101
100101 110101 110011 110011 010011 001011 001001 001000 101000 101000 101000 101000 011000 010000
010000 010010 000010 000010 000010 000001 000001 000001 000001 000000 100000 110000 111000 111000
011000 011000 010100 010110 110110 101110 101110 101101 011101 011101 011011 011011 001011 000111
000111 000111 000111 000011 000001 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000100 000100 000100 000100 000010 000010 000010 001010 001001 001001 011001
010101 010100 010100 001100 001010 001010 001010 000110 000100 000100 000100 001000 001000 001000
001000 000000 000000 000010 000010 000010 000110 000101 000101 000101 000011 000011 000011 000011
000001 000000 000000 000000 000000 000000
000000 000000 000000 000010 000010 000010 000010 000001 000001 000001 001001 001000 001000 001000
000100 000100 000100 000100 000000 000010 000010 000010 000110 000101 000101 000101 000011 000011
000011 000011 000001 000000 000000 000000 000000 000000 000000 000000 000000 000010 000010 000010
000010 000001 000001 000001 000001 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
188
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
000000 000000
Experiment 8: Evolved Rule-Sets
Below are shown the evolved rule-sets for the system. Since the Divided controller plan
was used, two distinct rule-sets were evolved, one for the first two agents and one for the
last two agents. Each rule-set below is preceded by a number reflecting the number of
rules that were evolved within the rule-set. Each rule is of the format ‘C1C2C3C4-
Action’ where a series of condition settings have an associated subscribed action. Note
that the total number of possible rules is equal to the total number of possible condition
states which in this problem is 64. The evolution of a reduced number of rules is a
significant improvement over trying to pre-determine all 64 rules for each agent outside
of an evolutionary learning process. Keep in mind also that for every condition state there
are nine possible actions. Note also that the disabling of side-to-side taking and giving
actions has forced the agents to rely on the more awkward and laborious bench passing
189
and cell shifting in order to accomplish part transfer in the system. The rule-sets below is
a reduced set of rules necessary to elicit optimal collaborative performance within the
simulation. Operating the agents by this common rule-set achieves the necessary inter-
dependence that leads to collaborative performance. Single controller was evolved in
only 38 generations. This controller took 191 steps in the simulation to reach the
collaborative goal. The number of rules evolved for each controller is 47 and 44
respectively for the two controllers out of a possible 64. Thus some states were never and
could never be visited.
# 47 [ShBnLvRn-Gb SnBhLhRn-Tb ShBhLwRv-Mr ShBnLvRh-Gb ShBhLvRn-Ml SnBhLhRh-Tb
SnBhLhRv-Tb ShBhLhRv-Mr ShBnLvRv-Dn ShBhLvRv-Dn ShBhLvRh-Gb SnBhLnRn-Dn SnBhLvRh-
Tb ShBhLhRn-Gb SnBhLvRv-Mr SnBhLnRv-Dn SnBhLwRv-Tb ShBhLhRh-Dn SnBhLwRn-Tb
ShBhLnRv-Mr SnBhLnRh-Tb ShBnLhRv-Mr SnBnLvRh-Dn ShBhLnRn-Gb ShBhLnRh-Gb ShBnLwRv-
Mr ShBhLwRh-Gb SnBhLvRn-Ml ShBnLnRv-Mr ShBnLhRh-Gb ShBnLnRh-Dn SnBnLvRv-Dn
SnBnLwRn-Dn SnBhLwRh-Tb ShBnLhRn-Dn SnBnLhRv-Mr SnBnLvRn-Ml ShBnLnRn-Gb SnBnLnRv-
Dn SnBnLwRv-Mr ShBnLwRn-Dn SnBnLhRn-Dn SnBnLwRh-Dn SnBnLnRn-Dn SnBnLhRh-Dn
ShBnLwRh-Gb SnBnLnRh-Dn ]
# 44 [SnBhLvRh-Tb ShBnLvRw-Dn ShBhLhRv-Mr ShBnLhRv-Mr SnBhLvRn-Ml SnBhLvRw-Ml
SnBhLnRh-Dn ShBnLnRv-Mr ShBhLnRv-Mr SnBhLnRn-Dn SnBhLnRv-Tb SnBhLvRv-Ml ShBnLvRv-
Mr ShBnLnRw-Gb SnBhLhRn-Dn ShBhLvRn-Gb SnBhLhRw-Dn ShBhLhRh-Dn SnBhLnRw-Dn
ShBnLhRh-Dn ShBnLnRn-Gb ShBnLvRn-Gb ShBhLhRw-Dn ShBhLnRw-Gb SnBnLnRv-Mr SnBnLvRv-
Mr SnBnLhRv-Mr ShBnLhRn-Dn ShBnLvRh-Gb ShBhLhRn-Gb ShBhLvRv-Mr SnBnLvRn-Dn
SnBnLvRh-Ml SnBhLhRh-Dn ShBnLhRw-Dn SnBnLnRn-Dn ShBhLvRh-Gb ShBhLnRn-Dn ShBhLvRw-
Gb ShBnLnRh-Gb ShBhLnRh-Gb SnBnLhRn-Dn SnBhLhRv-Tb SnBnLnRh-Dn ]
Experiment 8: Graph Analysis
Figure 4.32 below shows the fitness score of every fifth solution ranked low to high
within the gene pool in every generation leading up to the final 38th generation. This
graph reflects the changing nature of the gene pool throughout the evolutionary process.
190
Figure 4.32: Evolving Fitness Scores of Experiment 8
(Vertical Y-axis: Fitness, Horiz. X-axis: Generation, Inward Z-axis: Solution Index)
Figure 4.33 below reflects the fitness measure of the best solution in each gene
pool from generation 1 to generation 38.
1 7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
S1
S17
S33
0
1000
2000
3000
4000
5000
6000
7000
8000
191
Figure 4.33: Generational Fitness for Experiment 8
(Y-axis: Fitness Measure, X-axis: Generation)
Figure 4.34 below reflects the above fitness measure broken down in to its three
components. The pre-goal steps (pgs) is the number of simulation steps before the system
successfully achieves (and maintains) its goal state of all system parts transferred to the
dump cell. The blue line denotes the maximum number of steps in the simulation (800)
minus the pre-goal steps. It is desired to maximize this quantity. Similarly maximized is
the number of legal steps out of a possible 800 (ls) as denoted by the pink line. The
solution is not optimal unless all steps taken within the simulation do not prompt a
failure. The third component is the configscore (cfg) which for this problem is maximally
6561, a weighted value representing all parts in the system transferred to the dump cell.
This value is denoted by the yellow line. These three values are divided by their
maximum values and the ratios are plotted below. Note that not all solutions leading to a
goal are achieved through legal steps (yellow optimal but pink suboptimal). Note also
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
192
that in early generations, a solution can be comprised entirely of legal steps and yet not
achieve goal behavior (pink optimal but yellow suboptimal).
Figure 4.34: Scores of Three Components of Fitness for Experiment 8
(Y-axis: Coefficient of Optimality, X-axis: Generation)
4.11 Experimentation Results
Many more experiments could have been performed given the program that was written.
These experiments include looking at the effect of varying controller designations, initial
system configurations, controller seeding, sensor failure introduction, genetic process
parameters and methods, simulation parameters and methods, fitness functions, etc. The
method could be applied to a larger problem such as an expanded grid space or with two
or three agent group types each with their own set of sensor and action capabilities. For
the moment, a discussion below will compare results of the 9-action vs. the 5-action
experiments for the different controller plan designations.
0
0.25
0.5
0.75
1
1 21
(max-pgs)/max (ls/max) (cfg/max)
193
The evolutionary method devised was able to successfully evolve rule-sets which
govern the behavior of agents in a simulation of the midsize AGV problem system. The
rule-sets successfully achieve collaborative behavior even though each agent in the
system had a limited, localized knowledge of its environment during the evolutionary
process. The individual, bookend, divided and universal controller designations resulted
in 4, 3, 2 and 1 rule-set(s) accordingly for the 4 agents in the system.
Nine-Action Experiments: All Actions Enabled
The experiments were run for the problem with all nine actions enabled. The rule-sets as
detailed in the above sections demonstrate that a strategy of passing parts was adopted
when all nine actions were enabled.
The individual controller plan evolved 24 rules for the leftmost agent and 24 for
the rightmost agent. The rule-sets for these two types of agents differ but mostly with
regards to left- or righthandedness of the conditional states. The middle two agents both
yielded evolved rule-sets each containing 36 rules. This, again, does not indicate the rule-
sets are identical but rather indicates the logic involved is comparable for both types of
agents.
The bookends approach had the middle two agents influencing the evolution of
the second controller and they in turn were ultimately governed by this controller. Note
that the central rule-set needed only 36 rules to control both agents B and C. This is the
same number of rules needed to govern both individually under the individual controller
plan. This is a reasonable result given that the functionality of agents B and C are
principally the same within the system. They are both denied experiencing a wall as they
194
cannot pass the two agents on either end of the corridor. They both pass to and from the
bench and to other agents in a similar fashion. This result suggests the appropriateness of
combining agents B and C into the same agent group so that their behavior can be
governed by the same set of rules.
The divided plan convolved the control of agents A and B together as well as the
control of agents C and D. The number of rules in each rule-set increased to 47 since it
had to accommodate the logic of both the 24-rule individual rule-set of agent A and the
36-rule individual rule-set of agent B (though some common rules likely applied to both).
The universal plan grouped all agents into the same agent group and resulted in a single
controller with 57 rules out of a possible 64. There were 64-57=7 conditional states that
never occur in the operation of the system thus these rules were never evolved during the
evolutionary learning process. This demonstrates how the rule-set which evolves for a
given controller is always compact in its sufficiency.
Note that the number of simulation steps necessary to accomplish goal behavior is
55 in the case of the nine-action problem for the first three controller designations. The
universal plan was only able to find a solution in 59 steps after 73 generations. Use of a
standard genetic algorithm string encoding for this problem would have entailed
thousands if not millions of generations to yield a similarly effective set of controller
mechanisms. The evolutionary learning strategy drastically reduced the number of
generations necessary to find an optimal solution to the problem by concentrating on the
evolution on reduced rule-sets in a simulation-learning scenario.
195
Table 4.1 Midsize Problem Results: Nine Actions vs. Five Actions
Five-Action Experiments: Agent Passing Actions Disabled
The results of the method as applied to the five-action problem are given in above
experiment sections as well as summarized in the Table 4.1 above. The five-action
problem is identical to the nine-action problem except for the passing actions between
agents having been disabled. The nine-action problem evolved a strategy that involved
taking parts from the bench and moving or passing them down the line of agents
whenever possible. The collaborative rule-sets evolved for this problem were able to find
an effective and compact rule-set in 55 simulation steps. In the five-action system, the
agents do not have the benefit of passing between each other so an alternative strategy
must be evolved to transfer all parts down the line towards the dump cell. This strategy
must involve removing a part from the bench, carrying the part into open cells to the
196
right of the agent, redepositing the part on the bench, then moving back to the left to
allow another agent to come in from the right to acquire the part. This process involves
many more steps than when interagent passing was allowed. One would expect the goal
to be reached in far more simulation steps than the 55 steps required by the nine-action
problem.
As shown above in Table 4.1, though the number of rules evolved in the rule-set
is roughly the same as in the nine-action experiments, the five-action experiments do
indeed take more steps in the simulation to reach their goal. The individual controller
designation plan achieves a goal state in 183 steps after 196 generations. As the number
of controllers decreases, the number of generations required also decreases. Bookends
designation with 3 controllers finds the goal in 179 steps after 45 generations. The
Divided plan with 2 controllers achieves the goal in 191 steps in 36 generations. The
Universal plan with one controller to rule all agents achieves the goal in 195 steps after
27 generations. What is notable is that with all 9 actions enabled in the first set of
experiments, rule-sets were evolved which allow the agents to find the goal in roughly 55
steps, whereas with only 5 actions enabled the evolved rule-sets take nearly four times as
many simulation steps to achieve collaborative goal behavior. It should be noted that
these experiments were run once for each controller designation. Multiple runs would
yield slightly different solutions with slightly different rule-sets. However, the general
strategies differentiating the nine-enabled from the five-enabled problems would still be
reflected in the results.
Experiments such as this can demonstrate how much a system relies upon a
particular behavior in order to achieve goal behavior. The most efficient subset of atomic
197
behaviors may be determined through this form of experimentation. By performing
additional experiments, it may be possible with this method to optimize the
collaborative goal achievement of a multi-agent system with regards to agent class
designation, agent behavior selection and other parameters.
198
Chapter Five: Generalizability & Scalability
In the course of this dissertation, the development of a method was documented and it
was applied to a sample problem of multiple collaborating AGVs within a small
manufacturing cell. Additional discussion is necessary to reflect on which aspects of the
method are generalizable and scalable to more complex problems and which aspects are
grounded to the specific application problem and need to be altered if the application
changes.
Within the representation objective, the probabilistic finite state automata, which
models the intelligent controller of an agent, was encoded as a series of action probability
vectors associated with each state transition. Only those rules encountered within the
simulation runs were required to be stored thus achieving compactness of the rules to be
evolved. For larger application problems, there are more conditions and actions inherent
to the system. This necessarily makes the complexity and size of the associated automata
larger. This translates to more potential states encountered within the simulation and
therefore more potential rules stored. But this is exactly where the method is scalable
because it still only needs to store rules associated with those states visited. Deriving the
solution by inspection, for example, would likely require a determination of action for
every combination of conditions in the system which quickly becomes exhaustive as the
problems become bigger. One way in which the rule format developed is not as easily
generalizable is when you are dealing with systems which are not engaging in simple
atomic actions. Complex systems with layered behavior architectures deal with more
involved procedures to be enacted based on conditional states as opposed to the simpler
199
actions made as shown in development of this method. But it may be possible to still use
this method for larger problems. For example, instead of a rule dictating a simple action it
may dictate a function call to be made. This is imagined as part of the future work to be
done using this method.
In the course of development of this method, a simulation environment was
required to test the performance of the pool of controller solutions. Because the
simulation is a feature nested within each iteration of the evolutionary process, it would
have been difficult to use pre-existing, more sophisticated simulation software. This is
because there was a need to perform reinforcement learning of the agents within the
simulation and to pass performance measures of the agents to the evolutionary process
which is wrapped around it. The need to control and integrate these features of the
method necessitated that it all be written in one program for the purposes of this research.
For larger problems, it would be difficult to write such a program. The more complex
features of professional simulation software would be preferred to visualize the behavior
of the agents within the environment and to provide performance statistics. It still leaves
the problem of needing reinforcement learning as part of the simulation. Perhaps with the
growing interest in swarm applications, simulation packages will start to feature
probabilistic features as shown in this method and may even make themselves more
easily integrated with wraparound evolutionary learning capabilities. The use of
probabilistic rules with reinforcement learning is not ground to the particular application
problem of this research. But the simulation software would be need to be more
sophisticated and would need to be part of a more encompassing evolutionary tool for
development of intelligent controllers for multi-agent systems.
200
The evolutionary process and its parameters are most certainly scalable and
generalizable to larger problems. As long as the pool of controller solutions are cast in the
same format of nested probabilistic vectors, this method can be applied. One of the nicer
features of the method is the enhanced use of genetic operators to maintain a disruptive
gene pool of solutions and to encourage a fuzzy progression towards optimality so as to
avoid local optima. This approach should be applicable and effective to larger problems
as well.
Of course, there are other parameters and functions that are required to be tailored
to every new application. The conditions and actions of the system must be determined
from the outset and this will affect the length of the controller rules but not necessarily
how it functions within the method. The performance measure functions are used within
the simulation to determine how well the agents accomplished their individual and global
tasks. These performance measures are not generalizable and must be altered given the
different collaborative tasks associated with the problem. However, tailoring performance
measures to larger problems isn’t necessarily something which will make the method
unusable, it will merely take longer time.
In the results section, numerous experiments were done to evolve the intelligent
control rules in the presence of a sensor fault or with loss of agent functionality. Other
experiments were done to determine what the best allocation of controller types was for a
heterogeneous agent system. These types of experiments could be performed with other
applications as well. As shown in the experiments done, the use of multiple controller
types for different agents is easily accommodated by this method.
201
Chapter Six: Uniqueness & Conclusions
The method developed in the course of this research was demonstrated through its
application to a scenario of a series of AGVs collaborating together in a manufacturing
cell. It was shown to successfully evolve intelligent control rules for the governance of
behavior in the system. The method was then utilized to run numerous experiments where
the functionality of the agents or its sensors was changed, or the designation of controller
logic was varied amongst the agents. Running these experiments provided insight into
how behavior of the system changes when the functionality of an agent is altered or when
the sharing designation of controller logic is varied. These experiments show differences
in the number of steps and the number of generations required to find the optimal
solutions under the given circumstances. In this particular problem scenario, it was found
that using ‘bookend’ controller designation with all nine agent actions enabled is the most
efficient allocation. This makes sense as the functionality of agents on either end of the
work cell are similar and it is reasonable they are governed by the same controller and
benefit from each other’s experiences during the simulation training part of the method.
This type of experiment demonstrates the usefulness of the method as a tool to optimize
the configuration of a multi-agent system.
The research area of evolutionary learning techniques for the intelligent control of
multi-agent systems is still an evolving field. The imagined applications far exceed the
current technical capability of this research area. Typically, control strategies are
expressed as a series of complex interacting finite state automata. Programs explicitly
202
code all of the behavioral states within the automata and the necessary conditions which
dictate agent actions which accomplish the state to state transitions. This programming
can only be performed if the controller automata model is known beforehand, and even
then, the process to properly model an IC program is labor intensive and ultimately
inflexible for future adaptation. Simply put, a new approach which allows for automata
code to evolve on its own to accommodate the complex and often unforeseeable aspects
of an autonomous multi-agent system is in dire need. Additionally, such a code would
ideally be able to adapt on its own to changing situational factors and respond by
synthesizing new behavioral patterns. For example, when an evolved controller code is
implemented and it encounters a situation that is heretofore unknown, the agent could
enter an exploratory mode in which it attempts to problem-solve by engaging in trial
experimentation with the elemental actions available to it. Ideally, a great deal of this
unforeseeable learning could take place in a simulation facility which finetunes the
evolutionary intelligent controller. In this fashion, these autonomous agents could
become truly autonomous, alleviating themselves of the age-old problem of a
compendium of complex human-initiated rules that crumbles under its own weight
rendering itself obsolete. The key is to provide a simulation-based evolutionary learning
intelligent controller which can learn and adapt autonomously.
The method developed in this dissertation is unique in a number of ways. The
encoding of the state transition rules of a probabilistic finite state automaton as a
compactly expanding series of probabilistic vectors is novel. The use of more genetic
operators than is customary is also unique to this method. Both aspects point to an effort
to create a fuzzy and disruptive evolutionary technique to avoid local optima while
203
successfully accomplishing collaborative multi-agent behavior. The probabilistic solution
vector format used also was done with an eye towards the need for multi-agent systems to
continue to adapt when deployed in remote, autonomous, dynamic environments. The use
of reinforcement learning as well as the maintaining of a diverse gene pool may allow for
controller solutions to migrate away from well-learned behaviors towards new ones if
changes in the environment or the system require adaptation.
The specific problem studied in this research may be very simple, but in theory it
can be scaled up to address any problem that can be modeled in automata form (see
Chapter 5). The novel approaches to the evolutionary heuristic devised in this research, as
described in the report, are a step towards the future development of self-learning and
self-adapting intelligent controllers for use in numerous collaborative autonomous multi-
agent system applications.
204
Chapter Seven: Future Work
As discussed above, the program and method developed could serve as the basis for
many other experiments. There are many directions that this research could take. The
experiments could be repeated for different size AGV-corridor problems with different
initial configurations. Experiments could be conducted with respect to the variation of
genetic, simulation and fitness parameters. Alternative controller designations could also
be tested. The program also allows for a sensor failure to be implemented in an agent.
Preliminary results indicate this failure does not impede evolutionary progress towards
controller development since this particular evolutionary method utilizes repeated,
exhaustive exposure to conditional states within the myriad simulation learning phases of
the genetic process. Sensor failure would, however, figure into the performance of the
system once the controller is evolved. Other experiments could test the effects of seeding
the controllers with advanced knowledge. The evolutionary learning method that was
devised could also allow for flexible adaptation of a controller rule-set to new conditions.
This is because the use of probabilistic vectors to encode the transitional rules can be
evolved towards new adaptive solutions once deployed in a remote or dynamic
environment. This could serve as an ideal testbed for experiments involving a set of
controllers re-adapting to changing environmental circumstances which is of great
interest in the burgeoning field of intelligent control for swarm technologies. Another
future direction would be to study the method in a shopfloor with real-time information
from Internet-of-Things (IOT) enabled AGVs and Machines.
205
APPENDIX: Main Program Code
'**************************************************************************************
'MARC LATORELLA PENN STATE UNIVERSITY DEPT. OF INDUSTRIAL ENGINEERING
'**************************************************************************************
Public Type ConditionType 'System Conditions each with a number of choices
Name As String * 15
ChoiceCount As Integer
ChoiceName(10) As String * 20
ChoiceID(10) As String * 2
End Type
Public Type ActionType 'System Conditions each with a number of choices
Name As String * 15
ID As String * 2
Enabled As Boolean
End Type
Public InitBenchString As String 'number of parts in each bench cell
Public InitAgentString As String 'cell location of each agent
Public InitCarryString As String 'number of parts held by each agent
Public InitConfigScore As Integer
Public MaxConfigScore As Integer
Public InitActionHistory As String
Public InitBenchHistory As String
Public InitAgentHistory As String
Public InitCarryHistory As String
Public BenchString As String 'number of parts in each bench cell
Public AgentString As String 'cell location of each agent
Public CarryString As String 'number of parts held by each agent
Public Type ControllerType
Rules(10) As String
End Type
Public Type SimDataType
SimScore As Double
ConfigScore As Double
LegalStepCount As Integer
GoalStepCount As Integer
LegalPreGoalStepCount As Integer
DumpCount As Integer
ActionHistory As String
BenchHistory As String
AgentHistory As String
CarryHistory As String
End Type
Public Type SolutionType
TrainedController As ControllerType
TrainedSimData As SimDataType
206
'TestedSimData As SimDataType
CreatedController As ControllerType
End Type
'------------------------------------------------------------------------------------
Public OutputFileName1, OutputFileName2 As String
Public GenOutputCurb As Integer
Public PoolOutputCurb As Integer
Public TopPrint As Integer
Public HistorySpacer As String
Public RuleLength As Integer
Public HistoryLength As Integer
Public MiniStepCount As Integer
Public RuleDashPos As Integer
Public RuleStateLength As Integer
Public StateCount As Integer
Public GenerationCount As Integer
Public SolutionCount As Integer
Public ControllerCount As Integer
Public AgentCount As Integer
Public CycleCount As Integer
Public StepCount As Integer
Public CellCount As Integer
Public ActionCount As Integer
Public ConditionCount As Integer
Public FitnessCoefficient As Double
Public Condition(20) As ConditionType
Public Action(20) As ActionType
Public ActionString As String
Public BenchPartLimit As Integer
Public RestockOnDump As Boolean
Public RestockOnLoad As Boolean
Public MaxDumpCount As Integer
Public DumpCount As Integer
Public DumpGoalPartialSteps As Integer
Public AgentControllerIndex(10) As Integer
Public AgentOneSenseRightFaulty As Boolean
Public DumpScoreCoeff As Integer
Public BenchScoreBase As Integer
Public CarryScoreBase As Integer
Public RouletteCrossParentFraction As Double
Public OutCrossParentFraction As Double
Public RandomCrossParentFraction As Double
Public RandomChoiceFraction As Double
Public RouletteCrossParentMax As Integer
Public OutCrossParentMax As Integer
Public RandomCrossParentMax As Integer
Public RandomChoiceMax As Integer
207
Public RouletteCrossPercentage As Double
Public OutCrossPercentage As Double
Public RandomCrossPercentage As Double
Public RandomChoicePercentage As Double
Public EliteChoicePercentage As Double
Public WorstChoicePercentage As Double
Public RouletteCrossCount As Integer
Public OutCrossCount As Integer
Public RandomCrossCount As Integer
Public RandomChoiceCount As Integer
Public EliteChoiceCount As Integer
Public WorstChoiceCount As Integer
Public ProbChromosomeCrossoverPoint As Double
Public ProbGeneCrossoverPoint As Double
Public MutationDamper As Integer
Public RankRoulettePower As Double
Public ScoreRouletteMethod As Boolean
Public CurrSolutions(3000) As SolutionType
Public NextSolutions(3000) As SolutionType 'next generation's gene pool
Public BestSolutions(3000) As SolutionType
Public SeededController As ControllerType
Public GenerationIndex As Integer
Public SolutionIndex As Integer
Public CycleIndex As Integer
Public AgentIndex As Integer
Public ControllerPlan As String
Public PrintAll As Boolean
Public PrintIt As Boolean
Public GoalGenerations As Integer
Public Comment As String
Public StaticCount As Integer
Public StaticScore As Integer
Public StaticSolution As Boolean
Public StabilitySteps As Integer
Public GoalFound As Boolean
Public LegalFound As Boolean
Public SolutionFound As Boolean
Public PreGoalStepCount As Integer
Public SimScoreHistory As String
Public StablePGS As Integer
Public StableGens As Integer
Public Sub Main()
Call Setup
Call PoolCreate 'must create the pool of controller-tuple solutions
208
StaticSolution = False
StaticCount = 0
StaticScore = 0
LastScore = 0
StabilitySteps = 0 ' 2 * CellCount
StableGens = 0
StablePGS = 0
For GenerationIndex = 1 To 2000 'GenerationCount 'run the process for a specified # of generations
Call PoolTest 'must evaluate the solution pool of current generation
Call PoolSort
SimScoreHistory = ""
For i = 1 To SolutionCount Step PoolOutputCurb
SimScoreHistory = CStr(CurrSolutions(i).TrainedSimData.SimScore) + " " + SimScoreHistory
Next i
Write #1, SimScoreHistory
With CurrSolutions(1).TrainedSimData
GoalFound = (.DumpCount = MaxDumpCount): LegalFound = (.LegalStepCount =
StepCount)
PreGoalStepCount = StepCount - .GoalStepCount: SolutionFound = GoalFound And
LegalFound And .GoalStepCount > StabilitySteps
If (StablePGS <> PreGoalStepCount) Or (Not SolutionFound) Then
StablePGS = PreGoalStepCount
StableGens = 0
ElseIf (StablePGS = PreGoalStepCount) And SolutionFound Then
StableGens = StableGens + 1
If StableGens = 100 Then Exit For
End If
If SolutionFound Then
Comment = "LG"
ElseIf GoalFound > 0 Then
Comment = "-G"
ElseIf LegalFound Then
Comment = "L-"
Else
Comment = "--"
End If
Debug.Print GenerationIndex; "S="; StepCount; "D="; .DumpCount; "PGS="; PreGoalStepCount;
"LS="; .LegalStepCount; "CFG="; .ConfigScore; "SIM="; .SimScore; Comment
' Debug.Print " TRACE: " + .ActionHistory
' Debug.Print " BENCH: " + .BenchHistory
' Debug.Print " AGENT: " + .AgentHistory
' Debug.Print " CARRY: " + .CarryHistory
End With
If False Then
With CurrSolutions(1).TrainedSimData
GoalFound = (.DumpCount = MaxDumpCount): LegalFound = (.LegalStepCount =
StepCount)
PreGoalStepCount = StepCount - .GoalStepCount: SolutionFound = GoalFound And
LegalFound And .GoalStepCount > StabilitySteps
If SolutionFound Then
Comment = "LG"
'If .GoalStepCount > 220 Then Exit For
209
ElseIf GoalFound > 0 Then
Comment = "-G"
ElseIf LegalFound Then
Comment = "L-"
Else
Comment = "--"
End If
Debug.Print " *** D="; .DumpCount; "PGS="; PreGoalStepCount; "LS="; .LegalStepCount;
"CFG="; .ConfigScore; "SIM="; .SimScore; Comment; ""
If .SimScore = LastScore Then
StaticCount = StaticCount + 1
Else
LastScore = .SimScore
StaticCount = 0
End If
If StaticCount = 3000 Then
Exit For
'StaticCount = 0
'LastScore = 0
'CycleCount = CycleCount + 5
'StepCount = CycleCount * AgentCount
End If
If False Then
If .GoalStepCount > 0 And .LegalStepCount = StepCount Then
CycleCount = Int((StepCount - .GoalStepCount) / AgentCount + 0.5 + StabilitySteps)
StepCount = CycleCount * AgentCount
End If
End If
End With
End If
'If True Then Exit For '88888888888888888888888888
Call PoolBreed 'create the next generation of solutions
Next GenerationIndex
With CurrSolutions(1).TrainedSimData
Debug.Print "G:"; GenerationIndex; "SSTEPS="; StepCount; "DUMP="; .DumpCount;
"PREGOALSTEPS= "; PreGoalStepCount; "POSTGOALSTEPS= "; .GoalStepCount; "LEGALSTEPS= ";
.LegalStepCount; "CONFIG="; .ConfigScore; "SIMSCORE="; .SimScore; Comment
If SolutionFound Then
GoalGenerations = GoalGenerations + 1
Debug.Print
Debug.Print " TRACE: " + .ActionHistory
Debug.Print " BENCH: " + .BenchHistory
Debug.Print " AGENT: " + .AgentHistory
Debug.Print " CARRY: " + .CarryHistory
Debug.Print
For j = 1 To ControllerCount
Debug.Print " #"; Len(CurrSolutions(1).TrainedController.Rules(j)) / RuleLength; "[";
CurrSolutions(1).TrainedController.Rules(j); "] "
Next j
Debug.Print "G:"; GenerationIndex; "SIMSCORE="; .SimScore; "CFGSCORE="; .ConfigScore;
"GOALS="; .DumpCount; "LEGALSTEPS= "; .LegalStepCount; "PostGoalSTEPS= "; .GoalStepCount;
"STEPS="; StepCount
'If .GoalStepCount>CellCount * AgentCount Then Exit For
210
'If GoalGenerations > 10 Then Exit For
'Exit For
'Else
' GoalGenerations = 0
End If
End With
For i = 1 To 50: Beep: Next i
Close #1
End Sub
Public Sub Setup()
Dim i As Integer
Dim j As Integer
Call Randomize
PrintAll = False
StabilitySteps = 0
'set the conditions and their choices
ConditionCount = 4
Condition(1).Name = "Self Sensor": Condition(1).ChoiceCount = 2
Condition(1).ChoiceName(1) = "holding a part"
Condition(1).ChoiceName(2) = "not holding a part"
Condition(2).Name = "Bench Sensor": Condition(2).ChoiceCount = 2
Condition(2).ChoiceName(1) = "holding a part"
Condition(2).ChoiceName(2) = "not holding a part"
Condition(3).Name = "Left Sensor": Condition(3).ChoiceCount = 4
Condition(3).ChoiceName(1) = "holding a part"
Condition(3).ChoiceName(2) = "not holding a part"
Condition(3).ChoiceName(3) = "void"
Condition(3).ChoiceName(4) = "wall"
Condition(4).Name = "Right Sensor": Condition(4).ChoiceCount = 4
Condition(4).ChoiceName(1) = "holding a part"
Condition(4).ChoiceName(2) = "not holding a part"
Condition(4).ChoiceName(3) = "void"
Condition(4).ChoiceName(4) = "wall"
For i = 1 To ConditionCount
CondLetter = Left(Condition(i).Name, 1)
For j = 1 To Condition(i).ChoiceCount
Condition(i).ChoiceID(j) = CondLetter + Left(Condition(i).ChoiceName(j), 1)
Next j
Next i
'set the actions
ActionCount = 9
Action(1).Enabled = True: Action(1).Name = "Ml = Move to the left cell"
Action(2).Enabled = True: Action(2).Name = "Mr = Move to the right cell"
Action(3).Enabled = True: Action(3).Name = "Tb = Take part from the bench"
Action(4).Enabled = True: Action(4).Name = "Gb = Give part to the bench"
Action(5).Enabled = True: Action(5).Name = "Dn = Do nothing"
Action(6).Enabled = True: Action(6).Name = "Tl = Take part from the carrier in the left cell"
Action(7).Enabled = True: Action(7).Name = "Tr = Take part from the carrier in the right cell"
211
Action(8).Enabled = True: Action(8).Name = "Gl = Give part to the loiterer in the left cell"
Action(9).Enabled = True: Action(9).Name = "Gr = Give part to the loiterer in the right cell"
ActionString = ""
For i = 1 To ActionCount
Action(i).ID = Left(Action(i).Name, 2)
ActionString = ActionString + Action(i).ID
Next i
'system configuration parameters
' InitBenchString = "300": InitAgentString = "012": InitCarryString = "010"
InitBenchString = "302110": InitAgentString = "012340": InitCarryString = "010010" 'standard mid
problem
' InitBenchString = "1001100": InitAgentString = "0123450": InitCarryString = "0121010" 'alternate
mid problem
'InitBenchString = "0211002100": InitAgentString = "1020340567": InitCarryString = "0010000010"
'harder problem
CellCount = Len(InitBenchString)
AgentCount = 0
For i = 1 To CellCount
If Mid(InitAgentString, i, 1) <> "0" Then AgentCount = AgentCount + 1
Next i
MaxDumpCount = 0
For i = 1 To CellCount
MaxDumpCount = MaxDumpCount + Mid(InitBenchString, i, 1) + Mid(InitCarryString, i, 1)
Next i
StateCount = 1
For i = 1 To ConditionCount
StateCount = StateCount * Condition(i).ChoiceCount
Next i
'controller parameters
ControllerPlan = "Universal" '"Divided" '"Universal" ' "Bookends" '"Individual" '"Divided" '"Universal"
'"Bookends" '"Specified" '
Select Case ControllerPlan
Case "Divided":
ControllerCount = 2
For i = 1 To Int(AgentCount / 2)
AgentControllerIndex(i) = 1
Next i
For i = Int(AgentCount / 2) + 1 To AgentCount
AgentControllerIndex(i) = 2
Next i
Case "Individual":
ControllerCount = AgentCount
For i = 1 To AgentCount
AgentControllerIndex(i) = i
Next i
Case "Universal":
ControllerCount = 1
For i = 1 To AgentCount
AgentControllerIndex(i) = 1
Next i
Case "Bookends":
212
ControllerCount = 3
AgentControllerIndex(1) = 1
For i = 2 To AgentCount - 1
AgentControllerIndex(i) = 2
Next i
AgentControllerIndex(AgentCount) = 3
Case "Specified":
ControllerCount = 5
AgentControllerIndex(i) = 1: AgentControllerIndex(i) = 2: AgentControllerIndex(i) = 3
AgentControllerIndex(i) = 4: AgentControllerIndex(i) = 5 ': AgentControllerIndex(i) = 6
End Select
AgentOneSenseRightFaulty = False
'simulation parameters
BenchPartLimit = 9
RestockOnDump = False: RestockOnLoad = False
CycleCount = 200
StepCount = CycleCount * AgentCount
'genetic parameters
GenerationCount = 500
SolutionCount = 500 '1000
'output and formatting variables
OutputFileName1 = "OutFile1.txt"
Open OutputFileName1 For Output As #1
PoolOutputCurb = Int(SolutionCount / 100):
HistorySpacer = Space(CellCount - 3)
RuleStateLength = 2 * ConditionCount
RuleDashPos = RuleStateLength + 1
RuleLength = RuleStateLength + 4
HistoryLength = CellCount + 1
'fitness parameters
BenchScoreBase = 3: CarryScoreBase = 3
BenchScoreExpt = 3: CarryScoreExpt = 3
FitnessCoefficient = 2 * StepCount / (MaxDumpCount ^ CellCount)
DumpScoreCoeff = 0 'BenchScoreBase * CellCount
'reproduction variables
RouletteCrossPercentage = 0.99
OutCrossPercentage = 0
RandomCrossPercentage = 0
RandomChoicePercentage = 0
EliteChoicePercentage = 0.01
WorstChoicePercentage = 0
'reproduction variables
RouletteCrossCount = 2 * Int(RouletteCrossPercentage * SolutionCount / 2)
OutCrossCount = 2 * Int(OutCrossPercentage * SolutionCount / 2)
RandomCrossCount = 2 * Int(RandomCrossPercentage * SolutionCount / 2)
EliteChoiceCount = Int(EliteChoicePercentage * SolutionCount)
213
WorstChoiceCount = Int(WorstChoicePercentage * SolutionCount)
RandomChoiceCount = ToolMax(SolutionCount - EliteChoiceCount - WorstChoiceCount -
RandomCrossCount - RouletteCrossCount - OutCrossCount - RandomEliteChoiceCount -
RandomWorstChoiceCount, 0)
RouletteCrossParentFraction = 1: RouletteCrossParentMax = Int(RouletteCrossParentFraction *
SolutionCount)
OutCrossParentFraction = 0.25: OutCrossParentMax = Int(OutCrossParentFraction * SolutionCount)
RandomCrossParentFraction = 1: RandomCrossParentMax = Int(RandomCrossParentFraction *
SolutionCount)
RandomChoiceFraction = 1: RandomChoiceMax = Int(RandomChoiceFraction * SolutionCount)
ScoreRouletteMethod = False
RankRoulettePower = 0.5
ProbChromosomeCrossoverPoint = 0.5
ProbGeneCrossoverPoint = ProbChromosomeCrossoverPoint / (StateCount - 1)
MutationDamper = 12
InitConfigScore = 0
For i = 1 To CellCount
InitConfigScore = InitConfigScore + Mid(InitBenchString, i, 1) * BenchScoreBase ^ i
InitConfigScore = InitConfigScore + Mid(InitCarryString, i, 1) * CarryScoreBase ^ i
Next i
InitConfigScore = InitConfigScore + Right(InitBenchString, 1) ^ DumpScoreCoeff
MaxConfigScore = MaxDumpCount * (BenchScoreBase ^ CellCount + DumpScoreCoeff)
InitActionHistory = Left("ORIG ", CellCount)
InitBenchHistory = InitBenchString
InitAgentHistory = InitAgentString
InitCarryHistory = InitCarryString
'SeededController.Rules(1) = " ShBhLwRv-Mr ShBnLvRn-Gb SnBhLvRn-Ml SnBhLvRh-Tb
ShBnLvRv-Ml SnBhLwRv-Dn SnBhLwRn-Dn SnBhLvRv-Mr SnBnLvRn-Dn ShBhLwRn-Gb
SnBhLwRh-Tb ShBnLwRv-Mr ShBhLvRn-Ml ShBnLvRh-Gb SnBnLwRn-Dn SnBnLwRh-Dn
SnBnLwRv-Mr SnBnLvRv-Dn ShBnLwRn-Gb SnBnLvRh-Mr ShBhLvRv-Gb ShBhLwRh-Gb"
'SeededController.Rules(2) = " SnBhLhRn-Dn SnBhLvRh-Tb ShBhLhRv-Mr SnBhLnRh-Tb
ShBnLvRn-Gb SnBhLvRn-Ml SnBhLnRn-Dn SnBhLhRh-Mr ShBnLhRv-Mr SnBhLnRv-Tb ShBnLhRn-
Gb ShBhLvRv-Ml ShBnLvRv-Mr SnBnLvRn-Dn SnBnLnRv-Dn ShBnLnRv-Mr SnBnLnRn-Dn
SnBnLvRh-Ml SnBnLnRh-Dn SnBnLvRv-Mr ShBnLnRn-Gb SnBhLvRv-Dn SnBnLhRn-Dn SnBnLhRv-
Mr ShBnLhRh-Gb ShBnLvRh-Ml ShBhLhRh-Gb"
'SeededController.Rules(3) = " SnBhLnRv-Tb ShBnLhRv-Mr ShBnLhRw-Gb SnBhLnRw-Dn
SnBhLvRw-Ml ShBnLnRv-Mr ShBhLvRw-Gb ShBhLhRw-Gb SnBhLvRv-Dn ShBnLvRv-Dn
SnBnLnRv-Mr ShBhLnRw-Dn SnBhLhRv-Dn SnBnLvRv-Mr SnBhLhRw-Dn SnBnLhRv-Gb"
For i = 1 To ControllerCount
SeededController.Rules(i) = ""
Next i
End Sub
Public Sub PoolCreate()
'create a pool of solutions
Dim i As Integer
For i = 1 To SolutionCount
'NextSolutions(i).PreController = SeededController 'current method builds rules as simstates are
encountered
214
NextSolutions(i).CreatedController = SeededController
Next i
End Sub
Public Function EvaluateCondition(Term As String, Agent As Integer) As Boolean
'given the agent's current state in the system in the simulation, return the true or false value of the condition
Dim Cell As Integer
EvaluateCondition = False
Cell = InStr(AgentString, Agent) 'position of this agent in the row of cells
Select Case Term
Case "Sh": EvaluateCondition = Not (Mid(CarryString, Cell, 1) = "0")
Case "Bh": EvaluateCondition = Not (Mid(BenchString, Cell, 1) = "0")
Case "Sn": EvaluateCondition = (Mid(CarryString, Cell, 1) = "0")
Case "Bn": EvaluateCondition = (Mid(BenchString, Cell, 1) = "0")
Case "Ln":
If (Cell > 1) Then
If (Not (Mid(AgentString, Cell - 1, 1) = "0")) Then EvaluateCondition = (Mid(CarryString, Cell -
1, 1) = "0")
End If
Case "Lh":
If (Cell > 1) Then
If (Not (Mid(AgentString, Cell - 1, 1) = "0")) Then EvaluateCondition = Not (Mid(CarryString,
Cell - 1, 1) = "0")
End If
Case "Lv": If (Cell > 1) Then EvaluateCondition = (Mid(AgentString, Cell - 1, 1) = "0")
Case "Lw": EvaluateCondition = (Cell = 1)
Case "Rn":
If (Cell < CellCount) Then
If Not (Mid(AgentString, Cell + 1, 1) = "0") Then EvaluateCondition = (Mid(CarryString, Cell +
1, 1) = "0")
End If
Case "Rh":
If (Cell < CellCount) Then
If Not (Mid(AgentString, Cell + 1, 1) = "0") Then EvaluateCondition = Not (Mid(CarryString,
Cell + 1, 1) = "0")
End If
Case "Rv": If (Cell < CellCount) Then EvaluateCondition = (Mid(AgentString, Cell + 1, 1) = "0")
Case "Rw": EvaluateCondition = (Cell = CellCount)
End Select
End Function
Public Function ExecuteAction(Term As String, Agent As Integer, RealCondStr As String) As Boolean
'given the agent's current state in the system in the simulation, determine if the given action is legally
' executable and if so, then execute it and make the appropriate changes to the system
' note: some actions may not execute because they are flagged as disabled
Dim Cell As Integer
Dim BenchQuantity As Integer
ExecuteAction = False
If Action((1 + InStr(ActionString, Term)) / 2).Enabled Then
Cell = InStr(AgentString, Agent) 'agent's position in the row of cells
Select Case Term
Case "Dn": ExecuteAction = True 'Do-Nothing action always possible
Case "Ml":
If InStr(RealCondStr, "Lv") Then
215
''AgentString = Left(AgentString, Cell - 2) + StrReverse(Mid(AgentString, Cell - 1, 2)) +
Right(AgentString, CellCount - Cell)
AgentString = Left(AgentString, Cell - 2) + CStr(Agent) + "0" + Right(AgentString, CellCount
- Cell)
CarryString = Left(CarryString, Cell - 2) + StrReverse(Mid(CarryString, Cell - 1, 2)) +
Right(CarryString, CellCount - Cell)
ExecuteAction = True
End If
Case "Mr":
If InStr(RealCondStr, "Rv") Then 'Move right only if there is a void to the right
''AgentString = Left(AgentString, Cell - 1) + StrReverse(Mid(AgentString, Cell, 2)) +
Right(AgentString, CellCount - Cell - 1)
AgentString = Left(AgentString, Cell - 1) + "0" + CStr(Agent) + Right(AgentString, CellCount
- Cell - 1)
CarryString = Left(CarryString, Cell - 1) + StrReverse(Mid(CarryString, Cell, 2)) +
Right(CarryString, CellCount - Cell - 1)
ExecuteAction = True 'right move was executed and agent's location was updated
End If
Case "Tl":
If InStr(RealCondStr, "Sn") Then 'Take-from-the-Left action requires current agent to be
empty handed
If InStr(RealCondStr, "Lh") Then 'and also to have a carrying agent to the left of him
CarryString = Left(CarryString, Cell - 2) + "01" + Right(CarryString, CellCount - Cell)
ExecuteAction = True 'Take Left action was executed with carrying statuses updated
End If: End If
Case "Tr":
If InStr(RealCondStr, "Sn") Then
If InStr(RealCondStr, "Rh") Then
CarryString = Left(CarryString, Cell - 1) + "10" + Right(CarryString, CellCount - Cell - 1)
ExecuteAction = True
End If: End If
Case "Gl":
If InStr(RealCondStr, "Sh") Then
If InStr(RealCondStr, "Ln") Then
CarryString = Left(CarryString, Cell - 2) + "10" + Right(CarryString, CellCount - Cell)
ExecuteAction = True
End If: End If
Case "Gr":
If InStr(RealCondStr, "Sh") Then
If InStr(RealCondStr, "Rn") Then
CarryString = Left(CarryString, Cell - 1) + "01" + Right(CarryString, CellCount - Cell - 1)
ExecuteAction = True
End If: End If
Case "Tb":
If InStr(RealCondStr, "Sn") Then
If InStr(RealCondStr, "Bh") Then
'subtract from the parts on the bench unless it is a replenishing benchcell
If (Cell > 1) Or (Not RestockOnLoad) Then
BenchString = Left(BenchString, Cell - 1) + CStr(Mid(BenchString, Cell, 1) - 1) +
Right(BenchString, CellCount - Cell)
End If:
CarryString = Left(CarryString, Cell - 1) + "1" + Right(CarryString, CellCount - Cell)
ExecuteAction = True
End If: End If
Case "Gb":
If InStr(RealCondStr, "Sh") Then
216
BenchQuantity = Val(Mid(BenchString, Cell, 1))
''If BenchQuantity < BenchPartLimit Then
BenchString = Left(BenchString, Cell - 1) + CStr(BenchQuantity + 1) + Right(BenchString,
CellCount - Cell)
''End If
CarryString = Left(CarryString, Cell - 1) + "0" + Right(CarryString, CellCount - Cell)
'Restocks on dump: if agent gives a part to the benchdumpcell then new part added to
benchinitcell
If RestockOnDump Then
If (Cell = CellCount) Then
BenchQuantity = Val(Left(BenchString, 1))
If BenchQuantity < BenchPartLimit Then
BenchString = CStr(BenchQuantity + 1) + Right(BenchString, CellCount - 1)
End If: End If: End If
ExecuteAction = True
End If
End Select
End If
End Function
Public Sub PoolTest()
'must run entire solution pool through simulation runs as a means of obtaining fitness measure
Dim i As Integer
Dim j As Integer
Dim CycleIndex As Integer
Dim AgentIndex As Integer
Dim ControllerIndex As Integer
Dim RealCondString As String
Dim ImagCondString As String
Dim NowAction As String * 2
Dim RuleSet As String
Dim RuleLocation As Integer
Dim ConfigScore As Double
Dim LegalStepCount As Integer
Dim GoalStepCount As Integer
Dim DumpCount As Integer
Dim ActionHistory As String
Dim BenchHistory As String
Dim AgentHistory As String
Dim CarryHistory As String
Dim Controller As ControllerType
For SolutionIndex = 1 To SolutionCount
Controller = NextSolutions(SolutionIndex).CreatedController
ActionHistory = InitActionHistory
BenchHistory = InitBenchHistory
AgentHistory = InitAgentHistory
CarryHistory = InitCarryHistory
BenchString = InitBenchString
AgentString = InitAgentString
CarryString = InitCarryString
217
GoalStepCount = 0
LegalStepCount = StepCount
For CycleIndex = 1 To CycleCount
'run each simulation for a number of steps or until failure
For AgentIndex = 1 To AgentCount
'each agent takes consecutive actions during a sim Cycle
'determine the current state of this agent in the simulation by performing condition testing
ControllerIndex = AgentControllerIndex(AgentIndex)
RealCondString = "": ImagCondString = ""
For i = 1 To ConditionCount
j = 1
Do
If EvaluateCondition(Condition(i).ChoiceID(j), AgentIndex) Then Exit Do
j = j + 1
Loop Until j = Condition(i).ChoiceCount
''If AgentOneSenseRightFaulty And (AgentIndex = 1) And (i = 1) Then
'' ImagCondString = ImagCondString + "Sh" ' + Condition(i).ChoiceID(Int(Rnd() *
Condition(i).ChoiceCount) + 1)
''Else
'' ImagCondString = ImagCondString + Condition(i).ChoiceID(j)
''End If
RealCondString = RealCondString + Condition(i).ChoiceID(j)
Next i
'remove line below if above if-then ever reinstated
ImagCondString = RealCondString
With Controller
RuleSet = .Rules(ControllerIndex) 'current agent's controller
RuleLocation = InStr(RuleSet, ImagCondString) 'rule exists for conditions?
If RuleLocation <> 0 Then 'a rule prescribing an action for this condition set exists
NowAction = Mid(RuleSet, RuleLocation + RuleDashPos, 2) 'prescribed action
If Not ExecuteAction(NowAction, AgentIndex, RealCondString) Then
NowAction = "--"
LegalStepCount = LegalStepCount - 1
'remove failed action from controller
.Rules(ControllerIndex) = Left(RuleSet, RuleLocation - 1) + Right(RuleSet,
Len(RuleSet) - RuleLocation - RuleLength + 1)
End If
Else
'if no rule exists within this agent's controller then add a rule with a random action
NowAction = Action(Int(Rnd() * ActionCount) + 1).ID 'determine random action
If Not ExecuteAction(NowAction, AgentIndex, RealCondString) Then
NowAction = "--"
LegalStepCount = LegalStepCount - 1
'action caused a failure so do not add failed action
''RuleSet = RuleSet+ ImagCondString + "-" + NowAction + " " 'append new rule
Else
.Rules(ControllerIndex) = RuleSet + ImagCondString + "-" + NowAction + " " 'append
new rule
End If
End If
End With
ActionHistory = ActionHistory + " " + CStr(AgentIndex) + NowAction + HistorySpacer
BenchHistory = BenchHistory + " " + BenchString
AgentHistory = AgentHistory + " " + AgentString
CarryHistory = CarryHistory + " " + CarryString
218
DumpCount = Right(BenchString, 1)
If DumpCount = MaxDumpCount Then
GoalStepCount = GoalStepCount + 1
Else
GoalStepCount = 0
End If
Next AgentIndex
Next CycleIndex
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i
Next i
'ConfigScore = ConfigScore + DumpScoreCoeff * Right(BenchString, 1)
'ConfigScore = ConfigScore * FitnessCoefficient
CurrSolutions(SolutionIndex).TrainedController = Controller
With CurrSolutions(SolutionIndex).TrainedSimData
.ActionHistory = ActionHistory
.BenchHistory = BenchHistory
.AgentHistory = AgentHistory
.CarryHistory = CarryHistory
.ConfigScore = ConfigScore
.DumpCount = DumpCount
.GoalStepCount = GoalStepCount
.LegalStepCount = LegalStepCount
' .SimScore = ConfigScore / ToolMax((StepCount - GoalStepCount) + 1, (StepCount -
LegalStepCount) + 1)
' .SimScore = ConfigScore / ToolMax((StepCount - GoalStepCount) + 1, (StepCount -
LegalStepCount) + 1)
.SimScore = ConfigScore + GoalStepCount + LegalStepCount
End With
Next SolutionIndex
End Sub
Public Sub PoolSort()
'must sort the controller-tuple NowSolutions by their fitness measures; sorted with the best placed first in
list
Call ToolQuickSort(1, SolutionCount) 'SORTS HIGH TO LOW
End Sub
Public Sub PoolBreed()
'based on parameter settings make the appropriate selections and crosses to the current genepool of
solutions
'to create he next generation of solutions
Dim i As Integer
Dim TotalScore As Double
Dim Sum As Double
Dim PartialSums(3000) As Double
Dim Ran As Double
219
Dim Mother As Integer
Dim Father As Integer
Dim L As Integer
Dim R As Integer
Dim M As Integer
Dim PoolCount As Integer
If ScoreRouletteMethod Then
'a solution's chance of crossing is equal to its fitness divided by the sum of scores of all included
Solutions
TotalScore = 0
For i = 1 To RouletteCrossParentMax
TotalScore = TotalScore + CurrSolutions(i).TrainedSimData.SimScore
Next i
Sum = 0
For i = 1 To RouletteCrossParentMax - 1 'constructing partial-sum-probability index table for
selection
Sum = Sum + CurrSolutions(i).TrainedSimData.SimScore / TotalScore
PartialSums(i) = Sum
Next i
PartialSums(RouletteCrossParentMax) = 1
End If
Do
For i = 1 To EliteChoiceCount
Call GeneticsClone(i, "EliteChoice", i)
Next i
PoolCount = EliteChoiceCount
For i = 1 To RouletteCrossCount / 2
'perform roulette crosses to produce complementary pairs of solutions
If ScoreRouletteMethod Then
Ran = Rnd(): L = 1: R = RouletteCrossParentMax 'perform O(lg n) search for mother index
Do
M = Int((L + R) / 2): If Ran >= PartialSums(M) Then L = M + 1 Else R = M
Loop Until (R - L) <= 1 'mother index associated with random probability Ran has been found
Mother = Int((L + R) / 2)
Do
Ran = Rnd(): L = 1: R = RouletteCrossParentMax
Do
M = Int((L + R) / 2): If Ran >= PartialSums(M) Then L = M + 1 Else R = M
Loop Until (R - L) <= 1 'father index associated with random probability Ran has been found
Father = Int((L + R) / 2)
Loop Until Mother <> Father 'non-identical father index has been found as well
'cross these parents and store the complementary pairs in the nextsolution array in consecutive
positions
Call GeneticsCross(Mother, Father, "ScoreRouletteCross", PoolCount + 2 * i)
Else
'alternative roulette method based on linear rank of scores to remedy exponential fitness scores
'employs a function to return the selected parent index based on a random number
'an increase in 'RankRoulettePower' will increase the tendency to choose higher-ranked solutions
' if rrp=.5 then Prob of selecting the top scored solution is 0.20 with the Prob of selecting the worst
= 0
Mother = RouletteCrossParentMax - Int(RouletteCrossParentMax * (Rnd() ^ RankRoulettePower)
+ 0.5)
Do
220
Father = RouletteCrossParentMax - Int(RouletteCrossParentMax * (Rnd() ^ RankRoulettePower)
+ 0.5)
Loop Until Mother <> Father 'found non-identical parents
'cross these parents and store the complementary pairs in the nextsolution array in consecutive
positions
Call GeneticsCross(Mother, Father, "RankRouletteCross", PoolCount + 2 * i)
End If
Next i
PoolCount = PoolCount + RouletteCrossCount
For i = 1 To RandomChoiceCount
Call GeneticsClone(Int(Rnd() * SolutionCount) + 1, "RandomChoice", PoolCount + i)
Next i
PoolCount = PoolCount + RandomChoiceCount
If PoolCount = SolutionCount Then Exit Do
For i = 1 To WorstChoiceCount
Call GeneticsClone(SolutionCount - i + 1, "WorstChoice", PoolCount + i)
Next i
PoolCount = PoolCount + WorstChoiceCount
If PoolCount = SolutionCount Then Exit Do
For i = 1 To RandomCrossCount / 2
Mother = Int(Rnd() * RandomCrossParentMax) + 1
Do
Father = Int(Rnd() * RandomCrossParentMax) + 1
Loop Until Mother <> Father 'selected non-identical parents
Call GeneticsCross(Mother, Father, "RandomCross", PoolCount + 2 * i)
Next i
PoolCount = PoolCount + RandomCrossCount
If PoolCount = SolutionCount Then Exit Do
For i = 1 To OutCrossCount / 2
Mother = Int(Rnd() * OutCrossParentMax) + 1 'mother selected from top of pool
Father = SolutionCount - Int(Rnd() * OutCrossParentMax) 'father selected from bottom of pool
Call GeneticsCross(Mother, Father, "OutCross", PoolCount + 2 * i)
Next i
''PoolCount = PoolCount + OutCrossCount
Loop Until False
'at this point, new pool is complete
End Sub
Public Sub zzzSolutionPoolPrint()
'print out some of the best of the current generation's controller-tuple NowSolutions
Dim i As Integer
Dim j As Integer
Dim k As Integer
Dim L As Integer
k = 1
''PrintFreq = 1000
''For I = 1 To SolutionCount-2
''If ((i Mod PrintFreq) = 0) Or (i = 1) Or (i = SolutionCount - 2) Then
''For L = 2 To 1 step -1
221
'' Debug.Print "---------------------------------------------------------------------------------------------------"
'' k = SolutionCount - i + L + 1
With CurrSolutions(k).TrainedSimData
Debug.Print "G:"; GenerationIndex; " S:"; k; " SCORE="; .SimScore; " LEGALCOUNT=";
.LegalStepCount; " DUMPCOUNT= "; .DumpCount; " POSTGOALSTEPS= "; .GoalStepCount
' For j = 1 To ControllerCount
' Debug.Print " OLD S: $ --- "; " #"; Len(.PreController.Rules(j)) / RuleLength; " [";
.PreController.Rules(j); "] "
' Next j
Debug.Print " TRACE: " + .ActionHistory
Debug.Print " BENCH: " + .BenchHistory
Debug.Print " AGENT: " + .AgentHistory
Debug.Print " CARRY: " + .CarryHistory
' For j = 1 To ControllerCount
' Debug.Print " NEW S:"; k; " $"; .SimScore; " #"; Len(.Controller.Rules(j)) / RuleLength; "[";
.Controller.Rules(j); "] "
' Next j
' Debug.Print "G:"; GenerationIndex; " S:"; k; " SCORE="; .SimScore; " LEGALCOUNT=";
.LegalStepCount; " DUMPCOUNT= "; .DumpCount; " POSTGOALSTEPS= "; .GoalStepCount
End With
'' Next L
'' End If
'' Next i
End Sub
Public Sub GeneticsClone(Parent As Integer, BirthType As String, Child As Integer)
NextSolutions(Child).CreatedController = CurrSolutions(Parent).TrainedController
If False Then
PrintIt = PrintAll And ((Child1 Mod 200 = 0) Or (Child1 < 4) Or (Child1 = SolutionCount))
If PrintIt Then
Debug.Print "-----------------------------------------------------------------------------------------------------------
"
Debug.Print "G:"; GenerationIndex; " S:"; Child; BirthType; " P:"; Parent
For j = 1 To ControllerCount
Debug.Print "G:"; GenerationIndex; " P:"; Parent; " $";
CurrSolutions(Parent).TrainedSimData.SimScore; Len(CurrSolutions(Parent).TrainedController.Rules(j)) /
RuleLength; "["; CurrSolutions(Parent).TrainedController.Rules(j); "] "
Next j
For j = 1 To ControllerCount
Debug.Print "G:"; GenerationIndex; " CR:"; Child; "$";
CurrSolutions(Parent).TrainedSimData.SimScore; Len(CurrSolutions(Parent).TrainedController.Rules(j)) /
RuleLength; "["; CurrSolutions(Parent).TrainedController.Rules(j); "] "
Next j
End If
End If
''Call GeneticsMutate(Child)
End Sub
Public Sub GeneticsCross(Mom As Integer, Dad As Integer, BirthType As String, Child2 As Integer)
'NextSolution # Index and #Index+1 will be the complementary pairs produced
' as a result of the recombination of two controller sets indexed by Mom and Dad
Dim i As Integer
222
Dim j As Integer
Dim MomString As String
Dim DadString As String
Dim MomRuleCount As Integer
Dim DadRuleCount As Integer
Dim ChildString1 As String
Dim ChildString2 As String
Dim TempChildString As String
Dim MomRule As String
Dim DadRule As String
Dim RulePosition As Integer
Dim Child1 As Integer
Child1 = Child2 - 1
For j = 1 To ControllerCount
'Gene Crossover: within chromosomes
MomString = CurrSolutions(Mom).TrainedController.Rules(j)
DadString = CurrSolutions(Dad).TrainedController.Rules(j)
MomRuleCount = Len(MomString) / RuleLength
DadRuleCount = Len(DadString) / RuleLength
ChildString1 = "": ChildString2 = ""
For i = 0 To MomRuleCount - 1 'first process all visible rules in common or specific to mom
If Rnd() < ProbGeneCrossoverPoint Then 'determine if gene crossover point exists
TempChildString = ChildString1 'achieve crossover by swapping build strings
ChildString1 = ChildString2
ChildString2 = TempChildString
End If
MomRule = Mid(MomString, i * RuleLength + 1, RuleLength) 'find next rule in controller
RulePosition = InStr(DadString, Left(MomRule, RuleStateLength)) 'see if rule exists in other parent
controller
If RulePosition = 0 Then 'unfound rule only in mom so add to a buildstring and have other get
random hidden rule
ChildString1 = ChildString1 + MomRule
'ChildString2 = ChildString2 + Left(MomRule, RuleStateLength) + "-" + Action(Int(Rnd() *
ActionCount) + 1).ID + " "
Else 'common rule found in dad too so append mom and dad rules to the two
buildstrings
DadRule = Mid(DadString, RulePosition, RuleLength)
ChildString1 = ChildString1 + DadRule
ChildString2 = ChildString2 + MomRule
End If
Next i
For i = 0 To DadRuleCount - 1 'now process all rules specific to dad controller (commons
processed by prior loop)
DadRule = Mid(DadString, i * RuleLength + 1, RuleLength) 'next rule in controller
RulePosition = InStr(MomString, Left(DadRule, RuleStateLength))
If RulePosition = 0 Then 'unfound rule only in dad so add to a buildstring and have other get
random hidden rule
If Rnd() < ProbGeneCrossoverPoint Then 'determine if gene crossover point exists
TempChildString = ChildString1 'achieve crossover by swapping build strings
ChildString1 = ChildString2
ChildString2 = TempChildString
End If
ChildString1 = ChildString1 + DadRule 'append to the buildstring after possible crossover
'ChildString2 = ChildString2 + Left(DadRule, RuleStateLength) + "-" + Action(Int(Rnd() *
ActionCount) + 1).ID + " "
223
End If
Next i
'Chromosome crossover: between chromosomes
If Rnd() < ProbChromosomeCrossoverPoint Then 'now that gene crossovers are implemented,
perform independent chromosome segregration
TempChildString = ChildString1
ChildString1 = ChildString2
ChildString2 = TempChildString
End If
NextSolutions(Child1).CreatedController.Rules(j) = ChildString1 'store new chromosome as
controller in nextsolutions
NextSolutions(Child2).CreatedController.Rules(j) = ChildString2 'complementary pair being
produced
Next j
For i = Child1 To Child2
If False Then
PrintIt = PrintAll And ((Child1 Mod 200 = 0) Or (Child1 < 4) Or (Child1 = SolutionCount))
If PrintIt Then
Debug.Print "--------------------------------------------------------------------------------------------------------
---"
Debug.Print "G:"; GenerationIndex; " S:"; i; BirthType; " M:"; Mom; " D:"; Dad
For j = 1 To ControllerCount
Debug.Print "G:"; GenerationIndex; " M:"; Mom; " $";
CurrSolutions(Mom).TrainedSimData.SimScore; Len(CurrSolutions(Mom).TrainedController.Rules(j)) /
RuleLength; "["; CurrSolutions(Mom).TrainedController.Rules(j); "] "
Next j
For j = 1 To ControllerCount
Debug.Print "G:"; GenerationIndex; " D:"; Dad; " $";
CurrSolutions(Dad).TrainedSimData.SimScore; Len(CurrSolutions(Dad).TrainedController.Rules(j)) /
RuleLength; "["; CurrSolutions(Dad).TrainedController.Rules(j); "] "
Next j
For j = 1 To ControllerCount
Debug.Print "G:"; GenerationIndex; " CR:"; i; " $ --- ";
Len(NextSolutions(i).CreatedController.Rules(j)) / RuleLength; "[";
NextSolutions(i).CreatedController.Rules(j); "] "
Next j
End If
End If
Call GeneticsMutate(i)
Next i
End Sub
Public Sub GeneticsMutate(Index As Integer)
'the current set of controllers for nextsolution # index must be mutated
Dim i As Integer
Dim j As Integer
Dim ControllerLength As Integer
Dim RuleSet As String
Dim NumRules As Integer
Dim NumMutations As Integer
''Dim RandomRule As Integer
Dim RandomRuleActionPos As Integer
224
Dim RandomAction As String * 2
For i = 1 To ControllerCount
RuleSet = NextSolutions(Index).CreatedController.Rules(i)
ControllerLength = Len(RuleSet)
If ControllerLength > 0 Then
NumRules = ControllerLength / RuleLength
NumMutations = Int(0.5 + StateCount * (Rnd() ^ MutationDamper)) 'number of mutations to occur
NumMutations = Int(NumRules * NumMutations / StateCount + 0.5) 'probabilistic reduction given
rule count
For j = 1 To NumMutations
RandomRule = Int(NumRules * Rnd()) + 1 ' in conjunction with probabilisitic
''RandomRule = Int(StateCount * Rnd()) + 1 'which rule (hidden or visible) to mutate
''If RandomRule <= NumRules Then 'mutate only visible rules
RandomAction = Action(Int(Rnd() * ActionCount) + 1).ID
RandomRuleActionPos = RandomRule * RuleLength - 3
NextSolutions(Index).CreatedController.Rules(i) = Left(RuleSet, RandomRuleActionPos) +
RandomAction + Right(RuleSet, ControllerLength - RandomRuleActionPos - 2)
''End If
Next j
End If
Next i
If False Then
PrintIt = PrintAll And ((Child1 Mod 200 = 0) Or (Child1 < 4) Or (Child1 = SolutionCount))
If PrintIt Then
For j = 1 To ControllerCount
Debug.Print "G:"; GenerationIndex; " MU:"; Index; " $ ---";
Len(NextSolutions(Index).CreatedController.Rules(j)) / RuleLength; "[";
NextSolutions(Index).CreatedController.Rules(j); "] "
Next j
End If
End If
End Sub
Public Function ToolMax(Argument1 As Integer, Argument2 As Integer) As Integer
If Argument1 < Argument2 Then ToolMax = Argument2 Else ToolMax = Argument1
End Function
Public Function ToolMin(Argument1 As Integer, Argument2 As Integer) As Integer
If Argument1 > Argument2 Then ToolMin = Argument2 Else ToolMin = Argument1
End Function
Public Sub ToolQuickSort(Left As Integer, Right As Integer)
Dim PivotPoint As Integer
Dim PivotElement As SolutionType
Dim LHold As Integer
Dim RHold As Integer
LHold = Left: RHold = Right
PivotElement = CurrSolutions(Left)
While Left < Right
While (CurrSolutions(Right).TrainedSimData.SimScore <= PivotElement.TrainedSimData.SimScore)
And (Left < Right)
Right = Right - 1
Wend
If Left <> Right Then
CurrSolutions(Left) = CurrSolutions(Right)
Left = Left + 1
225
End If
While (CurrSolutions(Left).TrainedSimData.SimScore >= PivotElement.TrainedSimData.SimScore)
And (Left < Right)
Left = Left + 1
Wend
If Left <> Right Then
CurrSolutions(Right) = CurrSolutions(Left)
Right = Right - 1
End If
Wend
CurrSolutions(Left) = PivotElement
PivotPoint = Left
Left = LHold: Right = RHold
If Left < PivotPoint Then Call ToolQuickSort(Left, PivotPoint - 1)
If Right > PivotPoint Then Call ToolQuickSort(PivotPoint + 1, Right)
End Sub
Public Sub zzzImplementSolution(Index As Integer, Train As Boolean)
'must run entire solution pool through simulation runs as a means of obtaining fitness measure
Dim i As Integer
Dim j As Integer
Dim CycleIndex As Integer
Dim AgentIndex As Integer
Dim ControllerIndex As Integer
Dim RealCondString As String
Dim ImagCondString As String
Dim NowAction As String * 2
Dim RuleSet As String
Dim RuleLocation As Integer
Dim ConfigScore As Integer
Dim LegalStepCount As Integer
Dim GoalStepCount As Integer
Dim DumpCount As Integer
Dim ActionHistory As String
Dim BenchHistory As String
Dim AgentHistory As String
Dim CarryHistory As String
Dim AbortSim As Boolean
Dim Controller As ControllerType
If Train Then
Controller = NextSolutions(Index).CreatedController
Else
Controller = CurrSolutions(Index).TrainedController
End If
AbortSim = False
ActionHistory = InitActionHistory
BenchHistory = InitBenchHistory
AgentHistory = InitAgentHistory
CarryHistory = InitCarryHistory
BenchString = InitBenchString
AgentString = InitAgentString
226
CarryString = InitCarryString
GoalStepCount = 0
LegalStepCount = StepCount
For CycleIndex = 1 To CycleCount
'run each simulation for a number of steps or until failure
For AgentIndex = 1 To AgentCount
'each agent takes consecutive actions during a sim Cycle
'determine the current state of this agent in the simulation by performing condition testing
ControllerIndex = AgentControllerIndex(AgentIndex)
RealCondString = "": ImagCondString = ""
For i = 1 To ConditionCount
j = 1
Do
If EvaluateCondition(Condition(i).ChoiceID(j), AgentIndex) Then Exit Do
j = j + 1
Loop Until j = Condition(i).ChoiceCount
''If AgentOneSenseRightFaulty And (AgentIndex = 1) And (i = 1) Then
'' ImagCondString = ImagCondString + "Sh" ' + Condition(i).ChoiceID(Int(Rnd() *
Condition(i).ChoiceCount) + 1)
''Else
'' ImagCondString = ImagCondString + Condition(i).ChoiceID(j)
''End If
RealCondString = RealCondString + Condition(i).ChoiceID(j)
Next i
'remove line below if above if-then ever reinstated
ImagCondString = RealCondString
With Controller
RuleSet = .Rules(ControllerIndex) 'current agent's controller
RuleLocation = InStr(RuleSet, ImagCondString) 'rule exists for conditions?
If RuleLocation <> 0 Then 'a rule prescribing an action for this condition set exists
NowAction = Mid(RuleSet, RuleLocation + RuleDashPos, 2) 'prescribed action
If Not ExecuteAction(NowAction, AgentIndex, RealCondString) Then
NowAction = "--"
LegalStepCount = LegalStepCount - 1
' GoalStepCount = 0
If Train Then
'remove failed action from controller
.Rules(ControllerIndex) = Left(RuleSet, RuleLocation - 1) + Right(RuleSet,
Len(RuleSet) - RuleLocation - RuleLength + 1)
Else
AbortSim = True
End If
End If
Else
If Train Then
'if no rule exists within this agent's controller then add a rule with a random action
NowAction = Action(Int(Rnd() * ActionCount) + 1).ID 'determine random action
If Not ExecuteAction(NowAction, AgentIndex, RealCondString) Then
NowAction = "--"
LegalStepCount = LegalStepCount - 1
' GoalStepCount = 0
'action caused a failure so do not add failed action
''RuleSet = RuleSet+ ImagCondString + "-" + NowAction + " " 'append new rule
Else
227
.Rules(ControllerIndex) = RuleSet + ImagCondString + "-" + NowAction + " " 'append
new rule
End If
Else
NowAction = "--"
LegalStepCount = LegalStepCount - 1
'GoalStepCount = 0
AbortSim = True
End If
End If
End With
ActionHistory = ActionHistory + " " + CStr(AgentIndex) + ":" + NowAction + HistorySpacer
BenchHistory = BenchHistory + " " + BenchString
AgentHistory = AgentHistory + " " + AgentString
CarryHistory = CarryHistory + " " + CarryString
DumpCount = Right(BenchString, 1)
If DumpCount = MaxDumpCount Then
GoalStepCount = GoalStepCount + 1
Else
GoalStepCount = 0
End If
If AbortSim Then Exit For
Next AgentIndex
If AbortSim Then Exit For
Next CycleIndex
ConfigScore = 0
For i = 1 To CellCount
ConfigScore = ConfigScore + Mid(BenchString, i, 1) * BenchScoreBase ^ i '^ I
ConfigScore = ConfigScore + Mid(CarryString, i, 1) * CarryScoreBase ^ i '^ I
Next i
ConfigScore = ConfigScore + DumpScoreCoeff * Right(BenchString, 1)
If Train Then
CurrSolutions(Index).TrainedController = Controller
With CurrSolutions(Index).TrainedSimData
.ActionHistory = ActionHistory
.BenchHistory = BenchHistory
.AgentHistory = AgentHistory
.CarryHistory = CarryHistory
.ConfigScore = ConfigScore
.DumpCount = DumpCount
.GoalStepCount = GoalStepCount
.LegalStepCount = LegalStepCount
' .SimScore = ConfigScore / ToolMax((StepCount - GoalStepCount) + 1, (StepCount -
LegalStepCount) + 1)
'' .SimScore = ConfigScore / ToolMax((StepCount - GoalStepCount) + 1, (StepCount -
LegalStepCount) + 1)
.SimScore = ConfigScore + GoalStepCount + LegalStepCount
End With
Else
With CurrSolutions(Index).TrainedSimData
.ActionHistory = ActionHistory
.BenchHistory = BenchHistory
.AgentHistory = AgentHistory
228
.CarryHistory = CarryHistory
.ConfigScore = ConfigScore
.DumpCount = DumpCount
.GoalStepCount = GoalStepCount
.LegalStepCount = LegalStepCount
'.SimScore = LegalStepCount + GoalStepCount + Int(StepCount * (DumpCount /
MaxDumpCount)) ';ConfigScore +
End With
End If
End Sub
Public Sub zzzTrainPool()
Dim i As Integer
For i = 1 To SolutionCount
'Call ImplementSolution(i)
Next i
End Sub
Public Sub zzzTestPool()
Dim i As Integer
For i = 1 To SolutionCount
'Call ImplementSolution(i)
Next i
End Sub
229
REFERENCES
Adams, J.A. and Paul, R. (1994) Human-Managed, Hierarchical Control of Multiple
Mobile Agents. Proceedings of the 33rd Conference on Decision and Control. Lake
Buena Vista, FL. 12:3524-3529.
Agah, A. and Bekey, G.A. (1994) Autonomous mobile Robot teams. In Proceedings of
the AIAA NASA Conference on Intelligent Robots in Field, Factory, Service and Space.
Houston Texas. 1:246-251.
Agah, A. and Bekey, G.A. (1994) Exploration and discovery during task performance by
simulated autonomous intelligent robots. In Proceedings of the Fifth International
Conference on Robots and Manufacturing. Maui, Hawaii. 5:1-4.
Agah, A. and Bekey, G.A. (1996) Robot Colony Simulator: A Research Tool for
Studying Colonies of Intelligent Robots. Simulation. 67:82-93.
Anderson, T.L. and Donath, M. (1990) Animal Behavior as a Paradigm for Developing
Robot Autonomy. 6:145-168.
Andre, D. (1995) The Automatic Programming of Agents that Learn Mental Models and
Create Simple Plans of Action. In Proceedings of the Fourteenth International Joint
Conference on Artificial Intelligence. C.S. Mellish (Ed.). Morgan Kaufmann. 741-747.
Angle, C.M. and Brooks, R.A. (1990) Small Planetary Rovers. IEE/RSJ International
Workshop on Intelligent Robots and Systems. Ikabara, Japan. 383-388.
Arkin, R.C. (1989) Homeostatic Control for a Mobile Robot: Dynamic Replanning in
Hazardous Environments. SPIE Proceedings 1007, Mobile AGVs III, William J. Wolfe
(Ed.). 407-413.
Arkin, R.C. (1992) Cooperation without communication: multi-agent schema-based robot
navigation. Journal of Robotic Systems. 9:351-364.
Assad, A.M. and Packard, N.H. (1990) Emergent colonization in an artificial ecology. In
Varela, F.J. and Bourgine, P. (Eds.) Toward a Practice of Autonomous Systems. MIT
Press. Cambridge, Massachusetts. 143-152.
Axelrod, R. (1997) Advancing the art of simulation in the social sciences. Working Paper
97-05-048. Santa Fe Institute.
Balch, T. (1997) Learning Rules: Behavioral diversity in Robot teams. In Collected
Papers from the AAAI-97 Workshop on Multi-Agent Learning. AAAI. 7-12.
230
Beasley, D., Bull, D.R. and Martin, R.R. (1993) An Overview of Genetic Algorithms.
Available on WWW. University Computing. Cardiff University. 15(2):58-69.
Beer, R.D., Chiel, H.J. and Sterling L.S. (1990) A Biological Perspective on Autonomous
Agent Design. Robotics and Autonomous Systems. 6:169-186.
Bellman, R. (1961) Adaptive Control Processes – A Guided Tour. Princeton: Princeton
University Press.
Berlanga, A., Sanchis, A., Isasi, P. and Molina, J.M. (2000) A General Learning Co-
Evolution Method to Generalize Autonomous Robot Navigation Behavior. In
Proceedings of the 2000 Congress on Evolutionary Computation. 16-19.
Brooks, R.A. (1986) A Robust Layered Control System for a Mobile Robot. IEEE
Journal of AGVs Automation. March.
Brooks, R.A. (1991) Intelligence without Representation. Artificial Intelligence. 47:139-
160.
Brooks, R.A. and Flynn, Anita. (1989) Rover on a Chip. Aerospace America. 10:22-27.
Curtin, J., Bellingham, G., Catiporic, J. and Webb, D. (1993) Autonomous
Oceanographic Sampling Networks. Oceanography. 6(3):86-94.
Dorigo, M. and Bersini, H. (1994) A Comparison of Q-Learning and Classifier Systems.
In Proceedings of From Animals to Animats, Third International Conference on
Simulation of Adaptive Behavior.
Douglas, S. A. (2013) Autonomous Agents in Complex Event Driven Software Systems.
Engineering of Computer Based Systems (ECB). 1.
Finin, T., Fritzson, R., McKay, D., McEntire, R. (1994) KQML As An Agent
Communication Language. Proceedings of the Third International Conference on
Information and Knowledge Management. 456-463.
Fischer, K., Muller, J.P. and Pischel, M. (1996) Cooperative Transportation Scheduling:
An Application Domain for DAI. Applied Artificial Intelligence. 10:1-33.
Fogel, D.B. (1995) Evolutionary Computation: Toward a New Philosophy of Machine
Intelligence. IEEE Pres. Piscataway, NJ.
Genovese, V., Dario, P., Magni, R., and Odetti, L. (1992) Self-organizing behavior and
swarm intelligence in a pack of mobile miniature Robots in search of pollutants.
Proceedings of the IEEE/RSJ International Conference on Intelligent AGVs and Systems.
1575-1582.
231
Goldberg, D.E. (1989) Genetic Algorithms in Search, Optimization and Machine
Learning. Addison-Wesley.
Gowda, C.C. and Mayya, S.G. (2014) Comparison of Back Propagation Neural network
and Genetic Algorithm Neural Network for Stream Flow Prediction. Journal of
Computational Environmental Sciences 2014. Article ID 290127. 6 pages.
Gregorin, L., Freire, E., Caravalo, E. and Molina, L. (2016) Evolutionary Robotics
Applied to the Multi-Robot Worst-Case Pursuit-Evasion Problem. In Proceedings of
2016 IEEE 7th Annual Ubiquitous Computing, Electronics & Mobile Communication
Conference (UEMCON). 20-22 Oct. 2016. 1-7.
Griffiths, A.J., Gelbart, W.M., Miller, J.H. and Lewontin, R.C. (1999) Modern Genetics
Analysis. W.H. Freeman and Co. 549.
Habib, M.K., Asama, H., Endo, I., Matsumoto, A. and Ishida, Y. (1992) Simulation
Environment for an Autonomous Decentralized Multi-Agent Robotic System.
Proceedings of the IEEE/RSJ International Conference on Int. Robotics and Systems.
Raleigh, NC. 1550-1557.
Holland, J.H. (1975) Adaptation in natural and artificial systems. Ann Arbor: The
University of Michigan Press.
Iba, H. (1996) Emergent Cooperation for Multiple Agents using Genetic Programming.
Late Breaking Papers of the Genetic Programming 1996 Conference. J.R. Koza (Ed.).
Stanford, CA. 66-74.
Iwanaga, S., Ohkura, K. and Matsuda, T. (2009) Evolving Adaptive Group Behavior in a
Multi-Robot System. ICROS-SICE International Joint Conference 2009. August 18-21,
2009, Fukuoka International Congress Center, Japan. 2009: 2235-2240.
Jennings, N.R. (1993) Commitments and conventions: the foundations of coordination in
multi-agent systems. Knowledge Engineering Review. 8(3):223-250.
Jennings, N.R. (1996) Coordination Techniques for Distributed Artificial Intelligence. In
G.M.P. O’Hare and N.R. Jennings (Eds.). Foundations of Distributed Intelligence. John
Wiley & Sons, Inc. New York. 187-210.
Juang, C., Lai, M. and Zeng, W. (2014) Evolutionary Fuzzy Control and Navigation for
Two Wheeled Robots Cooperatively Carrying an Object in Unknown Environments.
IEEE Transactions on Cybernetics, Vol. 45, No. 9, September 2015. 1731-1743.
Jun, H. and Sim, K. (1997) Behavior Learning and Evolution of Collective Autonomous
Mobile Robots based on Reinforcement Learning and Distributed Genetic Algorithms.
IEEE International Workshop on Robot and Human Communication. 1997: 248-253.
232
Kaelbing, L.P., Littman, M. and Moore, A. (1996) Reinforcement Learning: A Survey.
Journal of AI Research. 4:237-285.
Karakatic, S., Podgorelec, V., Barros, R. and Basgalupp, M. (2015) Evolving Balanced
Decision Trees with a Multi-Population Genetic Algorithm. IEEC Congress on
Evolutionary Computation. 2015: 54-61.
Kenzin, M. Yu., Bychkov, I.V. and Maksimkin, N.N. (2017) An Evolutionary approach
to route the heterogeneous groups of underwater robots. MIPRO. 2017:1116-1119.
Lane, D.M. and McFadzean, A.G. (1994) Distributed Problem Solving and Real-Time
Mechanisms in Robot Architectures. Engineering Applications of Artificial Intelligence.
7(2). 105-117.
Liu, X. and Louis, S. (1996) Combining genetic algorithms and case-based reasoning for
structure design. In M.E. Cohen and D.L. Hudson (Eds.), Proceedings of the ICSA
Eleventh International Conference on Computers and their applications. ISCA. 103-106.
Luke, S. and Spector, L. (1996) Evolving Teamwork and Coordination with Genetic
Programming. In Proceedings of the First Annual Conference on Genetic Programming.
J.R. Koza et al (Eds.). MIT Press. Cambridge, MA. 150-156.
Maes, P. (1989) The Dynamics of Action Selection. Proceedings of Int. Joint Conference
of Artificial Intelligence (IJCAI-89). 991-997.
Maes, P. (Ed.) (1990) Designing Autonomous Agents: Theory and Practice from Biology
to Engineering and Back. MIT Press. Cambridge, MA.
Mahadevan, S. and Connell, J. (1990) Automatic Programming of Behavior-based
Robots using Reinforcement Learning. IBM T.J. Watson Research Report.
Maskell, B. and Wilby, M. (1996) Evolving Software Agent Behaviours. IEEE Global
Telecommunications Conference. 1: 90-94.
McElroy, J.F., Stephens, L.M., Bonnell, R.D. and Gorman, J. (1989) Communication and
Cooperation in a Distributed Automated Guided Vehicle System. IEEE Conference
Proceedings of Southeastcon. 999-1003.
Michalewicz, Z. (1992) Genetic Algorithms + Data Structures = Evolutionary Programs.
Springer-Verlag.
Miller, D.P. (1990) Multiple Behavior-Controlled Micro-AGVs for Planetary Surface
Missions. Proceedings of the IEEE International Conference on Systems, Man and
Cybernetics. Los Angeles, CA. 11:281-292.
233
Minar, N., Burkhart, R., Langton, C. and Askenazi, M. (1996) The Swarm simulation
system: A toolkit for building multi-agent simulations. Working Paper 96-06-042. Santa
Fe Institute.
Morik, K., Kaiser, M. and Klingspor, V. (Eds.). (1999) Making Robots Smarter:
Combining Sensing and Action through Robot Learning. Kluwer Academic Publishers.
Norwell. MA.
Muhlenbein, H. (1989) Parallel Genetic Algorithms, Population Genetics and
Combinatorial Optimization. Proc. Third Int’l Conf. On Genetic Algorithms. Morgan
Kaufmann. 416-421.
Murray, A. and Louis, S. (1995) Design Strategies for evolutionary robots. In E.A.
Yfantis (Ed.), Proceedings of the Third Golden West International Conference on
Intelligent Systems. Kluwer Academic Press. 609-616.
Narendra, K. and Thathachar, M.A.L. (1989) Learning Automaton: An Introduction.
Prentice Hall, Inc. Englewood Cliffs, NJ.
Nolfi, S. and Parisi, D. (1995) Learning to adapt to changing environments in evolving
neural networks. Technical Report 95-15. Institute of Psychology. National Research
Council. Rome, Italy.
Nowak, M. A. (2006) Five Rules for the Evolution of Cooperation. Science 314 (5805).
1560-1563.
O'Dowd, P.J., Winfield, Alan F.T. and Studley, M. (2011) The Distributed Co-Evolution
of an Embodied Simulator and Controller for Swarm Robot Behaviours. IEEE/RSJ
International Conference on Intelligent Robots and Systems. September 25-30, 2011. San
Francisco, CA, USA. 4995-5000.
Paic-Antunovic, L. and Jakobovic, D. (2012) Evolution of Automatic Robot Control with
Genetic Programming. MIPRO 2012. May 21-25, 2012. 817-822.
Palacios-Leyva, R. E., Cruz-Alvarez, R., Montes-Gonzalez, F., Rascon-Perez, L. and
Santos, J. (2013) Combination of Reinforcement Learning with Evolution for
Automatically Obtaining Robot Neural Controllers. In Proceedings of IEEE Congress on
Evolutionary Computation. June 20-23, Cancun, Mexico. 119-126.
Parker, L.E. (1996) On the design of behavior-based multi-robot teams. Advanced
Robotics. 10(6):547-578.
Pigliucci, M. (2008) Adaptive Landscapes, Phenotypic Space, and the Power of
Metaphors, The Quarterly Review of Biology. 83 (3): 283-287.
234
Quinn, M. (2000) Evolving Co-operative Homogeneous Multi-Robot Teams. In
Proceedings of the 2000 IEEE/RSJ International Conference on Intelligent Robots and
Systems. 2000: 1798-1803.
Qureshi, A. (1996) Evolving Agents. In Proceedings of the First Annual Conference on
Genetic Programming. J.R. Koza et al. (Eds.). MIT Press. Cambridge, MA. 369-374.
Ramsey, C. and Grefensttete, J. (1993) Case-based initialization of genetic algorithms. In
Proceedings of the Fifth International Conference on Genetic Algorithms. S. Forrest
(Ed.). Morgan Kaufmann. 84-91.
Ruini, F. and Cangelosi, A. (2008) Distributed Control in Multi-Agent Systems: A
Preliminary Model of Autonomous MAV Swarms. IEEE Xplore
Conference: 11th International Conference on Information Fusion. 2008. 1043-1050."
Russell, S. and Norvig, P. (1995) Artificial Intelligence: A Modern Approach. Prentice-
Hall.
Salomon, R. (1997) The evolution of different neuronal control structures for
autonomous agents. Robotics and Autonomous Systems. 22:199-213.
Schoppers, M.J. (1987) Universal plans for reactive robots in unpredictable
environments. In Proceedings of the Tenth International Joint Conference on Artificial
Intelligence (IJCAI-87). Milan, Italy. 1039-1046.
Shibata, T. and Fukuda, T. (1993) Coordinative Behavior by Genetic Algorithm and
Fuzzy in Evolutionary Multi-Agent System. In Proceedings of International Conference
on Robotics and Automation, 1993. 760-763.
Simoes, E.D.V. and Dimond, K.R. (1999) An Evolutionary Controller for Autonomous
Multi-Robot Systems. In Proceedings of IEEE International Conference on Systems,
Man, and Cybernetics, 1999. VI-596-VI601.
Smith, J.M. (1964) Group Selection and Kin Selection. Nature 201. 14 March, 1964.
1145-1157.
Stillwell, D.J. and Bay, J.S. (1993) Toward the Development of a Material Transport
System using Swarms of Ant-Like Robots. IEEE Conference on Robotics and
Automation. Atlanta, GA. 766-771.
Suparerk, P. and Yuta, S. (1990) Considerations on the cooperation of multiple
autonomous mobile robots. In IEEE International Workshop on Intelligent Robots and
Systems.
Tecuci, G. (1998) Building Intelligent Agents. Academic Press. San Diego, CA.
235
Traulsen, A. and Nowak, M (2006) Evolution of Cooperation by Multilevel Selection.
PNAS. 103(29). 10952-5.
Uchibe, E. and Asada, M. (2006) Incremental Coevolution with Competitive and
Cooperative Tasks in a Multirobot Environment. In Proceedings of the IEEE. Vol. 94,
No. 7, July 2006. 1412-1424.
Watkins, C.J.C.H (1989) Learning from Delayed Records. Ph. D. Thesis. King’s College.
Cambridge University.
Weise, T., Chiong, R. and Tang, K. (2011) Evolutionary Optimization: Pitfalls and
Booby Traps. Journal of Computer Science and Technology. 27(5). 907-936.
Weiss, G. (1999) Multi-Agent Systems: A Modern Approach to Distributed Artificial
Intelligence. MIT Press. Cambridge, MA.
Wilson, D.S. and Sober, E. (1994) Reintroducing Group Selection to the Behavioral
Sciences. Behavioral and Brain Sciences 19 (4): 585-654.
Wilson, D.S. and Wilson, E.D. (2007) Rethinking the Theoretical Foundations of
Sociobiology. Quarterly Review of Biology 82: 327-348.
Wooldridge, M. and Jennings, N.R. (1995) Intelligent Agents: Theory and Practice. The
Knowledge Engineering Review. 10(2):115-152.
Wright, S. (1932) The Roles of Mutation, In-Breeding, Cross-Breeding and Selection in
Evolution. Proceedings of the Sixth International Congress on Genetics. 356-366.
Zelinka, I. (2016) On Possibilities of Evolutionary Synthesis of Robot Control
Sequences. In Proceedings of the 2016 IEEE International Conference on Real-Time
Computing and Robotics. June 6-9, 2016. Angkor Wat, Cambodia. 332-337.
VITA
MARC LATORELLA
EDUCATION:
Pennsylvania State University, State College, PA
December 2017 Ph.D. Industrial and Manufacturing Engineering
Dissertation: “Evolving Intelligent Control Rules for a Collaborative Multi-Agent Manufacturing Cell”
State University of New York at Buffalo, Buffalo, NY
January 1997 M.S. Operations Research
State University of New York at Geneseo, Geneseo, NY
August 1991 B.A. Mathematics with Physics and Computer Science minors
Cornell University, Ithaca, NY
August 1988 - May 1989, August 1985 - May 1987 Computer Science Engineering Curriculum
EXPERIENCE:
Software Developer / Researcher, Trelletics Consulting, Inc. Yorktown, VA.
05/09 - Current Worked on research and software projects in simulation-based training.
Programmer / Analyst, Metrigenics, Inc. Beverly, MA
04/03 - 02/09 Developed facial recognition software to create composite faces from DNA.
Graduate Teaching Assistant, Industrial Engineering Dept., Penn State University. State College, PA
08/02 - 12/02, “Information Technology for Industrial Engineering" - Information Modeling, Java,
08/98 - 05/00 Data Communications, Distributed Computing, Modeling & Integration, Databases,
Distributed Collaborative Design, Internet-Based Manufacturing.
NASA Graduate Research Fellow, Crew Systems Ops Branch, NASA Langley. Hampton, VA
07/00 - 06/02 Conducted Ph.D. research on development and evaluation of evolutionary control
strategies towards semi-autonomous planning and operation in Freeflight program.
Assistant Research Analyst, Dept. of Autonomous Agents, Info. Science & Intelligent Systems
Division of the Applied Research Laboratory, Penn State University. State College, PA
05/96 - 08/98 "SAMON Project" - Designed and developed web-based simulation testbed facility for the
Office of Naval Research for collaborative autonomous underwater vehicle missions.
Graduate Teaching Assistant, Industrial Engineering Dept., Penn State University. State College, PA
08/94 - 05/96 "Introduction to Manufacturing Systems" - Manufacturing Systems, RETAD, Quality
Control, Resource Allocation, Production & Inventory Control, Vendor Integration.
Assistant Operations Analyst, Bulk Gases Distribution Division, Praxair, Inc. Tonawanda, NY
01/93 - 12/93 Upgraded large-scale logistics program to schedule national distribution of bulk gases.
ACTIVITIES:
Graduate Student Association: Industrial Engineering Senator
08/95 - 05/96 Pennsylvania State University. State College, PA
08/92 - 05/94 State University of New York at Buffalo. Buffalo, NY
Human Factors Society: Founding Member and Treasurer
08/86 - 05/87 Cornell University. Ithaca, NY
SKILLS & INTERESTS:
Python, C++, XML, SQL, JAVA. Evolutionary Learning, Information Technology, Genetic Algorithms, Databases,
Datamining, Multi-Agent Systems, Neural Nets, Simulation Modeling, Expert Systems, Finite State Automata.
