Relationships between Evolutionary Computation &

Biology

Focal Points:

Combining Evolution & Learning in Hybrid Adaptive Systems

Exploiting Coevolution in Evolutionary Computation

2

Biology in Evolutionary Computation

• Basic Neo-Darwinian Evolution

– The essence of most EAs

• Genotype-Phenotype Distinction

– Developmental processes that derive ptype from gtype

• Fundamental Theoretical Results

– Hardy-Weinberg Law

– Fisher’s Theorem

• Combinations of Evolution & Learning

– Lamarckianism

– The Baldwin Effect

• Coevolution

3

Diploid Genetics

AbCd

aBCd

• For each gene, the allele pair will interact in some way to produce a protein (phenotypic trait).

• In some cases, one allele “dominates” while the other is “recessive” => one allele is completely expressed; the other is ignored.

• In other cases, the two alleles have a combined effect.

Heterozygous for genes 1 & 2

Homozygous for genes 3 & 4

Chromosome Pair

4

Diploid Recombination

ABCD

abcd

WXYZ

wxyz

abCD

ABcd

AbcD

aBCd

WXyz

wxyZ

wxYZ

WXYz

Gamete Pool

Crossover

Generation K

abCD

wxYZ

Generation K+1

aBCd

wxyZ

Sex*

5

Population Genetics Notation

Gene Pool

AA

Aa

aa

aaaaAAAA AA

Gene frequenciesp = fraction of A’s = 9/16q = fraction of a’s = 1 - p = 7/16

Genotype frequenciesD = fraction of AA’s = NAA/N = 4/8H = fraction of Aa’s = NAa/N = 1/8R = fraction of aa’s = Naa/N = 3/8

Genotype NumbersNAA(4), NAa(1),Naa(3)

Population SizeN = 8

p = (2NAA + NAa) / 2N = D + H/2q = (2Naa + NAa) / 2N = R + H/2

6

The Hardy-Weinberg Law

Given: • Initial genotype distribution: D0,H0,R0 => p0, q0

Assume:• Random mating• No mutation• No selection - all genotypes have same fitness• No genetic drift - population size large enough that the mating probabilities and child genotypedistributions equal the statistical estimates.

Results:• D1 = p0

2 H1 = 2p0q0 R1 = q02

• p1 = p0 q1 = q0

• D, H, R values remain constant after gen 1.• Gene and Genotype freqs at stable equilibrium.

7

Lessons for Evolutionary Computation

• Diploid EAs can be useful– Allow recessive traits to linger in the population without always being

expressed. If the environment (problem) changes such that expression of the recessive gene is desirable, then aa’s will have a selective advantage and rise in frequency.

– Without diploidy, we would have to wait for the proper mutation to create an a.

– We get recessiveness for free in GP, since some subtrees may not be expressed in one individual, but when combined with other code, they get executed. E.g. (if nil SubtreeA SubtreeB)

• Don’t forget Hardy, Weinberg & Fisher!– Without selective pressure and mutation, evolution can quickly

stagnate, even when random recombination (I.e., crossover) is used.

– Evolutionary speed is a function of fitness variance.

– Fitness functions need to have a wide range of outputs.

– Selection mechanisms need to maintain selection pressure, or create it in cases where all individuals have similar fitness.

8

Learning + Evolution

• Individuals improve during their lifetimes (plasticity/learning) & populations improve (avg fitness increases) over many generations.

• Learning speeds up evolution by:

– Lamarckianism: Direct inheritance of acquired traits

• Results of learning (the improved phenotype) is reverse-encoded into the genome prior to reproduction => children inherit what their parents learned/acquired.

• Disprove biologically, but useful for EC.

– The Baldwin Effect: Indirect inheritance of acquired traits

• Some genotypes are not inherently optimal, but their corresponding phenotypes can become so via learning. Hence, these genotypes have a selective advantage and will remain in the population until a mutation produces optimal genotypes.

• No ptype-to-gtype reverse encoding necessary.

• Accepted by biologists, but difficult to test.

• Useful in EC.

9

Lamarck -vs- Baldwin

• Direct -vs- Indirect inheritance of acquired characteristics

• Below, the blue 3-fingered component is an acquired trait.

Lamarck

Genotype

Phenotype

Baldwin

SelectiveAdvantage

Mutation

Many generations

1 generation

10

Lamarckianism• Jean-Baptiste Lamarck (1744-1829)

• Earliest advocate of an evolutionary theory

• Inheritance of Acquired Characteristics (1809)

Basic Idea:

If organisms adapt during their lifetimes, then those changes can be DIRECTLY passed on to their offspring. (inheritance of acquired traits).

Modern Implications

The results of individual plasticity, such as learning or physical changes, can be reverse encoded into the genome prior to reproduction. So children inherit the fruits of their parents plasticity as innate traits.

Biologically disproven, except in a few rare cases.

11

Contributions of Lamarckianism

3 Things that Lamarck may have gotten right (Schull, 1996):

1. Individual plasticity can influence evolution.

2. Similar individual & populational adaptivity.

3. Collective activity is relevant to evolutionary theory.

Agents behave (and learn) purposely, so these

collective activities could have an emergent effect upon the course of evolution.

Lamarck’s main mistake is still very useful:

Lamarckian inheritance can improve evolutionary computation!!

12

Evolutionary Computation + Local Search

Fit

ness

Genotype

Phenotype

Local Search = Learning/Plasticity

Lamarckianism

13

Lamarckianism in EC

Basic Process– Convert genotypes to phenotypes– Phenotypes change via learning– At generation end, CONVERT THE

UPDATED PHENOTYPES BACK INTO A CORRESPONDING GENOTYPE

– Crossover and mutate new genotypes to create the next generation

14

Partial Lamarckianism (Houck, et. al., 1997)

– Only N% of genotypes are changed via the reverse encoding from phenotypes.

– N = 20-40% gives best results on benchmarks.

– This beats pure Lamarckianism (N = 100) and pure Baldwin Effect (N = 0 with plastic ptypes)

– N = 100 is dangerous• Leads to premature convergence in static environments

• Kids should NOT inherit too much of what parents learned in dynamic environments, since kids’ world is different from parents’

15

The Baldwin Effect

A new factor in evolution (James Baldwin, 1896)

Basic Idea:

If organisms adapt during their lifetimes, then those changes can be INDIRECTLY passed on to their descendants, although it may take many generations.

Two Phases

1. Plasticity enhances evolution by INCREASING FITNESS DIVERSITY and smoothing the fitness landscape. **************************

Individuals who adapt their way to higher fitness have a selective advantage over those who, despite adaptivity, cannot.

2. The results of plasticity are assimilated into the gene pool via chance

mutations.

16

• Biologically plausible, but hard to prove in natural systems.

• Hinton & Nowlan (1987) - Trivial evolutionary simulations clearly show it!

17

Baldwin Effect Phase I: Learning EnhancementF

itne

ss &

# P

heno

type

s

Genotype

•Learning/Plasticity = Local Hill Climbing in Fitness Space•Phenotypes near the base of the peak can achieve fitness increases.

Phenotype

AD1

SmootherFitnessLandscape

D2

18

• D1, D2: All phenotypes have the ability to learn, but only those near the base of the peak can achieve fitness increases. This selective advantage moves the genotype/phenotype distribution from D1 to D2, hence closer to the optimal phenotype P*.

19

Baldwin Effect Phase II: Genetic AssimilationF

itne

ss &

# P

heno

type

s

Genotype

Phenotype

A

Economy of Flexibility 1. Learning accelerates evolution,2. Evolution removes the need for learning.3. Evolution removes the ability to learn!

D2

• Mutation produces genotypes hard-wired for phenotype A.

D3

• Learning has a cost => Selection favors Natural Born A’s

D4

20

• Genetic operations spread the population distribution from D2 to D3, producing individuals with hard-wired optimal penotypes, P*. Due to the cost of learning. These natural-born P*s have a selective advantage over the learned P*s, and the distribution moves from D3 TO D4.

21

Preconditions for the Baldwin Effect (Mayley, 1996)

1. Learning has both benefits (phase I) and costs (phase II)

2. The genotype and phenotype spaces are correlated

• There needs to be an achievable genotypic change for each learned phenotypic change (Observe nature !).

• Without this, the mutations that should produce phase II will fail to find a genotype that encodes the previously-learned trait (There should be a possible genetypic change for each acquired trait).

It’s the exploitation of the benefits and reduction of the costs that provide the selection pressure for :

1. The adoption of a learned behaviour.

2. Its genetic assimilation.

22

Benefits of Learning (Mayley, 1996)

1. Adaptive members of the population are able to ”find” new advantageous behaviours that less plastic individuals are unable to perform. This means that these adaptive individuals gain the upper hand and are selected for.

2. Learning provides individuals with the ability to adapt to an environment that is changing at a faster time scale than that on which evolution operates.

3. A learning mechanism may be able to provide an individual with behaviours that are simply very hard to evolve. (i.e. acquiring a language)

23

Costs of Learning (Mayley, 1996)1. Cost as function of time in the juvenile (learning) stage

– Time wasted in “ignorant” state.

– Energy expended during knowledge acquisition

– Delayed reproduction (until beyond juvenile stage)

2. Catastrophic costs

– Unreliability - environment may never provide the necessary stimuli for learning

– Death due to state of ignorance

3. Fixed costs

– Extra expense to support plasticity (particularly evident in EAs with a learning component)

4. Population-level costs (don’t affect juveniles directly)

– Parental investment in teaching the young.

All 4 types of costs can affect The Baldwin Effect

– Phase I: Too high costs => no learning => no Baldwin Effect Phase II: Higher costs => more selective pressure for assimilation => faster convergence (Leaning phase is cut short)

24

Genotype-Phenotype CorrelationF

itne

ss &

# P

heno

type

s

Genotype

Phenotype

P1 P2

G1G2

Correlated: near(G1,G2) <=> near(P1,P2)

P1 P2

G1G2

Easy mutation Harder mutation

25

Hinton & Nowlan (1987)• First evidence of the Baldwin Effect in a natural or artificial systemNeedle-in-a-haystack Task:

Goal: Find a particular 20-bit stringEvolution: GA with 20-gene chromosomes

3 Alleles: 0, 1, * (wildcard)Learning: Max of 1000 random guesses per individual.

Guess = replace a * with a 0 or 1.Fitness Function:

F = 1 + 19(1000 – NG)/1000 NG = # guesses (low better!)

Learning has a cost!

Fit

ness

Gtype/Ptype

Without learning: needle-in-haystack search

With learning: smoother, navigable space

26

Hinton & Nowlan (2)• Without learning: no fitness variance, since all individuals are equally

bad (unless, by pure chance, the needle is found).

– Thus, no evolutionary progress (avg fitness does not change)

– Search is random.

• With learning:partial solutions (that can learn their way to the correct solution) get partial credit and move search in the proper direction (up the smooth hill).

– Learning smoothes the fitness landscape by making more informative the points in the neighborhood of the optimum.

50

1

All

ele

Rat

ios

Generations

Correct

Wrong

Wildcard

Baldwin EffectPhase I Phase II

Degree of geneticassimilation = size ofCorrect-Wildcard gap and is directly proportional to thelearning cost.

27

Hinton & Nowlan (3)

• PHASE 1 : Larger diversity with learning, hence the probability that some and more of those individuals will get closer to the correct solution is higher. That’s the reason why number of corrects increase while wrongs are low in proportion.

• PHASE 2: The ones closer to the correct phenotype are favored and mutated. Wrong ones are almost completely eliminated while the correct ones are even more. Natural born ones are also favored over others.

28

How does Baldwin effect work in Evolution?

• Acquired traits are not coded back into the genotype. However, the genes which could acquire highly fit traits are expected to acquire similar highly fit traits in future if they are allowed to stay in the populations of coming generations. If so, with mutation these genes are expected to have traits acquired by learning until now to be CODED into their genoypes. This the way how learning effects evolution. Ordinary genes are not expected to be genetically mutated in a way to acquire traits that they are not able to do so with learning.

29

Evolutionary Reinforcement Learning (ERL)Ackley & Litmann (1991)

Complementary Neural Networks:

#1 learns via backprop; original weights from GA.

#2 provides state evaluations for #1; static weights from GA.

Current State

Action Network

Action

Evaluation Network

State Evaluation

Next State

World

#2 #1 Baldwin Effect:

Phase I: Genes for ANN #2 arerelatively fixed early on => selective pressure forlearning in ANN #1, whichdepends on ANN#2.Phase II:ANN #2 no need for learning. Goodbehaviors are assimilatedin ANN #1.

30

Using Learning to Faciliate the Evolution of Features for Recognizing Visual Concepts (Bala

et. al. 1996)

Objectives:

1. Assess the ability of a hybrid evolution and learning architecture to identify feature sets which result in good classificiation performance.

2. Understand clearly the role that learning played in this process.

3. Understand the extend to which various aspects of the Baldwin effect were present.

31

Enhancing Machine Learning with Evolution

Training Data

Feature Set Decision Tree

Fitness (Feature Set) = -1*Size + Entropy + Accuracy

GeneticAlgorithm

C4.5

Test Data

ClassifyTest

Bala et. al, 1996

• Each GA genome is a subset of the universal feature set.• Size(Feature Set) = Learning Effort/Cost

32

Feature Set Size

Testing data

Infomax measure

Tree Evaluation Tree Induction

FeatureMaskImage Data

Feature Extraction Examples

Training data

Trees

EVALUATION MODULE

GA MODULEFull Feature Set

Feature Subsets

Optimal Feature Subset

Acuracy

Entropy

Fitness Measure

33

GA + C4.5 Tests

Tasks:

Satellite Imaging & Facial Recognition

Evolution + Learning beats:

Learning alone = C4.5 using universal feature set.

Evolution alone =Fitness function without the accuracy term.

Baldwin Effect:

Stage 1: Learning selected for. Large feature sets dominate initially.

Stage 2: GA population converges to small feature sets that match the optimal sets learned by solo C4.5 (Genetic assimilation).

First example of the Baldwin Effect on a significant problem.

34

Reinforced Genetic Programming (RGP)

• Strongly-typed classify-and-act genetic programs.

• Actions = simple moves or monitored choices.

• Choice nodes enable Q-Learning by keeping track of previous actions and resulting rewards.

• Evolution provides problem-space abstractions for RL to work with.

• RL enhances GP evolution via Baldwin Effect or Lamarckianism.

Downing (2001) if

>

X

if

5 =

Y 3

PickMove

movenorth

PickMove

350 100 200 300 4000

0.5

1

1.5

2

2.5

3

3.5

4

Generation

Avg

-#-D

eci

sio

ns

RGP Maze Walkers

Start

Goal*

10 x 10 maze

0 100 200 300 4000

0.5

1

1.5

2

2.5

Generation

Fitn

ess

MaxAvgMin

Baldwin Effect

Phase I

Phase II

36

Coevolution

• In nature, the playing field is always changing. Populations are never evolving against a static environment, but against an ever-changing world, consisting of many other evolving populations.

• So changes to both the environment and to the other populations need to be taken into account w.r.t. design changes for improved fitness.

Competitive Coevolution (Arms Race)

• Competitors (e.g. predators & prey) must constantly improve to keep pace with one another.

• Over time, the individuals have higher absolute fitness (I.e. they are much better than their ancestors), but their relative fitness (vis-à-vis their current competitors) remains unchanged.

Cooperative Coevolution

• 2 or more species evolve such that each enhances its own selective advantage plus that of the other(s).

37

Coevolution in EC

Dilemma #1: Problem Impossibility

The problem can be so difficult, and the fitness function so hard to design, that all individuals in early generations get very low fitness => low fitness variance => slow or no evolution.

Dilemma #2: Problem Mastery

After many generations, all individuals may have the same high fitness but may not yet have found a solution. Once again, fitness variance is low => evolution may stop, and the optimal solutions may never be found.

Competitive Coevolution to the Rescue!• Use 2 competing populations: problems & solutions.• Fitness(problem) = # errors/omissions made by the solution individuals on that

particular problem.

Early: Problems are easier, so at least some of the early solutions can solve some of the problems => fitness variance => evolution.

Late: Problems become more difficult, and all solutions are pretty good. But there are usually tough problems that only some solutions can handle => fitness variance => evolution.

• So coevolution keeps selection pressure from getting too low, thus avoiding dilemmas #1 and #2 and maintaining evolutionary progress.

38

EC in Evolutionary Biology (Mitchell, 1996)

Standard Evol Bio techniques (drawbacks/problems in italics)

• Fossil records: incomplete

• View adaptations in real ecosystems: can’t do controlled experiments.

• Lab experiements (e.g. fruit flies): too few generations for major evolutionary changes.

• DNA analysis to reconstruct phylogenic trees: ambiguous & junk DNA.

• Mathematical models (usually differential equations): extremely simplified - abstract away many individual behaviors.

Computer Simulations of Evolution (e.g. Artificial Life)

• Controllable (via modifiable parameters)

• Repeatable

• Many generations can be run.

• Individual-based models require fewer abstracting assumptions: but it’s still a model.

• Generate lots of data: data interpretation problems

39

Summary: EC & Biology• Biological metaphors (even failed ones) have useful applications in

artificial adaptive systems.

– NeoDarwinism

– Lamarckianism

– Baldwinism

– Diploidy

– Coevolution

• Evolutionary Computation can contribute back to biology

– Existence proof of the Baldwin Effect.

– Assessing roles of mutation & crossover in evolution

– GP for discovering metabolic pathways and solving protein-folding problems.

– GA in some algorithms that help to map the human genome.

– Evolutionary Simulations for exploring possible complex interactions among evolving species.

40

Lamarckian Modesty

Can there be any more important conclusion...or any to which more attention should be paid that that which I have just set forth?

... Lamarck’s final sentence of an excerpt published in 1914.