Impact of SelectionStrength on the Evolutionof Regulatory Networks

W. Garrett MitchenerCollege of Charleston Mathematics DepartmentMitchenerG@cofc.eduhttp://mitchenerg.people.cofc.edu

Introduction

The problem that led to this project was a result from Plotkin and Nowak [2000], inwhich an inequality from information theory puts an upper bound on the fitness ofa communication system with a one-to-one mapping of signal to meaning. If thereare enough topics of conversation, switching to a sequential coding system (string-of-signals to one meaning) yields a fitness advantage that overcomes that bound.See also Zuidema and de Boer [2009]. Evolutionary game theory can explain thesurvival and spread of such a system, but not its origin from organisms that do nothave it.

The basis of biological computation is the reaction or regulatory network. How aresuch networks discovered by selection-mutation processes?

The Utrecht Machine (UM) is a discrete abstraction of a gene regulatory network.For this project, an evolutionary simulation is used to discover UM-based agentsthat solve a data encoding problem. Details of the selection process have a signifi-cant impact on population dynamics: Weak selection leads to shorter overall timesto first perfect solution, shorter genomes, fewer repairs to auxiliary mechanismsafter an improvement to the main mechanism, and fewer atypical synaptic codes,and is more likely to discover improved partial solutions by breeding outliers. Thetimes at which key innovations appear and spread depends on the population statein surprising ways.

FYI: I call it the Utrecht Machine because the conference where I first presented it was

EvoLang 2010 in Utrecht.

Traditional population genetics

Abstracts away the generegulatory network.Models the genotype as alist of alleles, mapped to anumerical fitness for thephenotype. Markovian &memoryless.

A1 A1

B1 B1

C2 C2

♂ ♀Parents

A2 A2

B1 B2

C3 C3

F[...B1B1...]=4.5F[...B1B2...]=4.8

Fitness table

...

P[B1⇝B2]=0.01

Mutation table

...

A1 A2

B1 B1

C2 C3

A1 A2

B1 B2

C2 C3

A1 A2

B1 B1

C2 C3

A1 A2

B1 B2

C2 C3

B2B1B1

B1B1

B1B1

♂

♀Offspring

Systems biology

Genes produce functionalproducts (enzymes, etc.)but also regulatoryproducts that promote orrepress other genes.Complex networks ofinteracting molecules.

lacIrepressor

lacZenzyme lacY lacA

lactose

glucose

galactose

lacI disabledby lactose

lac operon

Population game dynamics: replicator-mutator

(1) xj =

K∑k=1

xkFkQkj − xjφ

• k = 1, 2, . . .K: types of individuals• xk = fraction of population of type k• Bkj = payoff for type k interacting with type j• F = Bx: vector of average payoffs by type• φ = xTF: overall average payoff• Qkj = probability offspring of type k is type j

Population genetics: Moran-Mendel

The population is finite. Dynamics form a Markov chain: Kill one agent uni-formly, replace by cloning another chosen in proportion to fitness.

• A1,A2, . . . ;B1,B2 . . . : alleles at loci A & B

• Phi, . . . : prob of mutation from Ah to Ai . . .• Fj1j2k1k2 = fitness of genotype Aj1Aj2Bk1Bk2

How to combine selection-mutation populationdynamics with relevant molecular details?

• Genome = inherited record of instructions:Should be a string of letters, subject to mutation and recombination like DNA

• Virtual machine = how to build the phenotype:Should be an artificial regulatory network

The Utrecht Machine or UM is designed to meet those requirements. The state ofthe UM is a table mapping patterns p to integer activation levels Ap. Each patternis metaphorically a protein that binds to a promoter or inhibitor sequence on DNA.The activation level of a pattern is how many units of it are present.

Each UM reaction instruction [As > θ =⇒ incAp, decAq] is parameterized by fourintegers: a switch pattern s, a threshold θ, a pattern p to activate called p-up, anda pattern q to inhibit called p-down. An instruction is active if the activation level ofits switch pattern As satisfies As > θ. An active instruction adds 1 to the activationof its p-up pattern and subtracts 1 from the activation of its p-down pattern. Allactive instructions are performed simultaneously.

A bit of input is provided by increasing the activation of a particular pattern duringeach time step the input bit is set.

Example: UM for binary NAND

Output

0

NAND

6

4

5

0

1

2

3

1

1

4

Input

0

1

Network representation, essentials only b0 b1 ¬(b0 ∧ b1)

F F TF T TT F TT T F

A0 > 0 =⇒ incA0, decA7

A1 > 0 =⇒ incA6, decA7

A2 > 0 =⇒ incA6, decA7

A4 > 1 =⇒ incA7, decA6

A5 > 1 =⇒ incA7, decA6

Two bits of input supplied through patterns 4 and 5, one bit of output read as1(A6 > 3). Stop when A0 > 4 or after a maximum of 10 time steps.

UM reaction instructions are just tuples of 4 integers, and their effects are indepen-dent of the order in which they are listed. They can therefore be encoded as a binarygenome and subject to biologically realistic mutation and recombination.

Building the network representation

Output

0

06

4

3

6

1

7

6

2

7

7

4

6

0 7

7

5

6

0

0

0

0

1

1

0

0

1

1

4

5

Input

0

1

One link for each. . .

• input bit

• output bit

• stop signal

Each instructiongenerates two linksfrom the switchpattern node: one forp-up, one for p-down.

Links for p-up, input, & output are solid with black arrowhead. Links for p-down are dashed with white arrowhead. Pattern nodes are circular for mostpatterns, keystones for those involved with input and output. Numbers on linksindicate a threshold. A link with no number is activated by an input bit insteadof a threshold.

Reduction from full to essential network

Network representation, all details

Input

0

1

NAND

6

4

5

0

7

1

2

1

1

0

0

1

10

0

00

Output

03

4

Merge pattern nodes, layout so that infor-mation mostly flows left to right.

Trim non-essential details...

does not affect output

switch pattern irrelevant

Remove pattern nodes that don’t mat-ter, like 7, which is used as a no-op or bitbucket in this machine. Turn nodes wherethe pattern doesn’t really matter into dia-monds. This happens to patterns 0, 1 and 2because all instructions with these for theswitch pattern happen to always be active.Then neaten up the layout.

Example: NAND with genome and step-by-step interface

0 0 0 7

1 0 6 7

2 0 6 7

4 1 7 6

5 1 7 6

genome bits

instruction bits

instruction integers

input / output bits

activation levels

pointer to next instruction

time

switch pattern

threshold p-upp-down

Triples of genome bits are decoded using majority-of-three into instruction bits,which are read left-to-right as the binary representations of integers.

Why the extra genome encoding layer, majority-of-three?

GA

A

A

A

A

G

G

G

G

C

C

C

C

U

U

U

U

UC AG UCAGUCAGUCAGUCAGU

CAG

UCAGUCAGUCAGUCAGU

CA

GUCAGUCAGUCAG

UCAG UCAG

P

S

U

nG

nG

oG

oG

oG

G

P

P

P

P

P

nM

nM

M

M

nM

nM

nM

Phenylalanine

Phe

Leucine

Leu

Leucine

Leu

Proline

Pro

Histidine

His

Glutamine

Gln

Isoleucine

Ile

Methionine

Met

Threonine

Thr

Asparagine

Asn

Lysine

Lys

Arginine

Arg

Arginine

Arg

Valine

Val

Alanine

Ala

Glutamic acid

Glu

Aspartic acid

Asp

Glycine

Gly

Serine

Ser

Serine

Ser

Tyrosine

Tyr

Cysteine

Cys

Tryptophan

Trp

Stops

Stop

E G F LS

S

Y

C

WL

P

H

R

R

QIM

TN

K

V

A

D89.09

75.07

174.20

174.20

146.19

165.19

133.11

117.15

147.13

146.15155.16

115.13

105.09

105.09

131.18

132.12

MW

= 14

9.21 D

a

131.18

119.12

204.23

131.18

181.19

121.16

HN

NH2

NH

H2N

OH

O

H2N

CH3 OH

O

H2N

O

H2N

OH

O

O

HO

H2N

OH

O

HS

H2N

OH

O

H2N

O

NH2

OH

O

O

OH

H2N

OH

OH2N

OH

O

NH

H2N

OH

O

N

CH3 CH3

H2N

OH

O

CH3

CH3

H2N

OH

O

CH3

CH3

H2N

OH

O

H2N

H2N

OH

O

CH3 S

H2N

OH

O

H2N

OH

O

NH

OH

O

H2N

HO OH

O

H2N

HO OH

O

H2N

HO

CH3

OH

O

NH

H2N

OH

O

HO

H2N

OH

O

H2N

CH3

CH3

OH

O

BasicAcidicPolarNonpolar(hydrophobic)

S -M - P - U - nM -oG - nG -

SumoMethyl

PhosphoUbiquitinN-Methyl

O-glycosylN-glycosyl

Modification

am

ino a

cid

2nd1st position 3rdUC

Picture by Kosi Gramatikoff and Seth Miller, posted on Wikipedia

The natural genetic code maps 64three-letter codons to 20 or so amino acidsplus a few control codes. Most amino acidsare represented by several synonymouscodons. Some of these codons are robust inthat substitution mutations are likely toresult in another synonymous codon,leaving the resulting protein unchanged.Ex: CUU CUC, both code for leucine.Other codons are fragile in that they aremore likely to mutate to a different aminoacid. Ex: UUA UUC switches fromleucine to phenylalanine. An excess offragile codons is thought to be evidencethat a gene has recently been subject toselection [Plotkin and Kudla, 2011]. Themajority-of-three encoding in the simulationis so that a future project can address thisphenomenon.

Recombination via haploid crossover

Agents in the simulation have a single chro-mosome. They reproduce in pairs. Thegenomes are aligned at the beginning andsplit at a random location. The first part ofone (blue) is attached to the second part ofthe other (red) to form the offspring genome.

Supported mutations

1 0 0 1 → 1 1 0 1single bit substitutionsprob 0.005 per genome bit

→ ✂

gene (instruction) deletionprob 0.001 per gene

→gene (instruction) duplicationprob 0.001 per gene

Experimental task: Transmit 2 bits over time

We use a selection-mutation process to evolve solutions to a sequential coding prob-lem. A genome is built into an agent follows: Two UMs are built from the genome,one sender and one receiver. The sender gets two bits of input, plus constant inputinto its role pattern 1. The receiver must generate two bits of output that reproducethe input, and signal when to stop. The receiver gets input from the sender througha single synapse, plus constant input into its role pattern 2.

Sketch of agent & scoring details

Input

0

1

Sender

3

8

9

1

Receiver

4

12

13

02

10

Output

0

1

10

10

10

Each agent is presented with allfour possible input words 〈00〉, 〈10〉,〈01〉, 〈11〉, starting at a zero statefor each and running for up to 100time steps. It earns 10, 000 points foreach bit correctly transmitted, plus100 − max(20, t) each time it stops af-ter t steps. Maximum possible score:10, 000× 4× 2 + 80× 4 = 80, 320.

Selection protocols

...

Agents are sorted in descending order by score. Then 300 newagents are created as follows: k1 new agents are created by breed-ing pairs selected uniformly at random from among the top h1, thenk2 are bred from the top h2, . . . . New agents are added to the top*of the list, and agents on the bottom are killed to maintain a con-stant population of 500. The list is re-sorted, preserving the originalorder* in case of a tie.

*These details prevent stagnation.

Selection strengths:

Protocol name (h1,k1), (h2,k2) . . .

Very weak (300, 500)—all agents breed equallyWeak (100, 200), (500, 100)

Strong (100, 300)—only top 100 breedVery strong (10, 300)—only top 10 breed

Even under very weak selection there is still some selection because agents withlow ratings are more likely to end up at the bottom of the list and die. All of theseprotocols eventually yield maximum-scoring solutions starting from an initial popu-lation of 500 genomes each with 32 genes generated uniformly at random. Patternshave six bits (0-63) and thresholds have four bits (0-15). Each gene has 22 instruc-tion bits, which are encoded by majority-of-three as 66 genome bits.

Example solution to bit transmission problem

Output

0

1

Receiver

0

2

3

4

12

13

22

42

47 10

15

3

10

610

7

6

4

3 4

5

4

012

Sender

1

0

3

5

8

9

47

10

7

3

3

1

3

9

0

12

6

Input

0

1

Timing

Pre-boost

Early activation

Late activation

cruft

00100111

0 5 10 15 20

input words stop signal

synapse turned on

synapse turned off

Synaptic code

time

Typical synaptic code:

〈00〉 is represented bynever activating thesynapse. 〈11〉 isrepresented by veryearly activation of thesynapse. 〈10〉 and 〈01〉are represented byintermediate activationtimes. This code istypical of solutionsevolved by thesimulation.

• Chicken-and-egg problem solved incrementally

• Timing mechanism entangled with other mechanisms

• Spurious but harmless activation of receiver mechanism in sender

• Recombination assembles new combinations of genes, discovers better net-works, often from sub-optimal parents

Rating vs. time

0k 50k 100k 150k 200k0

20 000

40 000

60 000

80 000

æ

ææ

æ ææææ æ æ ææ æ

0 12k

60 000

60 320

æ

æææ ææ æ

200k 230k80 160

80 320

Maximum rating as a func-tion of agent ID number.Agents are numbered asthey are born, so ID numbermeasures time. Large jumpsoccur at each major innova-tion, that is, when an agentappears that transmits anadditional bit correctly. Re-ductions in time spent yieldsmaller jumps, shown asinserts.

Typical trajectory is as follows: (1) Initially, most agents always output 〈00〉, scoring40, 000 = half credit. (2) Recombination joins a link from one input bit directly tothe synapse with a link from the synapse directly to the corresponding output bit,scoring 60, 000. (3) A sender mechanism appears that generates a fully informativesynaptic code, but no benefit is realized until. . . (4) A mechanism appears that con-nects to the other output bit, scoring 70, 000. (5) Some sort of complex mechanismis finally discovered that transmits all 8 bits correctly. After each major innovation,the timing mechanism may need to be repaired to get the other 320 points.

Distribution of ratings by generation

10 20 30 40 50

100

200

300

400

500

250 260 270 280

100

200

300

400

500

690 700 710 720 730

100

200

300

400

500

20 40 60 80 100

1

5

10

50

100

500

240 260 280 300 320 340

1

5

10

50

100

500

700 720 740 760 780

1

5

10

50

100

500

Color scheme:

1005 1010

100

200

300

400

500

0

10000

20000

30000

40000

50000

60000

70000

80000

Upper row: Stacked bar chart of counts of agents whosescore is between 10, 000n and 10, 000(n + 1) in each genera-tion for 50 generations after each major innovation.Lower row: Log-scale plot of count of agents whose scoreis nearly maximal in each generation for 50 generations af-ter each major innovation. (That is, the count in the highestcorresponding sub-bar in the chart above it.)

Each major innovation is more difficult than the previous ones. When an agent thatcan transmit an additional bit appears, that ability spreads exponentially, but forhigher-scoring innovations, the initial delay is longer, the growth rate is lower, andthe equilibrium fraction of high-scoring agents is lower.

Varying selection strength

Increasing selection strength reduces the number of agents that can reproduce,which has some counter-intuitive consequences. Aggregating observations from10, 000 runs of the simulation:

Log length of genome

1.4 1.6 1.8 2.0 2.2 2.4

V. Weak

Weak

Strong

V. Strong

Log agent ID

4.5 5.0 5.5 6.0 6.5 7.0

V. Weak

Weak

Strong

V. Strong

Distributions of the log10 length of the genome (left) and ID number (right) of thefirst agent in each sample run to achieve the maximum score. Bands indicatedeciles. The correlation is imperfect, but in general stronger selection leads tolonger genomes, and more agents searched (= more search time).

Unscaled distribution charts

Length of genome

50 100 150 200

V. Weak

Weak

Strong

V. Strong

Agent ID

0 2.0 ´106 4.0 ´106 6.0 ´106 8.0 ´106 1.0 ´107 1.2´107

V. Weak

Weak

Strong

V. Strong

Distributions of unscaled genome length and agent ID number. Bands indicatedeciles. A few especially long genomes and long-running samples draw out theuppermost decile.

Innovations through outliers

During the long equilibrium phases, most of thepopulation hovers near the peak of a ridge inthe fitness landscape. Genetic diversity ensuresthat there are always a few outliers that don’tachieve the highest score present in the popu-lation at that time. But they are more likely tobe near the edge of the zone of attraction of theridge, in which case their offspring are morelikely to jump to another ridge—an innovation.

In weaker protocols, there aremore samples with at least onemajor innovation where at leastone parent was an outlier. Thisphenomenon appears to be aconsequence of the fact thatweaker selection maintainsgreater diversity. V. Weak Weak Strong V. Strong

0.2

0.4

0.6

0.8

1.0

With

Without

Typical vs. atypical synaptic codes

Example atypical synaptic codes:00

10

01

11

0 5 10 15 20

00

10

01

11

0 5 10 15 20

Most solutions use typicalsynaptic codes something likethe example solution. But 5-20% of solutions end up withsomething very different, andoften with much more complexmechanisms.

In stronger protocols, a greaterfraction of runs settle on atyp-ical synaptic codes. Possibly,the reduced diversity associ-ated with stronger selectionincreases the likelihood thatthe population gets trapped ina ridge with a smaller zone ofattraction, which yields moreatypical results.

V. Weak Weak Strong V. Strong

0.2

0.4

0.6

0.8

1.0

Atypical

Typical

Timing repairs

V. Weak6 7 8

Weak6 7 8

Strong6 7 8

V. Strong6 7 8

20

40

60

80

100 Percent of runs that required repair or tweak-ing of the timing mechanism after the inno-vation that enables the transmission of anadditional bit: The height of the bar over n isthe percent of runs that jump from a rating of10, 000n + 320 to between 10, 000(n + 1) and10, 000(n+ 1) + 319.

Frequently, a mutation that improves the bit transmission mechanism disruptsthe timing mechanism, sometimes stealing part of its structure. Such an eventyields a jump of just under 10, 000 in score, followed by minor jumps that re-store the additional 320 timing points. Almost all runs have to repair (or ini-tially discover) the timing mechanism after the first major innovation, but only10-40% do so after the others. The very strong selection protocol has to makethe most repairs, but the trend is imperfect: the very weak protocol makesmore repairs than the weak and strong protocols after the innovation at n = 7.This means that it is not at all unusual (1) for evolution to sacrifice one func-tional mechanism to build another, and (2) for the resulting mechanisms to beentangled.

Time for last major innovation

Very weak Weak Strong Very strong

2.0 ´106 4.0 ´106 6.0 ´106 8.0 ´106 1.0 ´107 1.2´107

1.´10-9

1.´10-8

1.´10- 7

1.´10-6

0.00001

2´106 4´106 6 ´106 8 ´106

1.´10-9

1.´10-8

1.´10- 7

1.´10-6

0.00001

2´106 4´106 6 ´106 8 ´106 1´107

1.´10-9

1.´10-8

1.´10- 7

1.´10-6

0.00001

2.0 ´106 4.0 ´106 6.0 ´106 8.0 ´106 1.0 ´107 1.2´107

1.´10-9

1.´10-8

1.´10- 7

1.´10-6

0.00001

0 500 000 1.0 ´106 1.5´106 2.0 ´1060

1.´10-6

2.´10-6

3.´10-6

4.´10-6

5.´10-6

6.´10-6

0 500 000 1.0 ´106 1.5´106 2.0 ´1060

1.´10-6

2.´10-6

3.´10-6

4.´10-6

5.´10-6

6.´10-6

0 500 000 1.0 ´106 1.5´106 2.0 ´1060

1.´10-6

2.´10-6

3.´10-6

4.´10-6

5.´10-6

6.´10-6

0 500 000 1.0 ´106 1.5´106 2.0 ´1060

1.´10-6

2.´10-6

3.´10-6

4.´10-6

5.´10-6

6.´10-6

Upper row: Log-scale PDF histogram of number of agents searched betweennext-to-last and last major innovation. Lower row: Hazard function histogramof the same.

In traditional population dynamics, mutations are modeled with a Markovchain: An allele Aj in a parent mutates to Ak in a child with a probability thatdepends only on j and k. Starting from a population of all Aj, the time to firstappearance of Ak is approximately exponentially distributed. However, in thesimulation, the time required for the last major innovation (jump from a maxi-mum rating of 70, 000± to 80, 000) is not exponentially distributed: Its PDF isnot linear on a log plot, and its hazard function is not constant.

Hazard function

For a random variable T with density function f and cumulative distributionfunction F, the hazard function is

h(t) = lim∆t→0

P (t 6 T 6 T + ∆t | T > t)∆t

=f(t)

P (T > t)=

f(t)

1 − F(t)

The name comes from the idea that T is the time at which a device fails, so thehazard function is the instantaneous failure rate of the device at time t giventhat it has survived to time t. If T has an exponential distribution, a calculationshows that its hazard function is a constant, equal to the reciprocal of its mean.

Discussion

These simulations show that the origin story must be complicated. What seemsto be happening: The population goes through periods of neutral drift and diver-sification. Recombination tries many combinations of genetic variants of partiallysuccessful genes, including those from parents that score lower than the currentmaximum (epistasis). Innovations frequently take the form of entangled mecha-nisms that often must be tuned or repaired.

Stronger selection results in generally longer genomes and greater search time.There are also indications that it traps the population tightly near ridges in thefitness landscape with small zones of attraction (fewer innovations through outliers,more atypical synaptic codes). A possible explanation is that strong selection lowersgenetic diversity (founder effect).

The time between innovations is not exponentially distributed, which indicatessome memory effect: Shortly after one innovation, the population is more likelyto have another. Possibly during a selective sweep (time during which an innova-tion is spreading) the population retains some diversity which increases the chanceof another innovation. If the sweep runs to completion, the process is reset andbecomes memoryless.

Next steps

• Find ways to measure diversity directly and check these possibilities. Thegenomes are all different lengths with different amounts of space betweenuseful genes, so an alignment algorithm is needed to measure genotypic dis-tances.

• Implement fully diploid genomes. This seems to be an important detail: Re-combination may be more likely to find useful combinations of genes whenthey are at a distance on the genome. Under haploid recombination, it islikely that none of the offspring of such an individual will inherit both partsbecause crossover will occur in between them. But with diploid recombina-tion, roughly 1/4 of the offspring will inherit both parts.

• Model regulatory network formation more formally. This will require Markovchains with huge state space.

References

W. Garrett Mitchener. A discrete artificial regulatory network for simulating the evolutionof computation. In EvoNet2012: Evolving Networks, from Systems/Synthetic Biology toComputational Neuroscience, East Lansing, Michigan, USA, 2012.

Joshua B. Plotkin and Grzegorz Kudla. Synonymous but not the same: the causes and conse-quences of codon bias. Nat Rev Genet, 12(1):32–42, January 2011.

Joshua B. Plotkin and Martin A. Nowak. Language evolution and information theory. Journalof Theoretical Biology, 205(1):147–159, July 2000.

Willem Zuidema and Bart de Boer. The evolution of combinatorial phonology. Journal ofPhonetics, 37(2):125–144, April 2009.