CS 591Complex Adaptive Systems

Spring 2009Measures of Complexity

Melanie Moses1/28/09

Topics

• Why do we need formal definitions of complexity?

• Complex Adaptive Systems

• Definitions of Complexity– Flake & Gell Mann

Computational Definitions of complexity– Running time complexity – Gell Mann’s Effective Complexity

• Kolmogorov (or Algorithmic) complexity• Mutual Information• Logical Depth

– Others• Per Bak• Stuart Kauffman• John Holland• Wolfram and Langton’s complexity of CAs

Why define complexity?• To estimate how long a particular system will take to solve a problem• To estimate difficulty in engineering complex systems • To understand the limits of prediction, approximation, and simulation • To answer fundamental scientific questions

• Does (why does?) complexity increase through evolution?– How can complexity arise when entropy should always increase?

“As evolution proceeded on the surface of the earth, there has been a progressive increase in size and complexity” J. T. Bonner Evolution of Complexity.

“Evolution tends to push systems towards the edge of chaos, where complex, interesting behaviors such as life can occur “ Chris Langton

“The universe, the biosphere, the econosphere have all become more complex” Stuart Kauffmann

Are these statements correct? If so, how do we quantify increases in complexity over time?

What are Complex Adaptive Systems?• Collections of agents:

– Molecules, cells, animals, planets, economic agents.

• Agents interact (locally) with one another and with their environment:– No central controller.– Interaction rules may be trivial or nontrivial (i.e. nonlinear); deterministic or

nondeterministic– Chemical reactions, cellular interactions, mating, buy/sell decisions.

• Unanticipated properties often result from the interactions:– Immune system responses, flocks of animals, settlement patterns, earthquakes,

speculative bubbles and crashes.

• Agents adapt their behavior to other agents and environmental constraints:

– Imitation, adaptation, learning.

• System behavior evolves over time:– Rules change, unfolding of complex behavior.

Example Complex Adaptive Systems• Biological systems

– Cells– Organs– Immune systems– Organisms– Ecosystems– Food webs– Symbioses– Colonies: ants, bees, termites, wasps; bacteria, corals…

• Social/Economic Systems– Nations– Firms– Households– Cities– Universities

• Technological Systems– The Internet– Peer to peer networks– Botnets– Computers?

• Which is most complex?

Characteristics of Complex Systems• What makes a system complex?

– Nonlinear interactions among components.– Multi-scale phenomena and hierarchy.– Evolution of underlying components and environments.

• Is complexity inherent in the system or in our understanding of it?– Ex 1: quantum computation and factoring– Ex 2: Differential equations vs simulations to model disease spread

• How to measure a system’s complexity?– By its unpredictability?– By how difficult it is to describe?

• Length of most concise description.• No single model adequate to describe system---the more models that are required,

the more complex the system. (Lee Segel)– The length of time before it halts or repeats itself, if ever?– Entropy?– Multiple levels of organization?– Number of interdependencies?

Caveat

Some believe that there is no general science of complex systems:

– “It’s becoming apparent that a theory of complexity in some sense of a great and transforming body of knowledge that applies to a whole range of cases may be untenable.” Sir Robert May (2001)

G.W. Flake’s Definition of Complexity

Flake’s scientific view• An intermediate view between Reductionism and Holism

– The interactions of agents…bind one level of understanding to the next• Nature is Frugal: Occam’s Razor• CAS characterized by

– Emergence, self-similarity, unpredictability, self-organizing– Collections/Multiplicity/Parallelism– Iteration, Recursion, Feedback– Adaptation, Learning, Evolution

• Conversion of the sciences facilitated by computation

Representative Measures of Complexity

• Computational complexity (Cook):– How long a program runs (or how much memory it uses).– Asymptotic.

• Language complexity:– Classes of languages that can be computed (recognized) by different kinds of abstract

machines.– Decidability, computability.

• Information-theoretic approaches:– Algorithmic Complexity (Solomonoff, Komogorov, and Chaitin):

• Length of the shortest program that can produce the phenomenon.– Mutual information (many authors)

• Logical depth (Bennett)• Effective complexity (Gell-Mann and Lloyd)

Computational Complexity• Introduced by Steve Cook (~1970).

– Asymptotic running time, and/or memory consumption of an algorithm.– Worst-case versus average-case.

• Important computational complexity classes:– P problems can be solved in polynomial time on a deterministic Turing Machine– for some polynomial function p: Cnk where c and K are constants independent of n

which is the size (length) of the input

– Polynomial time algorithms P are O (p(n))– NP (nondeterministic polynomial) problems can be verified in O(p(n)) or solved

on a non-deterministic (parallel) Turing Machine: subset sum {6,-1,-4, -2, 5}– NP hard: an algorithm for solving it can be translated to solve any NP problem,

e.g. traveling salesman– NP complete problems are both NP and NP hard

|)(| |)(| ),(| ck, iff ))(( is )(

ngcnfknngOnf

⋅≤>∀∃

Turing Machines

1. a tape2. a head that can r/w and move l/r3. instruction table4. state register

tape is infinite, all else is finite and discrete

Topics

• Why do we need formal definitions of complexity?

• Complex Adaptive Systems

• Definitions of Complexity– Flake & Gell Mann

Computational Definitions of complexity– Running time complexity – Gell Mann’s Effective Complexity

• Kolmogorov (or Algorithmic) complexity• Mutual Information• Logical Depth

– Others• Per Bak• Stuart Kauffman• John Holland• Wolfram and Langton’s complexity of CAs

Quicksort (P) vs Traveling Salesman (NP hard)

O(N2) or O(Nlog(N)) O(N!) or O(2N)

Computational Complexity

• Drawbacks • Constants can matter • Special cases vs worst cases• Approximations may suffice• Context dependent: what machine, what is already known..

• Benefit: Allows objective comparison of how hard problems are as input size increases

Murray Gell-Mann: Effective Complexity

• Complexity is highest for things that are neither strictly regular nor strictly random

• Kolmogorov or Algorithmic Information Complexity is based on the length of a concise description of a set.

• AIC is a measure of randomness…NOT a good measure of complexity

• Effective complexity is the length of a concise description of a set’s regularities

• Regularities can be found using Mutual Information

• What are AIC, entropy and Mutual Information?

Algorithmic Complexity (AC)(also known as Kolmogorov-Chaitin complexity)

• The Kolomogorov-Chaitin complexity K(x) is the length, in bits, of the smallest program that when run on a Universal Turing Machine outputs (prints) x and then halts.

• Example: What is K(x) where x is the first 10 even natural numbers? Where x is the first 5 million even natural numbers?

• Possible representations:– 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, … (2n - 2)– for (j = 0; j < n: j++) printf(“%d\n”, j * 2);

• How many bits?– Alternative 1: O(n log n)– Alternative 2: K(x) = O(log n)

• Two problems:– Calculation of K(x) depends on the machine we have available (e.g., what if we have a machine with an

instruction “print the first 10 even natural numbers”?)– In general, it is an uncomputable problem to determine K(x) for arbitrary x.

• How long is the program to do x?

Algorithmic Complexity cont.• AC formalizes what it means for a set of numbers to be

compressible and incompressible.– Data that are redundant can be more easily described and have lower AC.– Data that have no clear pattern and no easy algorithmic description have

high AC.

• What about random numbers? If a string is random, then it possesses no regularities:

– K(x) = | Print(x) |– The shortest program to produce x is to input to the computer a copy of x

and say “print this.”

• Implication: The more random a system, the greater its AC.– Gell Mann: Shakespeare vs monkeys

• AC is related to entropy:– The entropy rate of a symbolic sequence measures the unpredictability (in

bits per symbol) of the sequence. – For a sequence of n random variables, how does the entropy of the

sequence grow with n?

Shannon Information

• Shannon Entropy H to measure basic information capacity:– For a random variable X with a probability mass function p(x), the entropy of X

is defined as:

– Entropy is measured in bits. – H measures the average uncertainty in the random variable.

• Example 1: – Consider a random variable with uniform distribution over 32 outcomes.– To identify an outcome, we need a label that takes on 32 different values, e.g.,

5-bit strings.

)(log)()( 2 xpxpXH ∑−=

bits 532log321log

321)(log)()(

32

1

32

1

==−=−= ∑ ∑= =i i

ipipXH

• Entropy is the uncertainty of a single random variable.• The entropy of a random variable is a lower bound on the average number of bits

required to represent the variable. • The uncertainty (complexity) of a random variable can be extended to define the

descriptive complexity of a single string.• How much entropy is in the string, or how much information is required to

describe the string?

• Kolmogorov (or algorithmic) complexity is the length of the shortest computer program that prints out the string.

“Algorithmic information theory (AIT) is the result of putting Shannon's information theory and Turing's computability theory into a cocktail shaker and shaking vigorously. The basic idea is to measure the complexity of an object by the size in bits of the smallest program for computing it.” (G. J. Chaitin)

Murray Gell-Mann: Effective Complexity

• Effective complexity is the AIC of a set’s regularities• Mutual Information can be used to determine regularities

Mutual Information

• Measures the amount of information that one random variable contains about another random variable.

– Mutual information is a measure of reduction of uncertainty due to another random variable.

– That is, mutual information measures the dependence between two random variables.

– It is symmetric in X and Y, and is always non-negative.

• Recall: Entropy of a random variable X is H(X).• Conditional entropy of a random variable X given another

random variable Y = H(X | Y).• The mutual information of two random variables X and Y is:

∑=−=yx ypxp

yxpyxpYXHXHYXI, )()(

),(log),()|()(),(

Recap: Effective Complexity• The shortest description of the regularities in an entity (and their probabilities)

– Regularities are found by the mutual information

– Effective complexity is NOT • AIC itself (random strings have high AIC but low complexity)• or Mutual information (0011001100110011 has high MI but low complexity)• but AIC and MI are used to determine Effective complexity

• Exercise: regularities in spoken language

– Give the most concise description you can of regularities in verb tenses and usage• English• Spanish• Hindi (?)• Chinese (dialect?)

• Mutual Information to determine regularities? Context?

English Spanish Chinese Erdu…

Tenses Past--ed, present, future--will, conditional—would,Would haveGerundsSubjunctive—barely used

3 major tensesDefining and asking are different

Subject-verb agreement

1st, 2nd, 3rd person, singular, plural

Classes Ar er ir—7 temporal, 6 subject tenses2 situational tenses

char

Characters Drawn, constructed,

Pronunciation differences

Other Points from Gell-Mann• Schema are representations of the regularities: a ‘package of

information’ describing the regularities in a system, its environment and their interactions’

• The schemata evolve by competing to produce the most predictive description

• How long does the schema take to make predictions?• Logical Depth

– The Logical depth of x is the run time of the shortest program that will cause a UTM to produce x and then halt.

– Must distinguish between effective complexity and logical depth

• The universe has high effective complexity despite simple physical rules because of probabilities & alternative histories

– Frozen accidents produce mutual information or regularities – Most of the effective complexity of the universe lies in the AIC of a

description of those frozen accidents and their consequences

“an individual human being, owes its existence not only to the simple fundamental law of physics and the boundary condition on the early universe but also to the outcomes of an inconceivably long sequence of probabilistic events, each of which could have turned out differently”

Randomness

AlgorithmicC

omplexity

Simplicity: algorithmic regularity, statistical regularity, compressibleFlake, chapter 9

Other Concepts of Complexity

Per Bak: Sandpile model and the ‘edge of chaos’1. A powerlaw size distribution of extinction events: many small and few large2. The powerlaw lifetime distribution of species and genera.3. The powerlaw distribution of species per genus, genera per family, and so on.

Stuart Kauffmann: The Adjacent possible• Autonomous agents have to live the most complex game that they can• There is a tendency for self-constructing biospheres to enlarge their workspace, the

dimensionality of their adjacent possible

John HollandCAS: multiple diverse agents, stimulus response rules and adaptation

– Match environmental conditions at various time scales– Stimulus causes an immediate change in action (response)– Cumulative stimulus causes changes in rules (adaptation, learning, evolution)

Summary of Complexity Measures

• Computational complexity:– How much time (how many resources) does it take to compute something?

• The language/machine hierarchy:– How complex a machine is needed to compute a function?

• Information-theoretic methods:– Entropy– Algorithmic complexity– Mutual information– Gell Mann’s Effective Complexity: The shortest description of the regularities

• Logical depth:– Running time of the shortest program – Combines computational complexity and AIC

• Asymptotic behavior of dynamical systems:– Fixed points, limit cycles, chaos.– Wolfram’s CA classification: the outcome of complex CA can not be predicted any

faster than it can be simulated.

Neither random nor regularComplex systems occur at a transition point between two extremes

Often the only way to predict a future state of a complex system is to simulate it

•It cannot be described by a tractable formula or short program•It cannot be described statistically•Some cellular automata are complex: the results of running them are neither regular nor random

–Langton’s lambda parameter and Wolfram’s CA classification schemes go from regular to chaotic, with ‘complex’ in the middle

Suggested References

• Computational Complexity by Papadimitriou. Addison-Wesley (1994).• Elements of Information Theory by Cover and Thomas. Wiley (1991).• Kaufmann, At Home in the Universe (1996) and Investigations (2002).• Per Bak, How Nature Works: The Science of Self-Organized Criticality

(1988)• Gell-Mann, The Quark and the Jaguar (1994)• Holland, Adaptation in Natural and Artificial Systems, MIT Press; 1992