Springer Theses

Recognizing Outstanding Ph.D. Research

For further volumes:http://www.springer.com/series/8790

Aims and Scope

The series ‘‘Springer Theses’’ brings together a selection of the very best Ph.D.theses from around the world and across the physical sciences. Nominated andendorsed by two recognized specialists, each published volume has been selectedfor its scientific excellence and the high impact of its contents for the pertinentfield of research. For greater accessibility to non-specialists, the published versionsinclude an extended introduction, as well as a foreword by the student’s supervisorexplaining the special relevance of the work for the field. As a whole, the serieswill provide a valuable resource both for newcomers to the research fieldsdescribed, and for other scientists seeking detailed background information onspecial questions. Finally, it provides an accredited documentation of the valuablecontributions made by today’s younger generation of scientists.

Theses are accepted into the series by invited nomination onlyand must fulfill all of the following criteria

• They must be written in good English.• The topic should fall within the confines of Chemistry, Physics, Earth Sciences,

Engineering and related interdisciplinary fields such as Materials, Nanoscience,Chemical Engineering, Complex Systems and Biophysics.

• The work reported in the thesis must represent a significant scientific advance.• If the thesis includes previously published material, permission to reproduce this

must be gained from the respective copyright holder.• They must have been examined and passed during the 12 months prior to

nomination.• Each thesis should include a foreword by the supervisor outlining the signifi-

cance of its content.• The theses should have a clearly defined structure including an introduction

accessible to scientists not expert in that particular field.

Joseph T. Lizier

The Local InformationDynamics of DistributedComputation in ComplexSystems

Doctoral Thesis accepted byThe University of Sydney, Australia

123

AuthorDr. Joseph T. LizierCSIRO ICT CentreMarsfield, NSWAustralia

SupervisorsDr. Mikhail ProkopenkoCSIRO ICT CentreMarsfield, NSWAustralia

Prof. Albert ZomayaSchool of Information TechnologiesThe University of SydneySydney, NSWAustralia

ISSN 2190-5053 ISSN 2190-5061 (electronic)ISBN 978-3-642-32951-7 ISBN 978-3-642-32952-4 (eBook)DOI 10.1007/978-3-642-32952-4Springer Heidelberg New York Dordrecht London

Library of Congress Control Number: 2012945730

� Springer-Verlag Berlin Heidelberg 2013This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part ofthe material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformation storage and retrieval, electronic adaptation, computer software, or by similar or dissimilarmethodology now known or hereafter developed. Exempted from this legal reservation are briefexcerpts in connection with reviews or scholarly analysis or material supplied specifically for thepurpose of being entered and executed on a computer system, for exclusive use by the purchaser of thework. Duplication of this publication or parts thereof is permitted only under the provisions ofthe Copyright Law of the Publisher’s location, in its current version, and permission for use must alwaysbe obtained from Springer. Permissions for use may be obtained through RightsLink at the CopyrightClearance Center. Violations are liable to prosecution under the respective Copyright Law.The use of general descriptive names, registered names, trademarks, service marks, etc. in thispublication does not imply, even in the absence of a specific statement, that such names are exemptfrom the relevant protective laws and regulations and therefore free for general use.While the advice and information in this book are believed to be true and accurate at the date ofpublication, neither the authors nor the editors nor the publisher can accept any legal responsibility forany errors or omissions that may be made. The publisher makes no warranty, express or implied, withrespect to the material contained herein.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)

Parts of this thesis have been published in the following articles:

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘Information transfer by particlesin cellular automata,’’ in Proceedings of the Third Australian Conference onArtificial Life, Gold Coast, Australia, ser. Lecture Notes in Artificial Intelligence,M. Randall, H. A. Abbass, and J. Wiles, Eds., vol. 4828, pp. 49–60. Berlin/Heidelberg: Springer, 2007.

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘Detecting non-trivial computationin complex dynamics,’’ in Proceedings of the 9th European Conference on Arti-ficial Life (ECAL 2007), Lisbon, Portugal, ser. Lecture Notes in Artificial Intelli-gence, F. Almeida e Costa, L. M. Rocha, E. Costa, I. Harvey, and A. Coutinho, Eds.,vol. 4648, pp. 895–904. Berlin/Heidelberg: Springer, 2007.

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘Local information transfer as aspatiotemporal filter for complex systems,’’ Phys. Rev. E, vol. 77, no. 2, p. 026110,2008.

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘The information dynamics ofphase transitions in random Boolean networks,’’ in Proceedings of the EleventhInternational Conference on the Simulation and Synthesis of Living Systems (ALifeXI), Winchester, UK, S. Bullock, J. Noble, R. Watson, and M. A. Bedau, Eds.,pp. 374–381. Cambridge, MA: MIT Press, 2008.

J. T. Lizier, M. Prokopenko, I. Tanev, and A. Y. Zomaya, ‘‘Emergence of glider-like structures in a modular robotic system,’’ in Proceedings of the EleventhInternational Conference on the Simulation and Synthesis of Living Systems (ALifeXI), Winchester, UK, S. Bullock, J. Noble, R. Watson, and M. A. Bedau, Eds.,pp. 366–373. Cambridge, MA: MIT Press, 2008.

J. T. Lizier, J.-D. Haynes, J. Heinzle, and M. Prokopenko, ‘‘Directed informationstructure in inter-regional cortical interactions in a visuomotor tracking task,’’BMC Neuroscience, vol. 10, no. Suppl 1, p. P117, 2009, Proceedings of theEighteenth Annual Computational Neuroscience Meeting Computational Neuro-science 2009 (CNS*2009), Berlin, Germany.

J. T. Lizier, M. Prokopenko, and D. J. Cornforth, ‘‘The information dynamics ofcascading failures in energy networks,’’ in Proceedings of the European Confer-ence on Complex Systems (ECCS), Warwick, UK, p. 54, 2009, ISBN: 978-0-9554123-1-8.

J. T. Lizier and M. Prokopenko, ‘‘Differentiating information transfer and causaleffect,’’ European Physical Journal B, vol. 73, no. 4, pp. 605–615, 2010.

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘Information modification andparticle collisions in distributed computation,’’ Chaos, vol. 20, no. 3, p. 037109,2010.

J. T. Lizier, J. Heinzle, A. Horstmann, J.-D. Haynes, and M. Prokopenko,‘‘Multivariate information-theoretic measures reveal directed information structureand task relevant changes in fMRI connectivity,’’ Journal of ComputationalNeuroscience, vol. 30, no. 1, pp. 85–107, 2011.

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘Coherent information structure incomplex computation,’’ Theory in Biosciences, vol. 131, no. 3, pp. 193–203, 2012.

J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, ‘‘Local measures of informationstorage in complex distributed computation,’’ Information Sciences, vol. 208,pp. 39–54, 2012.

For Amanda,because I still like to impress you

Supervisor’s Foreword

In early 1990s, Chris Langton created the term ‘‘computation at the edge ofchaos’’. In doing so, he not only dissected computation into its three primitivefunctions: the transmission, storage, and modification of information, but alsoopened the way to study computation in the most generic form—information-theoretically. Moreover, a challenge was thrown out: how does one explainemergence of computation in a dynamic setting, and how is it related to complexityof the system in point?

It is a pleasure to introduce Dr. Joseph Lizier’s thesis ‘‘The local informationdynamics of distributed computation in complex systems’’. This work is not shy oftaking on and meeting Langton’s challenge: it goes to the heart of the phenomenonof computation, uncovers and analyses information-theoretic roots of each of theprimitives, and brings these parts back together in a multiplicity of ways, revealingdifferent shapes that coherence and complexity may take.

The dissertation leads us through complex, diverse, and dynamic domains thatshare an intricate fabric of computation: cellular automata (CA), random Booleannetworks, brain’s cortical networks, heart- and breath-rate interactions in sleepapnoea, modular robotic systems, cascading failures in power grids, etc. It revealsinner workings of these systems through their coherent use of memory (‘‘activeinformation storage’’), communications (‘‘information transfer’’), and processing(‘‘information modification’’).

Specifically, the active information storage is introduced, quantifying theinformation storage component that is directly in use in the computation of thenext state of a process. The obtained profiles of local active information storage inCA provide evidence that blinkers and background domains are dominant infor-mation storage processes in these systems, and reveal that local storage maysometimes misinform the observer when the computation is dominated by someother primitives.

Significantly, the work introduces a measure of local information transfer,applied to CA for filtering coherent structures—the measure provides the firstquantitative evidence for the long-held conjecture that particles (both gliders anddomain walls) are the dominant information transfer agents in CA. This is further

ix

developed into a novel method for interregional brain connectivity analysis, usingmultivariate extensions to the mutual information and transfer entropy. Thismethod is distinguished in using asymmetric, multivariate, information-theoreticalanalysis, which captures not only directional and non-linear relationships, but alsocollective interactions.

Bringing storage and transfer together in state-space diagrams, this dissertationdistinguishes complex computation via coherent structure in local informationdynamics profiles, and identifies both clear and ‘‘hidden’’ coherent structure viathe state-space diagrams of the local information dynamics.

Building up on these measures, the study investigates the well-known phasetransition between ordered and chaotic behavior in Random Boolean Networks,demonstrating maximisations in information storage and information transfer oneither side of the critical point—thus, explaining the phase transition in terms ofthe intrinsic distributed computation.

Touching on the subject of causality, the work contrasts transfer entropy andinformation flow, which can be used separately to quantify information transferand causal information flow respectively, and shows that causal information flowis a primary tool to describe the causal structure of a system, while informationtransfer can be used to describe the emergent computation in the system.

Subsequently, information modification is quantified at each spatiotemporalpoint in a system—via separable information, a novel measure which locallyidentifies events where separate inspection of the sources to a computation ismisleading about its outcome (as the points where ‘‘the whole is greater than thesum of the parts’’).

In summary, the work presented in this dissertation forms a new sub-field—‘‘information dynamics’’. It enables a rigorous characterization of diverse com-putation processes on a local spatiotemporal scale within a system, and the resultsprovide important insights into the fundamental nature of distributed computationand the dynamics of complex systems.

After reading this thesis, no longer may one entertain a singular view oncomplexity or hope to measure the latter with a single static measure—instead oneneeds to consider information dynamics within a rich multi-faceted space, whereeach primitive (storage, transfer, and modification) becomes a separate dimension.Thus, complexity acquires shapes and trajectories defined by these axes, whilecoherence can be quantitatively studied via suitably defined state-diagrams.

I have no doubt that this study is at the forefront of the fundamental researchthat will eventually lead to a comprehensive information- and computation-theoreticframework unifying multiple complex physical, biological, social, and technologicalsystems.

Sydney, 7 July 2012 Mikhail Prokopenko

x Supervisor’s Foreword

Acknowledgments

A wise friend recently told me: ‘‘You never finish a Ph.D. thesis, you just decide tostop.’’ Despite the endless frustrations that make research what it is, I have hadsuch a wonderful time during my Ph.D. studies that ‘‘deciding’’ to stop was ratherdifficult. As such, I would like to thank those who made this work possible and soenjoyable.

To my supervisor Mikhail Prokopenko at Australia’s Commonwealth Scientificand Industrial Research Organisation (CSIRO), I thank you first for the web pageof the ‘‘Entropy discussion group’’ that I stumbled across and met you through. Forthe original concept that information transfer in complex systems was a hot topicworth pursuing, and that we should harness it in guided self-organization. For acountless number of interesting and valuable discussions, and your patience withmy decisions (‘‘she’ll be right, mate’’). For your wise and diplomatic strategies,and support in finding funding for my ventures. For insightful feedback on mypapers and these chapters, and mentoring on how to be a scientist. For balancingsupervision and mateship. The debt I owe you is very large indeed.

To my academic supervisor Albert Zomaya at The University of Sydney(USyd), I thank you for always finding room for me in a very hectic schedule.I particularly appreciate your effort in maintaining our ‘‘long-distance relation-ship’’ for much of the time. Thank you for your insightful suggestions, and creativeperspectives on new horizons for us to exploit. For your strong support and trust inmy judgement, and the productive environment of your group. For the motivatingand thoroughly enjoyable consultations, and the many laughs. I am grateful for allthat I have learned from you.

In addition to my supervisors, I thank the others with whom I have collaboratedon research during these years, much of which does not appear in this thesis: IvanTanev, Larry Yaeger, Mahendra Piraveenan, Dany Pradhana, Oliver Obst,X. Rosalind Wang, Peter Wang, Vadim Gerasimov, Jakob Heinzle, John-DylanHaynes, Annette Horstmann, Valerio Sperati, Jürgen Pahle, John Mahoney, OlafBochmann, Gregor Obernosterer, Joseph DeRosa, Ben Mazzotta, Luciano Oviedo,Alexander Shpunt, Carver Tate, David Cornforth, Joschka Bödecker, N. MichaelMayer, and Mikhail Burtsev. I am grateful for the many discussions, including

xi

helpful comments and suggestions on my manuscripts and clarifications of theirown work, from my collaborators above as well as from the following: DanielPolani, Nihat Ay, Ralf Der, Melanie Mitchell, Thomas Schreiber, David Feldman,Cosma Shalizi, Kristian Lindgren, Andre Ribeiro, Alexander Kraskov, PaoloCrucitti, Masatoshi Funabashi, Christian Darabos, Alex Healing, Mikail Rubinov,and Antonio Lafusa. Similarly, I also particularly thank: Selvakennedy Selvaduraifor being an additional associate supervisor in the early days of my candidature;office mates Mahendra Piraveenan, Astrid Zeman, and Khaled Almi’Ani; lunch-mates Oliver Obst and Rosalind Wang; members of CSIRO’s Entropy discussiongroup; the Advanced Network Research Group (ANRG), and DistributedComputing discussion group at USyd; my early scientific mentors Ferg Brand andGraham Town; Terry Dawson and Mary Cudmore for helping put the title‘‘complex systems science’’ to my interest in this area; the anonymous reviewerswhose comments have improved the quality of my work, but in particular DanielPolani, Ralf Der, and David Feldman for the incredibly useful (formal) reviewsthey provided for this thesis; Leontina Di Cecco at Springer for her help inpublishing this thesis, and Michael Wibral for the suggestion and encouragementto do so.

Acknowledged contributions of others here include (specific references areincluded at the relevant chapters): Mirek Wójtowicz’s Java cellebration for sim-ulating and plotting cellular automata (e.g. in Fig. 3.4); Carlos Gershenson’sRBNLab for simulating random Boolean networks; Ivan Tanev’s original snakebotand genetic programming implementation; Jakob Heinzle, John-Dylan Haynes andAnnette Horstmann from the Bernstein Center for Computational Neuroscience inBerlin provided the fMRI data set analysed in Sect. 8.2 (these authors conceivedand performed the original experiment to obtain the data, obtained informedwritten consent from the subjects, and pre-processed the data as described in Sect.8.2.3.1); Jean-Baptiste Rouquier’s CimulA for analysing statistical complexity inCAs and plotting Fig. 2.2; Cytoscape for generating Figs. 8.3 and 8.4.

For financial support, I primarily thank: The University of Sydney for the awardof the A.E. and F.A.Q. Stephens scholarship, and the Stephens family for thebequests that made this possible; and CSIRO for their Postgraduate Award, as atop-up scholarship. This financial support enabled me to study full-time. I wouldlike to thank various sources of travel funding, the use of which has had a sig-nificant impact on the maturity of my work: CSIRO for funding via the Post-graduate Award, the Centre for Complex Systems Science, and the EnergyTransformed flagship; the USyd under the Postgraduate Research Support Scheme(PRSS); the Australian Research Council’s Complex Open Systems Network(COSNet); the Santa Fe Institute through NSF Grant No. 0200500 entitled‘‘A Broad Research Program in the Sciences of Complexity’’; and the organizersof ALifeXI for a travel bursary. In terms of resources, I thank CSIRO’s Infor-mation and Communication Technologies (ICT) Centre for the laptop and otherworkspace arrangements in hosting me, and CSIRO’s Advanced Scientific Com-puting group for access to high performance computing resources (the burnet

xii Acknowledgments

cluster) used for simulation and analysis in this work. I also thank the staff ofSchool of Information Technologies at USyd for facilitating my studies.

On the personal front, I thank my parents Pamela and George for theupbringing, education, sacrifices, and, along with my sister Sally, for the envi-ronment and encouragement that made me the person who could pull this off.

And of course my enormous gratitude goes to my wife and inspiration Amanda.First for the encouragement in continually asking me when I was going to enrol inmy Ph.D. The first time I truly believed I would do it was when you started tellingother people that was the plan, and that changed everything. For your support andstability, both emotional and financial. For your patience, and for more hours ofproof-reading about this information thing than you deserved. Most of all forsharing these fabulous years with me. I would not trade a minute of it. I only hopethat I can make as much of a contribution to your Ph.D. studies as you have madeto mine.

Acknowledgments xiii

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Hypothesis and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Contributions of this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Computation in Complex Systems. . . . . . . . . . . . . . . . . . . . . . . . . 132.1 Complex Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.1 Order, Disorder and Phase Transitions. . . . . . . . . . . . . . 152.1.2 Self-Organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.1.3 Motivation for Studying Complex Systems . . . . . . . . . . 17

2.2 Information theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.2.1 Information-Theoretic Measures . . . . . . . . . . . . . . . . . . 182.2.2 Localising Information-Theoretical Measures . . . . . . . . . 222.2.3 Information-Theoretic Measures

of Continuous Variables. . . . . . . . . . . . . . . . . . . . . . . . 242.2.4 Reasons for Application to Complex Systems. . . . . . . . . 25

2.3 Cellular Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.3.1 Functionality of Cellular Automata . . . . . . . . . . . . . . . . 292.3.2 Complex Behaviour in Cellular Automata . . . . . . . . . . . 302.3.3 Computation in Cellular Automata . . . . . . . . . . . . . . . . 322.3.4 Examples of Distributed Computation in CAs . . . . . . . . 352.3.5 Filtering Structure in Cellular Automata . . . . . . . . . . . . 37

2.4 The Dynamics of Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 382.4.1 Random Boolean Networks as a Model

of Dynamic Network Behavior . . . . . . . . . . . . . . . . . . . 402.5 Guided Self-Organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.6 Opportunity to Quantify the Information Dynamics

of Distributed Computation. . . . . . . . . . . . . . . . . . . . . . . . . . . 45References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

xv

3 Information Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.1 Excess Entropy as Total Information Storage . . . . . . . . . . . . . . 54

3.1.1 Single-Agent and Collective Excess Entropy . . . . . . . . . 543.1.2 Local Excess Entropy . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.2 Active Information Storage as Storage in Use . . . . . . . . . . . . . . 573.2.1 Local Active Information Storage . . . . . . . . . . . . . . . . . 583.2.2 Active Information Storage and Entropy Rate . . . . . . . . 59

3.3 Local Information Storage in Cellular Automata . . . . . . . . . . . . 613.3.1 Appropriate History Lengths . . . . . . . . . . . . . . . . . . . . 623.3.2 Periodic Blinker and Domain Processes

as Dominant Storage . . . . . . . . . . . . . . . . . . . . . . . . . . 633.3.3 Negative Informative Storage as Misinformation

at Particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683.3.4 Particles Create New Information Storage . . . . . . . . . . . 693.3.5 Structured Information Storage in Domain of Rule 18 . . . 703.3.6 Misinformation and New Storage Creation

by Domain Walls . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.3.7 Local Temporal Entropy Rate Highlights

Moving Particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723.3.8 Absence of Coherent Information Storage Structure . . . . 72

3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4 Information Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.1 Transfer Entropy as Predictive Information Transfer . . . . . . . . . 81

4.1.1 Transfer Entropy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814.1.2 Local Transfer Entropy . . . . . . . . . . . . . . . . . . . . . . . . 844.1.3 Apparent, Conditional and Complete

Transfer Entropy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.1.4 Total Information Composition and Collective

Information Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . 894.2 Local Information Transfer in Cellular Automata . . . . . . . . . . . 93

4.2.1 Inadequate Measures for Information Transfer . . . . . . . . 934.2.2 Particles as Dominant, Coherent Information

Transfer Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . 954.2.3 Ambient Transfer in Backgrounds Domains . . . . . . . . . . 964.2.4 Apparent and Complete Transfer

Entropy are Complementary . . . . . . . . . . . . . . . . . . . . . 984.3 Information Flow as Causal Effect. . . . . . . . . . . . . . . . . . . . . . 101

4.3.1 Information Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.3.2 Local Information Flow . . . . . . . . . . . . . . . . . . . . . . . . 104

4.4 Local Causal Information Flow in Cellular Automata . . . . . . . . 1044.4.1 Information Transfer, Causal Flow

and Emergent Structures . . . . . . . . . . . . . . . . . . . . . . . 106

xvi Contents

4.4.2 Information Transfer to be Measuredfrom Causal Sources Only . . . . . . . . . . . . . . . . . . . . . . 107

4.4.3 Complete Transfer Entropy as an Inferrerfor Information Flow . . . . . . . . . . . . . . . . . . . . . . . . . . 108

4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

5 Information Modification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1175.1 Separable Information as a Detector for Non-Trivial

Information Modification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.2 Local Information Modification in Cellular Automata . . . . . . . . 122

5.2.1 Hard Particle Collisions as DominantModification Events . . . . . . . . . . . . . . . . . . . . . . . . . . 123

5.2.2 Soft Collisions Between Gliders and the Domain . . . . . . 1255.2.3 Storage Modifications in Non-Periodic Domains. . . . . . . 1255.2.4 Proliferation of Information Modification

in Chaotic Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . 1265.2.5 Modification Only Understood in Context

of Past History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1265.3 Irreversibly Destroyed Information . . . . . . . . . . . . . . . . . . . . . 127

5.3.1 Measuring Information Destructionin Distributed Computation . . . . . . . . . . . . . . . . . . . . . 128

5.3.2 Irreversible Information Destructionin Cellular Automata . . . . . . . . . . . . . . . . . . . . . . . . . . 132

5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

6 Information Dynamics in Networksand Phase Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1416.1 Phase Transitions in Random Boolean Networks . . . . . . . . . . . . 143

6.1.1 Experimental Details . . . . . . . . . . . . . . . . . . . . . . . . . . 1436.1.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 145

6.2 Cascading Failures in Power Grids . . . . . . . . . . . . . . . . . . . . . 1496.2.1 Cascading Failures Model . . . . . . . . . . . . . . . . . . . . . . 1506.2.2 Measuring Information Dynamics

in Cascading Failures . . . . . . . . . . . . . . . . . . . . . . . . . 1516.2.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 152

6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

7 Coherent Information Structure in Complex Computation . . . . . . 1637.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1637.2 Local Information Dynamics State-Space . . . . . . . . . . . . . . . . . 1667.3 Measuring Coherent Information Structure in the State-Space . . . 169

Contents xvii

7.3.1 Coherent Information StructureMeasurements in CAs . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.3.2 Coherent Information StructureMeasurements in RBNs . . . . . . . . . . . . . . . . . . . . . . . . 171

7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

8 Information Transfer in Biological and Bio-Inspired Systems . . . . 1778.1 Heart and Breath Rate Interaction in Sleep Apnea. . . . . . . . . . . 1788.2 Establishing Directed Interregional Cortical

Information Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1798.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1808.2.2 Interregional Information Structure

Analysis Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . 1828.2.3 Application to fMRI Experimental Data . . . . . . . . . . . . 1878.2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

8.3 Evolution of Coherent Information Transfer Structure . . . . . . . . 1918.3.1 Evolving the Snakebot for Maximum

Information Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . 1938.3.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 1958.3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2039.1 Summary of Main Contributions . . . . . . . . . . . . . . . . . . . . . . . 203

9.1.1 Framework for the Information Dynamicsof Distributed Computation . . . . . . . . . . . . . . . . . . . . . 203

9.1.2 Measuring Information Storage. . . . . . . . . . . . . . . . . . . 2049.1.3 Measuring Information Transfer . . . . . . . . . . . . . . . . . . 2049.1.4 Measuring Information Modification . . . . . . . . . . . . . . . 2059.1.5 Quantitative Understanding of Information

Dynamics in CAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2059.1.6 Measuring Computational Properties in Phase

Transitions in Networks . . . . . . . . . . . . . . . . . . . . . . . . 2069.1.7 Methodology for Studying Coherent

Information Structure. . . . . . . . . . . . . . . . . . . . . . . . . . 2069.1.8 Demonstrated Application Areas

for Information Dynamics . . . . . . . . . . . . . . . . . . . . . . 2079.2 Directions for Future Work. . . . . . . . . . . . . . . . . . . . . . . . . . . 207References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

xviii Contents

Appendix A: Consideration of Alternative Method of Localisation . . . 213

Appendix B: Entropy Rate Convergence and DivergentExcess Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Appendix C: Relation of Transfer Entropy to Massey’sDirected Information. . . . . . . . . . . . . . . . . . . . . . . . . . . 219

Appendix D: Back-Door Adjustment . . . . . . . . . . . . . . . . . . . . . . . . . 221

Appendix E: Complete Transfer Entropy for CausalStructure Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

Appendix F: Information Destruction Only Measured in OpenComputational Systems . . . . . . . . . . . . . . . . . . . . . . . . . 225

Appendix G: Circumstantial Evidence of Maximum Coherencein Complex Computation . . . . . . . . . . . . . . . . . . . . . . . 227

Author Biography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

Contents xix

Symbols

Typefaces

X; Y ; Z Variable namesx; y; z Specific values taken by the variables X; Y; Zx; y; z Values imposed on the variables X; Y; ZX;Y;Z Sets of variablesx; y; z Values to the sets X;Y;Z

Notation

xn Value of variable X at time nxi;n Value of variable Xi at time n in a spatially-extended system Xxn Joint value of a spatially extended system X at time n

xðkÞnJoint value of the past k values of variable x, up to and includingtime n

xðkþÞnJoint value of the future k values of variable x, from time n onwards

pðaÞ Probability of the event apða; bÞ Probability of the joint event a ^ bpðaÞ Probability of the joint event apða j bÞ Probability of the event a given event bprðaÞ Probability of the continuous value a, estimated using parameter rHX An example average information-theoretic measure on variable XhXðn; :::Þ An example local information-theoretic measure on variable X at

time n (lower-case by convention)hðxn; :::Þ Alternate notation for a local information-theoretic measure on

variable X at time n (lower-case by convention)hði; n; :::Þ An example local information-theoretic measure in a spatially-

extendedsystem,onvariablexi at timen (lower-casebyconvention)

xxi

xnh in Expectation value of xn, averaged over all nVX Causal parents of variable XVY

X Causal parents of variable X except for variable Y and X itself

Information-Theoretic Measures

The following table indicates the notation for several average information-theoretical measures used (both existing and introduced measures). The notationfor their local values takes the lower-case of the same variable name, as indicatedin the list of notation above.

HX Shannon entropy of variable XHX;Y Joint entropy of X and YHXjY Conditional entropy of X given YIX;Y Mutual information of X and YIX Multi-information of XIX;Y jZ Conditional mutual information of X and Y given ZHlX Entropy rate of XEX Excess entropy of XAX Active information storage of XTY!X (Apparent) Transfer entropy from Y to XTY!XjZ Conditional transfer entropy from Y to X given ZTc

Y!X Complete transfer entropy from Y to XTX Collective transfer entropy to X

IpðY ! X j SÞ Information flow from Y to X imposing S

UX Intrinsic uncertainty of XSX Separable information of XDX Information destruction at XIssX Multi-information in the state-space of X

Abbreviations

1D One-dimensionalALife Artificial LifeBOLD Blood Oxygen Level-Dependent (time-series)CA Cellular AutomataCRBN Classical Random Boolean NetworkECA Elementary Cellular AutomataEEG ElectroencephalogramfMRI Functional Magnetic Resonance ImagingGRN Gene Regulatory NetworkMI Mutual Information

xxii Symbols

PDF Probability Distribution FunctionRBN Random Boolean NetworkRTZ Return To Zero (bit operation)TE Transfer EntropyXOR Exclusive OR (bit operation)

Symbols xxiii