BioMed CentralBMC Neuroscience

ss

Open AcceResearch articleMeasuring information integrationGiulio Tononi*1 and Olaf Sporns2

Address: 1Department of Psychiatry, University of Wisconsin, Madison, USA and 2Department of Psychology, Indiana University, Bloomington, USA

Email: Giulio Tononi* - gtononi@wisc.edu; Olaf Sporns - osporns@indiana.edu

* Corresponding author

AbstractBackground: To understand the functioning of distributed networks such as the brain, it isimportant to characterize their ability to integrate information. The paper considers a measurebased on effective information, a quantity capturing all causal interactions that can occur betweentwo parts of a system.

Results: The capacity to integrate information, or Φ, is given by the minimum amount of effectiveinformation that can be exchanged between two complementary parts of a subset. It is shown thatthis measure can be used to identify the subsets of a system that can integrate information, orcomplexes. The analysis is applied to idealized neural systems that differ in the organization of theirconnections. The results indicate that Φ is maximized by having each element develop a differentconnection pattern with the rest of the complex (functional specialization) while ensuring that alarge amount of information can be exchanged across any bipartition of the network (functionalintegration).

Conclusion: Based on this analysis, the connectional organization of certain neural architectures,such as the thalamocortical system, are well suited to information integration, while that of others,such as the cerebellum, are not, with significant functional consequences. The proposed analysis ofinformation integration should be applicable to other systems and networks.

BackgroundA standard concern of communication theory is assessinginformation transmission between a sender and a receiverthrough a channel [1,2]. Such an approach has been suc-cessfully employed in many areas, including neuroscience[3]. For example, by estimating the probability distribu-tion of sensory inputs, one can show that peripheral sen-sory pathways are well suited to transmitting informationto the central nervous system [4]. When considering thecentral nervous system itself, however, we face the issue ofinformation integration [5]. In such a distributed network,any combination of neural elements can be viewed assenders or receivers. Moreover, the goal is not just to trans-

mit information, but rather to combine many sources ofinformation within the network to obtain a unified pic-ture of the environment and control behavior in a coher-ent manner [6,7]. Thus, while a neural system composedof a set of parallel, independent channels or reflex arcsmay be extremely efficient for information transmissionbetween separate inputs and outputs, it would be unsuit-able for controlling behaviors requiring global access,context-sensitivity and flexibility. A system equipped withforward, backward and lateral connections among spe-cialized elements can perform much better by supportingbottom-up, top-down, and cross-modal interactions andforming associations across different domains [8].

Published: 02 December 2003

BMC Neuroscience 2003, 4:31

Received: 26 August 2003Accepted: 02 December 2003

This article is available from: http://www.biomedcentral.com/1471-2202/4/31

© 2003 Tononi and Sporns; licensee BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.

Page 1 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

The requirements for information integration are perhapsbest exemplified by the organization of the thalamocorti-cal system of vertebrates. On the "information" side, thereis overwhelming evidence for functional specialization,whereby different neural elements are activated in differ-ent circumstances, at multiple spatial scales [9]. Thus, thecerebral cortex is subdivided into systems dealing withdifferent functions, such as vision, audition, motor con-trol, planning, and many others. Each system in turn issubdivided into specialized areas, for example differentvisual areas are activated by shape, color, and motion.Within an area, different groups of neurons are furtherspecialized, e.g. by responding to different directions ofmotion. On the "integration" side, the information con-veyed by the activity of specialized groups of neuronsmust be combined, also at multiple spatial and temporalscales, to generate a multimodal model of the environ-ment. For example, individual visual elements such asedges are grouped together to yield shapes according toGestalt laws. Different attributes (shape, color, location,size) must be bound together to form objects, and multi-ple objects coexist within a single visual image. Imagesthemselves are integrated with auditory, somatosensory,and proprioceptive inputs to yield a coherent, unifiedconscious scene [10,11]. Ample evidence indicates thatneural integration is mediated by an extended network ofintra- and inter-areal connections, resulting in the rapidsynchronization of neuronal activity within and betweenareas [12-17].

While it is clear that the brain must be extremely good atintegrating information, what exactly is meant by such acapacity, and how can it be assessed? Surprisingly,although tools for measuring the capacity to transmit,encode, or store information are well developed [2], waysof defining and assessing the capacity of a system to inte-grate information have rarely been considered. Severalapproaches could be followed to make the notion ofinformation integration operational [5]. In this paper, weconsider a measure based on effective information, aquantity capturing all causal interactions that can occurbetween two parts of a system [18]. Specifically, the capac-ity to integrate information is called Φ, and is given by theminimum amount of effective information that can beexchanged across a bipartition of a subset. We show thatthis measure can be used to identify the subsets of a sys-tem that can integrate information, which are called com-plexes. We then apply this analysis to idealized neuralsystems that differ in the organization of their connec-tions. These examples provide an initial appreciation ofhow information integration can be measured and how itvaries with different neural architectures. The proposedanalysis of information integration may also be applica-ble to other non-neural systems and networks.

TheoryAs in previous work, we consider an isolated system Xwith n elements whose activity is described by a Gaussianstationary multidimensional stochastic process [19]. Herewe focus on the intrinsic properties of a neural system andhence do not consider extrinsic inputs from the environ-ment. The elements could be thought of as groups of neu-rons, the activity variables as firing rates of such groupsover hundreds of milliseconds, and the interactionsamong them as mediated through an anatomical connec-tivity CON(X). The joint probability density functiondescribing such a multivariate process can be character-ized [2,20] in terms of entropy (H) and mutual informa-tion (MI).

Effective informationConsider a subset S of elements taken from the system.We want to measure the information generated when Senters a particular state out of its repertoire, but only tothe extent that such information can be integrated, i.e. itcan result from causal interactions within the system. Todo so, we partition S into A and its complement B (B = S-A). The partition of S into two disjoint sets A and B whoseunion is S is indicated as [A:B]S. We then give maximumentropy to the outputs from A, i.e. substitute its elementswith independent noise sources of constrained maximumvariance. Finally, we determine the entropy of the result-ing responses of B (Fig. 1). In this way we define the effec-tive information from A to B as:

EI(A→B) = MI(AHmax:B) (1)

where MI(A:B) = H(A) + H(B) - H(AB) stands for mutualinformation, the standard measure of the entropy orinformation shared between a source (A) and a target (B).Since A is substituted by independent noise sources, theentropy that B shares with A is due to causal effects of A onB. In neural terms, we try out all possible combinations offiring patterns as outputs from A, and establish how dif-ferentiated is the repertoire of firing patterns they producein B. Thus, if the connections between A and B are strongand specialized, different outputs from A will produce dif-ferent firing patterns in B, and EI(A→B) will be high. Onthe other hand, if the connections between A and B aresuch that different outputs from A produce scant effects,or if the effect is always the same, then EI(A→B) will below or zero. Note that, unlike measures of statisticaldependence, effective information measures causal inter-actions and requires perturbing the outputs from A. More-over, by enforcing independent noise sources in A,effective information measures all possible effects of A onB, not just those that are observed if the system were leftto itself. Also, EI(A→B) and EI(B→A) will generally differ,i.e. effective information is not symmetric.

Page 2 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

For a given bipartition [A:B]S of subset S, the effectiveinformation is the sum of the effective information forboth directions:

EI(A B) = EI(A→B) + EI(B→A) (2)

Information integrationBased on the notion of effective information for a biparti-tion, we can assess how much information can be inte-grated within a system of elements. To this end, we notethat a subset S of elements cannot integrate any informa-tion if there is a way to partition S in two complementaryparts A and B such that EI(A B) = 0. In such a case wewould be dealing with two (or more) causally independ-

ent subsets, rather than with a single, integrated subset.More generally, to measure how much information can beintegrated within a subset S, we search for the biparti-tion(s) [A:B]S for which EI(A B) reaches a minimum.

EI(A B) is necessarily bounded by the maximumentropy available to A or B, whichever is less. Thus, to becomparable over bipartitions, min{EI(A B)} is normal-ized by min{Hmax(A), Hmax(B)}. Thus, the minimum infor-mation bipartition of subset S, or MIB(S), is its bipartitionfor which the normalized effective information reaches aminimum, corresponding to:

Schematics of effective informationFigure 1Schematics of effective information. Shown is a single subset S (grey ellipse) forming part of a larger system X. This sub-set is bisected into A and B by a bipartition (dotted line). Arrows indicate anatomical connections linking A to B and B to A across the bipartition, as well as linking both A and B to the rest of the system X. Bi-directional arrows indicate intrinsic con-nections within each subset and within the rest of the system. (Left) All connections are present. (Right) To measure EI(A→B), maximal entropy Hmax is injected into the outgoing connections from A (see Eq. 1). The resulting entropy of the states of B is then measured. Note that A can affect B directly through connections linking the two subsets, as well as indirectly via X.

X

SA B

X

SA B

Hmax

Page 3 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

MIB(S) = [A:B]S for which EI(A B)/(min{Hmax(A),Hmax(B)}) = min for all A (3)

The capacity for information integration of subset S, or

Φ(S), is simply the value of EI(A B) for the minimuminformation bipartition:

Φ(S) = EI(MIB(S)) (4)

The Greek letter Φ is meant to indicate the information(the "I" in Φ) that can be integrated within a single entity(the "O" in Φ). This quantity is also called MIBcomplexity,for minimum information bipartition complexity[18,21].

ComplexesIf Φ(S) is calculated for every possible subset S of a system,one can establish which subsets are actually capable ofintegrating information, and how much of it. Considerthus every possible subset S of m elements out of the n ele-ments of a system, starting with subsets of two elements(k = 2) and ending with a subset corresponding to theentire system (k=n). A subset S with Φ>0 is called a com-plex if it is not included within a subset having higher Φ[18,21]. For a given system, the complex with the maxi-mum value of Φ(S) is called the main complex, where themaximum is taken over all combinations of k>1 out of nelements of the system.

main complex (X) = S ⊆ X for which Φ(S) = max for all S(5)

In summary, a system can be analyzed to identify its com-plexes – those subsets of elements that can integrate infor-mation among themselves, and each complex will have anassociated value of Φ – the amount of information it canintegrate.

ResultsIn order to identify complexes and their Φ(S) for systemswith many different connection patterns, we imple-mented each system X as a stationary multidimensionalGaussian process such that values for effective informa-tion could be obtained analytically (see Methods). Inorder to compare different connection patterns, the con-nection matrix CON(X) was normalized so that the abso-lute value of the sum of the afferent synaptic weights perelement corresponded to a constant value w<1 (here,unless specified otherwise, w = 0.5). When considering abipartition [A:B]S, the independent Gaussian noiseapplied to A was multiplied by cp, the perturbation coeffi-cient, while the independent Gaussian noise applied tothe rest of the system was given by ci, the intrinsic noisecoefficient. By varying cp and ci one can vary the signal-to-noise ratio (SNR) of the effects of A on the rest of the sys-

tem. In the following examples, except if specified other-wise, we set cp = 1 and ci = 0.00001 in order to emphasizethe role of the connectivity and minimize that of noise.

An illustrative exampleWe first consider a system for which the results of thesearch for complexes should be easy to predict based onthe anatomical connectivity. The system comprises 8 ele-ments with a connection matrix corresponding to twofully interconnected modules, consisting of elements 1–4and 5–7, respectively. These modules are completely dis-connected from each other except for receiving commoninput from element 8 (Fig. 2A,2G). We derived the sys-tem's covariance matrix (Fig. 2B) based on the linearmodel and performed an exhaustive search for complexesas described in the Methods section. The search for mini-mum information bipartitions proceeded over subsets ofsizes k = 2,...,8, with a total of 247 individual subsetsbeing examined. The main output is a list of all the subsetsS of system X ranked by how much information can beintegrated by each of them, i.e. their values of Φ. In Fig.2C, the top 25 Φ values are ranked in descending order,regardless of the size of the corresponding subset. Thecomposition of each subset corresponding to each Φvalue is schematically represented in Fig. 2D, with differ-ent grey tones identifying the minimum informationbipartition. The most relevant information is contained atthe top of the matrix, which identifies the subsets withhighest Φ. The subset at the top of Fig. 2C,2D consists ofelements {1,2,3,4} and has Φ = 20.7954. Its minimuminformation bipartition (MIB) splits it between elements{1,2} and {3,4}. The next subsets consist of various com-binations of elements. However, many of them areincluded within a subset having higher Φ (higher up inthe list) and are stripped from the list of complexes, whichis shown in Fig. 2F. Fig. 2E shows the ranked values of Φfor each complex. In this illustrative example, our analysisrevealed the existence of three complexes in the system:the one of highest Φ, corresponding to elements{1,2,3,4}, the second-highest, corresponding to elements{5,6,7} with Φ = 20.1023, and a complex correspondingto the entire system (elements {1,2,3,4,5,6,7,8}) with amuch lower value of Φ = 7.4021. As is evident from thisexample, the same elements can be part of differentcomplexes.

Optimal networks for information integrationThe above example illustrates how, in a simple case, theanalysis of complexes correctly identifies subsets of a sys-tem that can integrate information. In general, however,establishing the relationship between the anatomical con-nectivity of a network and its ability to integrate informa-tion is more difficult. For example, it is not obvious whichanatomical connection patterns are optimal for

Page 4 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

information integration, given certain constraints onnumber of elements, total amount of synaptic input perelement, and SNR.

Fig. 3 shows connection patterns and complexes obtainedfrom a representative example of nonlinear constrainedoptimization of the connection matrix CON(X), startingfrom random connection matrices (n = 8, all columns

normalized to w = 0.5, no self-connections, and highSNR, corresponding to ci = 0.00001, cp = 1). A largenumber of optimizations for Φ were carried out and pro-duced networks having high values of information inte-gration (Φ = 73.6039 ± 0.5352 for 343 runs) that sharedseveral structural features. In all cases examined, com-plexes with highest (optimized) values of Φ were of max-imal size n. After optimization, connection weights

Measuring information integration: An illustrative exampleFigure 2Measuring information integration: An illustrative example. To measure information integration, we performed an exhaustive search of all subsets and bipartitions for a system of n = 8 elements. Noise levels were ci = 0.00001, cp = 1. (A) Con-nection matrix CON(X). Connections linking elements 1 to 8 are plotted as a matrix of connection strengths (column ele-ments = targets, row elements = sources). Connection strength is proportional to grey level (dark = strong connection, light = weak or absent connection). (B) Covariance matrix COV(X). Covariance is indicated for elements 1 to 8 (corresponding to A). (C) Ranking of the top 25 values for Φ. (D) Element composition of subsets for the top 25 values of Φ (corresponding to panel C). Elements forming the subset S are indicated in grey, with two shades of grey indicating the bipartition into A and B across which the minimal value for EI was obtained. (E) Ranking of the Φ values for all complexes, i.e. subsets not included within sub-sets of higher Φ. (F) Element composition for the complexes ranked in panel E. (G) Digraph representation of the connections of system X (compare to panel A). Elements are numbered 1 to 8, arrows indicate directed edges, arrow weight indicates con-nection strength. Grey overlays indicate complexes with grey level proportional to their value of Φ. Figs. 2 to 7 use the same layout to represent computational results.

1 2

34

5

6

7

8

A

B

C D

E F

G

Φ(subset)

Φ(complex)

Page 5 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

showed a characteristic distribution with a large propor-tion of weak connections and a gradual fall-off towardsconnections with higher strength ("fat tail"). The columnvectors of optimized connection matrices were highlydecorrelated, as indicated by an average angle betweenvectors of 63.6139 ± 2.5261° (343 optimized networks).Thus, the pattern of synaptic inputs were different for dif-ferent elements, ensuring heterogeneity and specializa-tion. On the other hand, all optimized connectionmatrices were strongly connected (all elements could bereached from all other elements of the network), ensuringinformation integration across all elements of the system.

Under conditions of low SNR (ci = 0.1, cp = 1), optimizedvalues of Φ (albeit much lower) were again obtained forcomplexes of size n (Fig. 4, Φ = 5.7454 ± 0.1189 for 21

runs). However, the optimized connection patterns haddifferent weight distributions and topologies compared tothose obtained with high SNR. Connection weightsshowed a highly bimodal distribution, with most connec-tions having decayed effectively to strengths of zero and afew connections remaining at high strength, correspond-ing to a sparse connection pattern. Again, decorrelation ofsynaptic input vectors was high (78.9453 ± 1.3946° for 21optimized networks), while at the same time networksremained strongly connected.

In order to characterize optimal networks for informationintegration in graph-theoretical terms, we examinedsparse networks with fixed numbers of connections ofequal strength. Starting from networks of size n = 8 having16 connections arranged at random, we used an

Information integration and complexes for an optimized network at high SNRFigure 3Information integration and complexes for an optimized network at high SNR. Shown is a representative example for n = 8, w = 0.5, ci = 0.00001, cp = 1. Note the heterogeneous arrangement of the incoming and outgoing connections for each element.

A

B

C D

E F

G

1

2

3

45

6

7

8

Φ(subset)

Φ(complex)

Page 6 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

evolutionary algorithm [22] to rearrange the connectionsfor optimal information integration. Also in this case,optimal networks yielded complexes corresponding to theentire system, both for high (ci = 0.00001, Φ = 60.7598,Fig. 5) and low levels of SNR (ci = 0.1, Φ = 5.943, notshown). Optimized sparse networks shared several struc-tural characteristics that emerged in the absence of anystructural constraints imposed on the evolutionary algo-rithm. As with non-sparse networks, heterogeneity of theconnection pattern was high, with no two elements shar-ing sets of either inputs or outputs. To quantify the simi-larity of input patterns, we calculated the matching index[23,24], which reflects the correlation (scaled between 0and 1) between discrete vectors of input connections to apair of vertices, excluding self- and cross-connections.After averaging across all pairs of elements of each opti-mized network, the average matching index was 0.1429,

indicating extremely low overlap between input patterns.Also, despite the sparse connectivity, all optimized net-works were strongly connected. In-degree and out-degreedistributions, which record the number of afferent andefferent connections per element, tended to be balanced,with most if not all elements receiving and emitting twoconnections each (see example in Fig. 5). Consistent withresults from nonlinear optimization runs, networksshowed few (if any) direct reciprocal connections but hadan over-abundance of short cycles. For example, an opti-mal pattern having maximal symmetry was made up bynested cycles of length 3 (3-cycles, or indirect reciprocal,as shown in Fig. 10). Finally, structural characteristics ofoptimized sparse networks differed significantly fromthose of random sparse networks. The latter had a muchlower value of Φ (Φ = 35.6622 ± 5.0382, 100 exemplars)and complexes of maximal size n = 8 were found only in

Information integration and complexes for an optimized network at low SNRFigure 4Information integration and complexes for an optimized network at low SNR. Shown is a representative example for n = 8, w = 0.5, ci = 0.1, cp = 1. Note the sparse structure of the optimized connection pattern.

A

B

C D

E F

G

1

2

3

45

6

7

8

Φ(subset)

Φ(complex)

Page 7 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

a fraction (46/100) of cases. Calculation of the matchingindex revealed a higher degree of similarity between inputvectors (average of 0.2180), despite random generation.Moreover, only a small number of random networks werestrongly connected (13/100) and the fraction of directreciprocal connections was higher.

In summary, all optimized networks, irrespective of size,sparse or full connectivity, and levels of noise, satisfiedtwo opposing requirements – that for specialization andthat for integration. The former corresponded to thestrong tendency of connection patterns of different ele-ments to be as heterogeneous as possible, yielding ele-ments with different functional roles. The latter wasrevealed by the fact that optimized networks, in all cases

examined in this study and for any combination ofparameters or constraints, formed a single, highly con-nected complex of maximal size.

Homogeneous and modular networksTo examine the influence of homogeneity and modularityof connection patterns on the integration of information,it is useful to study how values of Φ respond to changes inthese parameters. Specifically, if heterogeneity (specializa-tion) is essential, Φ should decrease considerably for con-nection matrices that lack such heterogeneity, i.e. are fullyhomogeneous. Fig. 6 shows a fully connected networkwhere all connection weights had the same value CONij =0.072, i.e. a fully homogenous network (matching index =1). Its main complex contained all elements of the system,

Information integration and complexes for an optimized sparse network having fixed number of connections of equal strengthFigure 5Information integration and complexes for an optimized sparse network having fixed number of connections of equal strength. The network was obtained through an evolutionary rewiring algorithm (n = 8, 16 connections of weight 0.25 each, ci = 0.00001, cp = 1). Note the heterogeneous arrangement of the incoming and outgoing connections for each ele-ment, the balanced degree distribution (two afferent and two efferent connections per element) and the low number of direct reciprocal connections.

A

B

C D

E F

G

1

2

3

45

6

7

8

Φ(subset)

Φ(complex)

Page 8 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

but its ability to integrate information was much less thanthat of optimized networks (Φ = 20.5203 against 73.6039,for n = 8 and ci = 0.00001). This result was obtained irre-spective of the size of the system or SNR.

On the other hand, if forming a single large complex(integration) is essential to reach high values of Φ, then asystem that is composed of several smaller complexes (e.g."modules") should integrate less information and havelower values of Φ. Fig. 7 shows a strongly modular net-work, consisting of four modules of two strongly intercon-nected elements (CONij = 0.25) that were weakly linkedby inter-module connections (CONij = 0.0417). This net-work yielded Φ = 20.3611 for each of its four modules,which formed the system's four main complexes. A com-plex consisting of the entire network was also obtained,

but its Φ value (19.4423) was lower than that of the indi-vidual modules.

Finally, to evaluate how Φ changes by varying parametri-cally along the continuum between full modularity to fullhomogeneity, we plotted the value of Φ for a series ofToeplitz connection matrices of Gaussian form forincreasing standard deviation σ (Fig. 8A,8B; Toeplitzmatrices have constant coefficients along all subdiago-nals). As σ was varied from 10-3 (complete modularity,self-connections only) to 101 (complete homogeneity, allconnections of equal weight), Φ reached a maximum forintermediate values of σ, when the coefficients in thematrix spanned the entire range between 0 and 1.

Information integration and complexes for a homogeneous networkFigure 6Information integration and complexes for a homogeneous network. Connectivity is full and all connections weights are the same (n = 8, w = 0.5, ci = 0.00001, cp = 1).

A

B

C D

E F

G

1

2

3

45

6

7

8

Φ(subset)

Φ(complex)

Page 9 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

Basic digraphs and their ΦIn this section, we examine the capacity to integrate infor-mation of a few basic digraphs (directed graphs) that con-stitute frequent motifs in biological systems, especially inneural connection patterns (Fig. 9). A directed path (Fig.9A) consists of a series of directed connections (edges)linking elements (vertices) such that no element isrepeated. A directed path composed of connections ofequal weight linking n elements formed a complex of sizen with a low value of Φ = 10.1266 (n = 8, connectionstrength = 0.25, ci = 0.00001, cp = 1). The minimum infor-mation bipartition (MIB) always separated two connectedelements. The value for Φ was fixed irrespective of thetotal length of the path. Addition of a connection linkingthe last and first elements results in a closed path or cycle(Fig. 9B). The complex integrating maximal informationcorresponded to the entire system and its MIB corre-

sponded to a cut into two contiguous halves. The value ofΦ = 20.2533 was exactly twice that for the directed path(Fig. 9A) and remained constant irrespective of the lengthof the cycle. A two-way cycle with reciprocal connectionslinking pairs of elements as nearest neighbors (Fig. 9C)had a main complex equal to the entire system, with aMIB that separated the system into two halves. The valuefor Φ was 40.5065, almost twice that of the one-way cycle,and was constant irrespective of length (n>4). If the sys-tem is a fan-out (divergent) digraph (Fig. 9D) with a singleelement (master) driving or modulating several other ele-ments (slaves), the entire system formed the main com-plex, the MIB corresponded to a cut into two contiguoushalves, and the value of Φ was low (Φ = 10.8198). How-ever, this architecture became more efficient, compared toothers, under low SNR. Predictably, the fan-outarchitecture was rather insensitive to size under high SNR.

Information integration and complexes for a strong modular networkFigure 7Information integration and complexes for a strong modular network. Weights of inter-modular connections are 0.0417 (n = 8, w = 0.5, ci = 0.00001, cp = 1).

A

B

C D

E F

G

1

2

3

45

6

7

8

Φ(subset)

Φ(complex)

Page 10 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

If the system is a fan-in (convergent) digraph (Fig. 9E),where a single element is driven by several other elements,the entire system formed the main complex and MIB cor-responded to a cut into two contiguous halves with low Φvalues (Φ = 10.8198).

It is also interesting to consider what happens if paths,cycles, or a fan-out architecture are added onto a complexof high Φ (Fig. 10). Specifically, adding a path did notchange the composition of the main complex, and did notchange its Φ if the path was out-going from the complex(Fig. 10A; the source element of the path in the complexis called a port out). If the path was in-going (Fig. 10B; the

target element of the path in the complex is called a portin), the Φ value of the main complex could be slightlyaltered, but its composition did not change. The additionof an in-going or an out-going path did form a newcomplex including part or all the main complex plus thepath, but its Φ value was much lower that that of the orig-inal complex, and corresponded to that of the isolatedpath, with the MIB cutting between the path and the portin/out. Adding cycles to a complex (Fig. 10C) also gener-ally did not change the main complex, although Φ valuescould be altered. It should be noted that, even if the ele-ments of a cycle do not become part of the main complex(except for ports in and ports out) they can provide an

Information integration as a function of modularity – homogeneityFigure 8Information integration as a function of modularity – homogeneity. Values of Φ were obtained from Gaussian Toeplitz connection matrices (n = 8) with a fixed amount of total synaptic weight (self-connections allowed). The plot shows Φ as a function of the standard deviation σ of the Toeplitz connection profile. Connection matrices (cases a to f) are shown at the left, for different values of σ. For very low values of σ (σ = 0.001, case a), Φ is zero: the system is made up of 8 causally independent modules. For intermediate values of σ (σ ≈ 0.01 to 0.1, cases b, c, d, e), Φ increases and reaches a maximum; the elements are interacting in a heterogeneous way. For high values of σ (σ = 1, case f) Φ is low; the interactions among the ele-ments are completely homogeneous.

a b

c d

e f

a

b

cd

e

f

σ σ

σ σ

σ σ

σ

Page 11 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

indirect connection between two elements of the maincomplex. Thus, information integration within a complexcan be affected by connections and elements outside thecomplex.

Scaling and joining complexesBecause of the computational requirements, an exhaus-tive analysis of information integration is feasible only forsmall networks. While heuristic search algorithms may beemployed to aid in the analysis of larger networks, herewe report a few observations that can be made usingnetworks with n≤12. Keeping all parameters except sizethe same, maximum values of Φ achieved using nonlinearconstrained optimization grew nearly linearly at bothhigh and low SNR, e.g. increasing from Φ = 39.95 at size4 to Φ = 103.96 at size 12 (w = 0.5, ci = 0.00001, cp = 1).

For larger networks, and especially for networks of biolog-ical significance, a relevant question is whether high levelsof information integration can be obtained by joiningsmaller components. To investigate this question, we firstoptimized two small component networks with n = 8 andthen joined them through 8 pairs of reciprocal connec-tions (Fig. 11A,11B). We refer to the connections insideeach component as "intra-modular", and to thosebetween the two components as "inter-modular". Beforebeing joined the two optimized component networkseach had Φ = 60.7598. After being joined by inter-modu-lar connections, whose strength was equal to the totalstrength of intra-modular connections, the value of Φ forthe system of size n = 16 was Φ = 109.5520. This repre-sents a significant gain in information integration withoutfurther optimization, considerably exceeding the average

Information integration for basic digraphsFigure 9Information integration for basic digraphs. (A) Directed path. (B) One-way cycle. (C) Two-way cycle. (D) Fan-out digraph. (E) Fan-in digraph. Complexes are shaded and values for Φ are provided in each of the panels.

A B C

D E

Φ = 10.1266

Φ = 10.8198

Φ = 40.5065Φ = 20.2533

Φ = 10.8198

Page 12 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

Φ for random networks of size n = 16 (Φ = 51.5930 ±5.2275, 10 exemplars).

The stability of the maximal complex and its capacity tointegrate information depends on the relative synapticstrengths of intra-modular and inter-modular connec-tions. In Fig. 11C, we plot the values for Φ and the size ofthe corresponding complex for different inter-modularcoupling strengths, using two n = 8 sparse optimized com-ponents joined by 8 pairs of reciprocal connections. Atvery low levels of inter-modular coupling, the two compo-nent networks formed separate complexes (see modularnetworks above) and their Φ was close to the average Φattained by optimizing networks of size n = 8(Φ≈60.7598). As coupling increases, at a couplingstrength of approximately 0.001, the two components

suddenly joined as one main complex. This complexreached a maximal Φ≈109.6334 at a coupling ratio of 2:3for intra- versus inter-modular connections. Stronger cou-pling resulted in weaker information integration.

Our results demonstrate that the value for Φ attained bysimply coupling optimized smaller components substan-tially exceeds that of random networks of equivalent size,although it typically does not reach the Φ of optimizednetworks. This gain in integrated information achieved byjoining complexes suggests that larger networks capable ofhigh information integration may be constructed effi-ciently starting from small optimized components.

Adding paths and cycles to a main complexFigure 10Adding paths and cycles to a main complex. (A) Out-going path added to a complex. Complex and directed path are shown separately on the left and joined on the right. (B) In-coming path added to a complex. (C) Cycle added to a complex. Complexes are shaded and values for Φ are provided in each of the panels.

A

B

C

Φ = 60.7596

Φ = 60.7275

Φ = 60.7275

Φ = 20.2533

Φ = 10.1266

Φ = 10.1266Φ = 10.1266 Φ = 60.7596

Φ = 60.7596 Φ = 10.1266

Φ = 60.7596

Φ = 10.1266

Page 13 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

DiscussionIn this paper, we have examined how the capacity to inte-grate information can be measured and how it variesdepending upon the organization of networkconnections. The measure of information integration – Φ– corresponds to the minimum amount of effective infor-mation that can be exchanged across a bipartition of asubset. A subset of elements capable of integratinginformation that is not part of a subset having higher Φconstitutes a complex. As was shown here with linear sys-tems having different architectures, these measures can beused to identify subsets of elements that can integrateinformation and to measure its amount.

Measuring information integrationDespite the significant development of techniques forquantifying information transmission, encoding and stor-age, measures for information integration have been com-paratively neglected. One reason for such neglect may bethat information theory was originally developed in thecontext of communication engineering [1]. In such appli-cation, the sender, channel, and receiver are typicallygiven, and the goal is to maximize information transmis-sion. In distributed networks such as the central nervoussystem, however, most nodes can be both senders andreceivers, multiple pathways can serve as channels, and akey goal is to integrate information for behavioral control[5].

Joining complexesFigure 11Joining complexes. (A) Graph representation of two optimized complexes (n = 8 each), joined by reciprocal "inter-modular" connections (stippled arrows). For simplicity, these connections are arranged as n pairs of bi-directional connections, repre-senting a simple topographic mapping. (B) Connection matrix of a network of size n = 16 constructed from two smaller opti-mized components (n = 8). (C) Information integration as a function of inter-modular coupling strength. Connection strength is given as inter-modular CONij values; all intra-modular connections per element add up to 0.5-CONij. Plot at the bottom shows the size of the complex with highest Φ as a function of inter-modular coupling strength.

1

2

3

45

6

7

8

9

10

11

1213

14

15

16

A

B

C

Page 14 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

In previous work, we introduced a measure called neuralcomplexity (CN), which corresponds to the averagemutual information for all bipartitions of a system [19].We showed that CN is low if the elements of the system areconnected too sparsely (no integration), or if they are con-nected too homogeneously (no specialization), but it ishigh if the elements are connected densely and in aspecific manner (both integration and specialization). CNcaptures an important aspect of the average integration ofinformation within a distributed system. However, it isinsensitive to whether the system constitutes a single, inte-grated entity, or is actually made up of independent sub-sets. For example, based on CN alone it is not possible toknow whether a set of elements is integrated or is arrangedinto parallel, independent channels. To identify whichelements of a system constitute an integrated subset, weintroduced an index of functional clustering (CI), whichreflects the ratio between the overall statistical depend-ence within a subset over the statistical dependencebetween that subset and the rest of the system [25].Because it is based on a ratio, CI requires statistical com-parison to a null hypothesis to rank subsets of differentsize. Like CN, CI cannot detect whether elements aremerely correlated, e.g. because of common input, or arecausally interacting.

The procedure discussed here represents a direct attemptat measuring intergration information, and it differs inseveral important respects from the above-mentionedtwo-step process of finding functional clusters andmeasuring the average mutual information between theirbipartitions. The analysis of complexes identifies subsetsin a single step based on their ability to integrate informa-tion, i.e. their Φ value. Moreover, Φ measures the actualamount of information that can be integrated, rather thanan average value, and eliminates the need for statisticalcomparisons to a null hypothesis. Φ provides therefore ameaningful metric to cluster subsets of neural elementsdepending on the amount of information they canexchange. At a more fundamental level, information canonly be integrated if there are actual interactions within asystem. Accordingly, because it is based on effective infor-mation rather than on mutual information, Φ measurescausal interactions, not just statistical dependencies. Inneural terms, this means that Φ characterizes not just thefunctional connectivity of a network, but the entire rangeof its effective connectivity (in neuroimaging studies,effective connectivity is generally characterized as achange in the strength of interactions between brain areasassociated with time, attentional set, etc. [26,27]). Finally,similar to the capacity for information transmission of achannel [1], the capacity for information integration of asystem is a fundamental property that should onlydepend on the system's parameters and not be affected byobservation time or nonstationarities. Accordingly, Φ

captures all interactions that are possible in a system, notjust those that happen to be observed over a period oftime.

Information integration and basic neuroanatomyA characterization of the structural factors that affect infor-mation integration is especially useful when consideringintricately connected networks such as the nervous sys-tem. Using linear systems of interconnected elements, wehave examined how, given certain constraints, the organ-ization of connection patterns influences informationintegration within complexes of elements. The resultssuggest that only certain architectures, biological orotherwise, are capable of jointly satisfying the require-ments posed by specialization and integration and yieldhigh values of Φ.

The present analysis indicates that, at the most generallevel, networks yielding high values of Φ must satisfy twokey requirements: i) the connection patterns of differentelements must be highly heterogeneous, corresponding tohigh specialization; ii) networks must be highly con-nected, corresponding to high integration (Figs. 3,4,5). Ifone or the other of these two requirements is not satisfied,the capacity to integrate information is greatly reduced.We have shown that homogenously connected networksare not suited to integrating large amounts of informationbecause specialization is lost (Fig. 6), while strongly mod-ular networks fail because of a lack of integration (Fig. 7).Moreover, we have shown that randomly connected net-works do not perform well given constraints on theamount of connections or in the presence of noise, sinceoptimal arrangements of connection patterns are highlyspecific. We have also examined simple digraphs that arewidely represented in biology, such as paths, cycles, andpure convergence and divergence (Fig. 8). Such simplearrangements do not yield by themselves high values of Φ,but they can be helpful to convey the inputs and outputsof a complex, to mediate indirect interactions betweenelements of a complex or of different complexes, and forglobal signalling. Finally, we have seen that, while opti-mizing connection patterns for Φ is unrealistic for largenetworks, high values of Φ can be achieved by joiningsmaller complexes by means of reciprocal connections(Fig. 11). These observations are of some interest whenconsidering basic aspects of neuroanatomicalorganization.

Thalamocortical systemThe above analysis indicates that Φ is maximized by hav-ing each element develop a different connection patternwith the rest of the complex (functional specialization)while ensuring that, irrespective of how the system isdivided in two, a large amount of information can beexchanged between the two parts (functional integration).

Page 15 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

The thalamocortical system appears to be an excellentcandidate for the rapid integration of information. It com-prises a large number of widely distributed elements thatare functionally specialized [8,9]. Anatomical analysis hasshown that specialized areas of cortex maintain heteroge-neous sets of afferent and efferent pathways, presumablycrucial to generate their individual functional properties[28]. At the same time, these elements are intricately inter-connected, both within and between areas, and theseinterconnections show an overabundance of short cycles[12,29]. These connections mediate effective interactionswithin and between areas, as revealed by the occurrence ofshort- and long-range synchronization, a hallmark offunctional integration, both spontaneously and duringcognitive tasks [13-17]. Altogether, such an organizationis reminiscent of that of complexes of near-optimal Φ. Theobservation that it is possible to construct a larger com-plex of high Φ by joining smaller, near-optimal complexesin a fairly straightforward manner, may also be relevant inthis context. For example, while biological learning algo-rithms may achieve near-optimal information integrationon a local scale, it would be very convenient if large com-plexes having high values of Φ could be obtained by lay-ing down reciprocal connections between smallerbuilding blocks.

From a functional point of view, an architecture yieldinga large complex with high values of Φ is ideally suited tosupport cognitive functions that call for rapid, bidirec-tional interactions among specialized brain areas.Consistent with these theoretical considerations, percep-tual, attentional, and executive functions carried out bythe thalamocortical system rely on reentrant interactions[30], show context-dependency [30-32], call for globalaccess [10,11], require that top down predictions arerefined against incoming signals [27], and often occur ina controlled processing mode [33]. Considerable evidencealso indicates that the thalamocortical system constitutesthe necessary and sufficient substrate for the generation ofconscious experience [11,34-36]. We have suggested that,at the phenomenological level, two key properties of con-sciousness are that it is both highly informative and inte-grated [11,18]. Informative, because the occurrence ofeach conscious experience represents one outcome out ofa very large repertoire of possible states; integrated,because the repertoire of possible conscious states belongsto a single, unified entity – the experiencing subject. Thesetwo properties reflect precisely, at the phenomenologicallevel, the ability to integrate information [21]. To theextent that consciousness has to do with the ability tointegrate information, its generation would require asystem having high values of Φ. The hypothesis that theorganization of the thalamocortical system is well suitedto giving rise to a large complex of high Φ would then pro-vide some rationale as to why this part of the brain

appears to constitute the neural substrate of conscious-ness, while other portions of the brain, similarly equippedin terms of number of neurons, connections, and neuro-transmitters, do not contribute much to it [21].

CerebellumThis brain region contains probably more neurons and asmany connections as the cerebral cortex, receives mappedinputs from the environment and controls several out-puts. However, the organization of synaptic connectionswithin the cerebellum is radically different from that ofthe thalamocortical system, and it is rather reminiscent ofthe strongly modular systems analyzed here. Specifically,the organization of the connections is such that individ-ual patches of cerebellar cortex tend to be activated inde-pendently of one another, with little interaction betweendistant patches [37,38]. This suggests that cerebellar con-nections may not be organized so as to generate a largecomplex of high Φ, but rather very many small complexeseach with a low value of Φ. Such an organization seems tobe highly suited for the rapid, effortless execution andrefinement of informationally encapsulated routines,which is thought to be the substrate of automatic process-ing. Moreover, consistent with the notion that consciousexperience requires information integration, even exten-sive cerebellar lesions or cerebellar ablations have littleeffect on consciousness.

Cortical input and output systems, cortico-subcortical loopsAccording to the present analysis, circuits providing inputto a complex, while obviously transmitting informationto it, do not add to its ability to integrate information iftheir effects are entirely accounted for by the elements ofthe complex to which they connect. This situation resem-bles the organization of early sensory pathways and ofmotor pathways. Similar conclusions apply to cyclesattached to a main complex at both ends, a situationexemplified by parallel cortico-subcortico-cortical loops.Consider for example the basal ganglia – large neuralstructures that contain many circuits arranged in parallel,some implicated in motor and oculomotor control, oth-ers, such as the dorsolateral prefrontal circuit, in cognitivefunctions, and others, such as the lateral orbitofrontal andanterior cingulate circuits, in social behavior, motivation,and emotion [39]. Each basal ganglia circuit originates inlayer V of the cortex, and through a last step in the thala-mus, returns to the cortex, not far from where the circuitstarted [40]. Similarly arranged cortico-cerebello-tha-lamo-cortical loops also exist. Our analysis suggests thatsuch subcortical loops, even if connected at both ends tothe thalamocortical system, would not change the compo-sition of the main thalamocortical complex, and wouldonly slightly modify its Φ value. Instead, the elements ofthe main complex and of the connected cycle form a jointcomplex that can only integrate the limited amount of

Page 16 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

information exchanged within the loop. Thus, the presentanalysis suggests that, while subcortical cycles or loopscould implement specialized subroutines capable ofinfluencing the states of the main thalamocortical com-plex, the neural interactions within the loop wouldremain informationally insulated from those occurring inthe main complex. To the extent that consciousness has todo with the ability to integrate information [18], suchinformationally insulated cortico-subcortical loops couldconstitute substrates for unconscious processes that caninfluence conscious experience [21,41]. It is likely thatinformationally encapsulated short loops also occurwithin the cerebral cortex and that they play a similar role.

Limitations and extensionsMost of the results presented in this paper were obtainedon the basis of systems composed of a small number ofelements, and further work will be required to determinewhether they apply to larger systems. For example, it is notclear whether the highest values of Φ can always beobtained by optimizing connection patterns among allelements, or whether factors such as noise, available con-nection strength, and dynamic range (maximum availableentropy) of individual elements would eventually forcethe largest complexes to break down. Similarly, it is notclear whether arbitrarily high values of information inte-gration could be reached by generating connection pat-terns according to some probability distribution andmerely increasing network size. Results obtained withsmall networks (n ≤ 12) indicate that the performance ofconnection patterns generated according to a uniform ornormal random distribution diverges strongly from thatof optimal networks as soon as realistic constraints areintroduced. For example, if we assume that each networkcan only maintain a relatively small number of connec-tions, thus enforcing sparse connectivity, optimized net-works develop specific wirings that yield high Φ (Φ =60.76, for n = 8 and ci = 0.00001), while sparse randomnetworks generated according to a uniform distributionhave low Φ and tend to break down into small complexes(Φ = 33.90). Similarly, if we assume that a certain amountof noise is unavoidable in biological systems (ci = 0.1, cp =1), architectures yielding optimal Φ turn out to be sparseand are clearly superior to random patterns with connec-tions drawn from uniform distributions (Φ = 5.71 versusΦ = 2.92).

The results presented in this initial analysis were obtainedfrom linear systems having no external connections. Realsystems such as brains, of course, are highly non-linear,and they are constantly interacting with the environment.Moreover, the ability of real systems to integrate informa-tion is heavily constrained by factors above and beyondtheir connectional organization. In neural systems, forinstance, factors affecting maximum firing rates, firing

duration, synaptic efficacy, and neural excitability, such asbehavioral state, can radically alter information integra-tion even if the anatomical connectivity is unchanged[21]. Such factors and their influence on Φ value of a com-plex need to be investigated in more realistic models[10,42,43]. Values of effective information and thus of Φare also dependent on both temporal and spatial scalesthat determine the repertoire of states available to a sys-tem. The equilibrium linear systems considered here hadpredefined elementary units and no temporal evolution.In real systems, information integration can occur atmultiple temporal and spatial scales. An interesting possi-bility is that the "grain size" in both time and space atwhich Φ/t reaches a maximum might define the optimalfunctioning range of the system [21]. With respect to time,Φ values in the brain are likely to show a maximumbetween tens and hundreds of milliseconds. It is clear, forexample, that if one were to perturb one half of the brainand examine what effects this produces on the other half,no perturbation would produce any effect whatsoeverafter just a tenth of a millisecond (EI = 0). After say 100milliseconds, however, there is enough time for differen-tial effects to be manifested. Similarly, biological consid-erations suggest that synchronous firing of heavilyinterconnected groups of neurons sharing inputs and out-puts, e.g. cortical minicolumns, may produce significanteffects in the rest of the brain, while asynchronous firingof various combinations of individual neurons may not.Thus, Φ values may be higher when considering as ele-ments cortical minicolumns rather than individual neu-rons, even if their number is lower.

A practical limitation of the present approach has to dowith the combinatorial problem of exhaustively measur-ing Φ in large systems, as well as with the requirement toperturb a system in all possible ways, rather than justobserve it for some time. In this respect, large-scale,realistic neural models offer an opportunity for examiningthe effects of a large number of perturbations in an effi-cient and economical manner [10,42]. In real systems,additional knowledge can often provide an informedguess as to which bipartitions may yield low values of EI.Moreover, as is the case in cluster analysis and other com-binatorial problems, advanced optimization algorithmsmay be employed to circumvent an exhaustive search[44]. In this context, it should be noted that the biparti-tions for which the normalized value of EI will be at aminimum will be most often those that cut the system intwo halves, i.e. midpartitions [18]. Similarly, a represent-ative rather than exhaustive number of perturbations maybe sufficient to estimate the repertoire of differentresponses that are available to neural systems. Work alongthese lines has already progressed in the analysis of sen-sory systems in the context of information transmission[3,45]. It will be essential to extend this work and evaluate

Page 17 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

the effects of perturbations applied directly to the centralnervous system, as could be done for instance by combin-ing transcranial magnetic stimulation with functionalneuroimaging [46,47], or microstimulation with multie-lectrode recordings. Finally, where direct perturbationsare impractical or impossible, a partial characterization ofthe ability of a system to integrate information can still beobtained by exploiting endogenous variance or noise andsubstituting mutual information for effective informationin order to identify complexes and estimate Φ (see exam-ples at http://www.indiana.edu/~cortex/complexity.html.

ConclusionsDespite these limitations and caveats, the approach dis-cussed here demonstrates how it is possible to assesswhich elements of a system are capable of integratinginformation, and how much of it. Such an assessmentmay be particularly relevant for neural systems, given thatintegrating information is arguably what brains are espe-cially good at, and may have evolved for. More generally,the coexistence of functional specialization and integra-tion is a hallmark of most complex systems, and for suchsystems the issue of information integration is para-mount. Thus, a way of localizing and measuring thecapacity to integrate information could be useful in char-acterizing many kinds of complex systems, including generegulatory networks, ecological and social networks, com-puter architectures, and communication networks.

MethodsModel ImplementationIn order to identify complexes and their Φ(S) for systemswith many different connection patterns, we imple-mented numerous model systems X composed of n neuralelements with connections CONij specified by a connec-tion matrix CON(X); self-connections CONii are generallyexcluded. In order to compare different architectures,CON(X) was normalized so that the absolute value of thesum of the afferent synaptic weights per element corre-sponded to a constant value w<1. If the system's dynamicscorresponds to a multivariate Gaussian random process,its covariance matrix COV(X) can be derived analytically.As in previous work [19], we consider the vector X of ran-dom variables that represents the activity of the elementsof X, subject to independent Gaussian noise R of magni-tude c. We have that, when the elements settle under sta-tionary conditions, X = X CON(X) + cR. By defining Q =(1-CON(X))-1 and averaging over the states produced bysuccessive values of R, we obtain the covariance matrixCOV(X) = <X X> = <Qt Rt R Q> = Qt Q, where thesuperscript t refers to the transpose.

Mutual InformationUnder Gaussian assumptions, all deviations from inde-pendence among the two complementary parts A and B of

a subset S of X are expressed by the covariances among therespective elements. Given these covariances, values forthe individual entropies H(A) and H(B), as well as for thejoint entropy of the subset H(S) = H(AB) can be obtained.For example, H(A) = (1/2)ln [(2π e)n|COV(A)|], where |•|denotes the determinant. The mutual informationbetween A and B is then given by MI(A:B) = H(A) + H(B)- H(AB) [2,20]. Note that MI(A:B) is symmetric andpositive.

Effective InformationTo obtain the effective information between A and Bwithin our model systems, we enforced independentnoise sources in A by setting to zero strength the connec-tions within A and afferent to A. Then the covariancematrix for A is equal to the identity matrix (given inde-pendent Gaussian noise), and any statistical dependencebetween A and B must be due to the causal effects of A onB, mediated by the efferent connections of A. Moreover,all possible outputs from A that could affect B are evalu-ated. Under these conditions, EI(A→B) = MI(AHmax:B), asper equation (1). The independent Gaussian noise Rapplied to A is multiplied by cp, the perturbation coeffi-cient, while the independent Gaussian noise applied tothe rest of the system is given by ci, the intrinsic noise coef-ficient. By varying cp and ci one can vary the signal-to-noise ratio (SNR) of the effects of A on the rest of the sys-tem. In what follows, high SNR is obtained by setting cp =1 and ci = 0.00001 and low SNR by setting cp = 1 and ci =0.1. Note that, if A were to be stimulated, rather than sub-stituted, by independent noise sources (by not setting tozero the connections within and to A), one would obtaina modified effective information (and derived measures)that would reflect the probability distribution of A's out-puts as filtered by its connectivity.

ComplexesTo identify complexes and obtain their capacity for infor-mation integration, we consider every subset S ⊆ X, com-posed of k elements, with k = 2,..., n. For each subset S, we

consider all bipartitions and calculate EI(A B) for eachof them according to equation (2). We find the minimuminformation bipartition MIB(S), the bipartition for whichthe normalized effective information reaches a minimumaccording to equation (3), and the corresponding value ofΦ(S), as per equation (4). We then find the complexes of Xas those subsets S with Φ>0 that are not included within asubset having higher Φ and rank them based on their Φ(S)value. As per equation (5), the complex with the maxi-mum value of Φ(S) is the main complex.

Algorithms: Measuring Φ and finding complexesPractically, the procedure for finding complexes can onlybe applied exhaustively to systems composed of up to twodozen elements because the number of subsets and bipar-

∗

∗ ∗ ∗ ∗ ∗

Page 18 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

titions to be considered increases factorially. When deal-ing with larger systems, a random sample of subsets ateach level (e.g. 10,000 samples per level) can provide aninitial, non-exhaustive matrix of complexes. A search forcomplexes of maximum Φ can then be performed byappropriately permuting the subsets with the highest Φ(cf. [44] for optimization procedures in cluster analysis).In most cases, this procedure rapidly identifies the subsetswith the highest Φ among millions of possibilities. MAT-LAB functions used for calculating effective informationand exhaustive search for complexes are available fordownload at http://www.indiana.edu/~cortex/complexity.html or http://tononi.psychiatry.wisc.edu/informationintegration/toolbox.html.

Algorithms: OptimizationTo search for connectivity patterns that are associated withhigh or optimal values of Φ, we employed two optimiza-tion strategies. When searching for connection patternscharacterized by full CON(X) matrices with real-valuedand continuous connection strengths CONij, we used anonlinear constrained optimization algorithm with Φ asthe cost function. When searching for sparse connectionpatterns characterized by binary connection matrices, weused an evolutionary search algorithm similar to an algo-rithm called "graph selection" described in earlier work[22]. Briefly, individual generations of varying connectionmatrices are generated and evaluated with Φ as the fitnessfunction. Networks with the highest values for Φ are cop-ied to the next generation and their connection matricesare randomly rewired to generate new variants. The proce-dure is continued until a maximal value for Φ is reached.

Authors' contributionsBoth authors were involved in performing the simulationsand analyzing the results. GT drafted the initial version ofthe manuscript, which was then revised and expanded byboth authors. Both authors read and approved the finalmanuscript.

AcknowledgementsThe authors were supported by US government contract NMA201-01-C-0034. The views, opinions, and findings contained in this report are those of the authors and should not be construed as an opinion, policy, or deci-sion of NIMA or the US government.

References1. Shannon CE, Weaver W: The mathematical theory of communication

Urbana: University of Illinois Press; 1949. 2. Cover TM, Thomas JA: Elements of information theory New York: Wiley;

1991. 3. Rieke F: Spikes : exploring the neural code Cambridge, Mass.: MIT Press;

1997. 4. Dan Y, Atick JJ, Reid RC: Efficient coding of natural scenes in the

lateral geniculate nucleus: experimental test of a computa-tional theory. J Neurosci 1996, 16:3351-3362.

5. Tononi G, Edelman GM, Sporns O: Complexity and the integra-tion of information in the brain. Trends Cognit Sci 1998, 2:44-52.

6. James W: The principles of psychology New York: H. Holt; 1890. 7. Sherrington CSS: The integrative action of the nervous system New York:

Yale Univ. Press; 1947. 8. Mountcastle VB: Perceptual neuroscience : the cerebral cortex Cambridge,

Mass.: Harvard University Press; 1998. 9. Zeki S: A vision of the brain Oxford ; Boston: Blackwell Scientific

Publications; 1993. 10. Tononi G, Sporns O, Edelman GM: Reentry and the problem of

integrating multiple cortical areas: simulation of dynamicintegration in the visual system. Cerebral Cortex 1992, 2:310-335.

11. Tononi G, Edelman GM: Consciousness and complexity. Science1998, 282:1846-1851.

12. Van Essen DC, Anderson CH, Felleman DJ: Information process-ing in the primate visual system: an integrated systemsperspective. Science 1992, 255:419-423.

13. Munk MH, Nowak LG, Nelson JI, Bullier J: Structural basis of cor-tical synchronization. II. Effects of cortical lesions. JNeurophysiol 1995, 74:2401-2414.

14. Nowak LG, Munk MH, Nelson JI, James AC, Bullier J: Structuralbasis of cortical synchronization. I. Three types of interhem-ispheric coupling. J Neurophysiol 1995, 74:2379-2400.

15. Llinas R, Ribary U, Joliot M, Wang X-J: Content and context intemporal thalamocortical binding. In: Temporal Coding in theBrain Edited by: Buzsaki G, Llinas RR, Singer W. Berlin: Springer Verlag;1994.

16. Engel AK, Fries P, Singer W: Dynamic predictions: oscillationsand synchrony in top-down processing. Nat Rev Neurosci 2001,2:704-716.

17. Varela F, Lachaux JP, Rodriguez E, Martinerie J: The brainweb:phase synchronization and large-scale integration. Nat RevNeurosci 2001, 2:229-239.

18. Tononi G: Information measures for conscious experience.Arch Ital Biol 2001, 139:367-371.

19. Tononi G, Sporns O, Edelman GM: A measure for brain com-plexity: relating functional segregation and integration in thenervous system. Proceedings of the National Academy of Sciences ofthe United States of America 1994, 91:5033-5037.

20. Jones DS: Elementary information theory New York: Oxford UniversityPress; 1979.

21. Tononi G: Consciousness: Theoretical Aspects. In: Encyclopediaof Neuroscience 3rd edition. Edited by: Adelman G, Smith BH. New York:Elsevier; 2003.

22. Sporns O, Tononi G, Edelman GM: Theoretical neuroanatomy:relating anatomical and functional connectivity in graphsand cortical connection matrices. Cereb Cortex 2000,10:127-141.

23. Hilgetag CC, Burns GA, O'Neill MA, Scannell JW, Young MP: Ana-tomical connectivity defines the organization of clusters ofcortical areas in the macaque monkey and the cat. Philos TransR Soc Lond B Biol Sci 2000, 355:91-110.

24. Sporns O, Tononi G: Classes of network connectivity anddynamics. Complexity 2002, 7:28-38.

25. Tononi G, McIntosh AR, Russell DP, Edelman GM: Functional clus-tering: Identifying strongly interactive brain regions in neu-roimaging data. Neuroimage 1998, 7:133-149.

26. Horwitz B: The elusive concept of brain connectivity. Neuroim-age 2003, 19:466-470.

27. Friston K: Beyond phrenology: what can neuroimaging tell usabout distributed circuitry? Annu Rev Neurosci 2002, 25:221-250.

28. Passingham RE, Stephan KE, Kotter R: The anatomical basis offunctional localization in the cortex. Nat Rev Neurosci 2002,3:606-616.

29. Sporns O, Tononi G, Edelman GM: Theoretical neuroanatomyand the connectivity of the cerebral cortex. Behav Brain Res2002, 135:69-74.

30. Edelman GM: Neural Darwinism: selection and reentrant sig-naling in higher brain function. Neuron 1993, 10:115-125.

31. McIntosh AR: Towards a network theory of cognition. NeuralNetw 2000, 13:861-870.

32. McIntosh AR, Rajah MN, Lobaugh NJ: Functional connectivity ofthe medial temporal lobe relates to learning and awareness.J Neurosci 2003, 23:6520-6528.

33. Shiffrin RM, Schneider W: Automatic and controlled processingrevisited. Psychological Review 1984, Vol 91(92):269-276.

34. Crick F, Koch C: A framework for consciousness. Nat Neurosci2003, 6:119-126.

Page 19 of 20(page number not for citation purposes)

BMC Neuroscience 2003, 4 http://www.biomedcentral.com/1471-2202/4/31

Publish with BioMed Central and every scientist can read your work free of charge

"BioMed Central will be the most significant development for disseminating the results of biomedical research in our lifetime."

Sir Paul Nurse, Cancer Research UK

Your research papers will be:

available free of charge to the entire biomedical community

peer reviewed and published immediately upon acceptance

cited in PubMed and archived on PubMed Central

yours — you keep the copyright

Submit your manuscript here:http://www.biomedcentral.com/info/publishing_adv.asp

BioMedcentral

35. Rees G, Kreiman G, Koch C: Neural correlates of consciousnessin humans. Nat Rev Neurosci 2002, 3:261-270.

36. Zeman A: Consciousness. Brain 2001, 124:1263-1289.37. Cohen D, Yarom Y: Patches of synchronized activity in the cer-

ebellar cortex evoked by mossy-fiber stimulation: question-ing the role of parallel fibers. Proc Natl Acad Sci U S A 1998,95:15032-15036.

38. Bower JM: The organization of cerebellar cortical circuitryrevisited: implications for function. Ann N Y Acad Sci 2002,978:135-155.

39. Alexander GE, Crutcher MD, DeLong MR: Basal ganglia-thalamo-cortical circuits: parallel substrates for motor, oculomotor,"prefrontal" and "limbic" functions. Prog Brain Res 1990,85:119-146.

40. Middleton FA, Strick PL: Basal ganglia and cerebellar loops:motor and cognitive circuits. Brain Res Brain Res Rev 2000,31:236-250.

41. Baars BJ: A cognitive theory of consciousness New York, NY, US: CambridgeUniversity Press; 1988.

42. Lumer ED, Edelman GM, Tononi G: Neural dynamics in a modelof the thalamocortical system .1. Layers, loops and theemergence of fast synchronous rhythms. Cerebral Cortex 1997,7:207-227.

43. Tagamets MA, Horwitz B: Integrating electrophysiological andanatomical experimental data to create a large-scale modelthat simulates a delayed match-to-sample human brainimaging study. Cereb Cortex 1998, 8:310-320.

44. Everitt B: Cluster analysis London, New York: E. Arnold, Halsted Press;1993.

45. Buracas GT, Albright TD: Gauging sensory representations inthe brain. Trends Neurosci 1999, 22:303-309.

46. Pascual-Leone A, Walsh V, Rothwell J: Transcranial magneticstimulation in cognitive neuroscience – virtual lesion, chro-nometry, and functional connectivity. Curr Opin Neurobiol 2000,10:232-237.

47. Paus T: Imaging the brain before, during, and after transcra-nial magnetic stimulation. Neuropsychologia 1999, 37:219-224.

Page 20 of 20(page number not for citation purposes)