Parameter Space Compression Underlies Emergent Theories and Predictive Models

Benjamin B. Machta,1, 2 Ricky Chachra,1 Mark Transtrum,1, 3 and James P. Sethna1

1Laboratory of Atomic and Solid State Physics, Department of Physics, Cornell University, Ithaca NY 148532Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton NJ 08854

3MD Anderson Cancer Center, University of Texas, Houston TX 77030

We report a similarity between the microscopic parameter dependance of emergent theories inphysics and that of multiparameter models common in other areas of science. In both cases, predic-tions are possible despite large uncertainties in the microscopic parameters because these details arecompressed into just a few governing parameters that are sufficient to describe relevant observables.We make this commonality explicit by examining parameter sensitivity in a hopping model of diffu-sion and a generalized Ising model of ferromagnetism. We trace the emergence of a smaller effectivemodel to the development of a hierarchy of parameter importance quantified by the eigenvalues ofthe Fisher Information Matrix. Strikingly, the same hierarchy appears ubiquitously in models takenfrom diverse areas of science. We conclude that the emergence of effective continuum and universaltheories in physics is due to the same parameter space hierarchy that underlies predictive modelingin other areas of science.

The success of science, and the comprehensibility ofnature owes in large part to the hierarchical character ofscientific theories [1, 2]. These theories of our physicalworld, ranging in scales from the sub-atomic to the as-tronomical, model natural phenomena as if physics atmacroscopic length scales were almost independent ofthe underlying, shorter length scale details. For exam-ple, understanding string theory or some other funda-mental high energy theory is not necessary for quantita-tively modeling the behavior of superconductors that op-erate in a lower energy regime. The fact that many lowerlevel theories in physics can be systematically coarsened(renormalized) into macroscopic effective models, estab-lishes and quantifies their hierarchical character. More-over, experience suggests that a similar hierarchy of the-ories is also at play in multiparameter models in otherareas of science even though a similarly systematic coars-ening or model reduction is often difficult [3–7]. In fact,as we show here, the effectiveness of these emergent the-ories in physics also relies on the same parameter spacehierarchy that is ubiquitous in multiparameter models.

Recent studies of nonlinear, multiparameter modelsdrawn from disparate areas in science have shown thatpredictions from these models largely depend only ona few ‘stiff’ combinations of parameters [6, 8, 9]. Thisrecurring characteristic (termed ‘sloppiness’) appears tobe an inherent property of these models and may be amanifestation of an underlying universality [11]. Indeed,many of the practical and philosophical implications ofsloppiness are identical to those of the renormalizationgroup (RG) and continuum limit methods of statisticalphysics: models show weak dependance of macroscopicobservables (defined at long length and time scales) onmicroscopic details. They thus have a smaller effectivemodel dimensionality than their microscopic parameterspace [12]. To clarify their connection to sloppiness, weapply an information theory based analysis to modelswhere the continuum limit and the renormalization groupalready give a quantitative explanation for the emergenceof effective models— a hopping model of diffusion and an

Ising model of ferromagnetism and phase transitions. Inboth cases, our results show that at long time and lengthscales a similar hierarchy develops in the microscopic pa-rameter space, with sensitive, or ‘stiff’ directions corre-sponding to the relevant macroscopic parameters (such asthe diffusion constant in the diffusion model). Moreover,as we show below, even where model reduction cannot besystemically generated, stiff combinations of parametersstill do describe a universal effective model of a smallerdimension that captures most collective observables.

We use information theory to track the developmentof this hierarchy in microscopic parameter space. Thesensitivity of model predictions to changes in parametersis quantified by the Fisher Information Matrix (FIM).The FIM forms a metric that converts parameter spacedistance into a unique measure of distinguishability be-tween a model with parameters θµ (for 1 ≤ µ ≤ N) anda nearby model with parameters θµ + δθµ (see supple-mentary text and [3, 14, 15]). This divergence is givenby ds2 = gµνδθ

µδθν where gµν is the FIM defined by:

gµν = −∑

observables ~x

Pθ(~x)∂2 logPθ(~x)

∂θµ∂θν(1)

where Pθ(~x) is the probability that a (stochastic) modelwith parameters θµ would produce observables ~x. In thecontext of nonlinear least squares, g is the Hessian of chi-squared, the sum of squares of independent standard nor-mal residuals of data-fitting (supplementary text). Dis-tance in this metric space is a fundamental measure ofdistinguishability in stochastic systems. Sorted by eigen-values, eigenvectors of g describe a hierarchy of linearcombinations of parameters that govern system behavior.Previously, it was shown that in nonlinear least squaresmodels, the eigenvalues form a roughly geometrical se-quence, reaching extremely small values in many models(figure 1). Thus, the eigenvalues of the FIM quantifya hierarchy in parameter space: few ‘stiff’ eigenvectorsin each model point along directions where observablesare sensitive to changes in parameters, while progres-

arX

iv:1

303.

6738

v1 [

cond

-mat

.sta

t-m

ech]

27

Mar

201

3

2

sively sloppier directions make little difference for observ-ables. These sloppy parameters cannot be inferred fromdata, and conversely, their exact values do not need to beknown to quantitatively understand system behavior [9].To see how this comes about, we turn to a ‘microscopic’model of stochastic motion from which the diffusion equa-tion emerges.

FIG. 1: Eigenvalues of the Fisher Information Matrix (FIM)of various models are shown. Diffusive hopping model andthe Ising model of ferromagnetism shown in first two columnsare explored in this paper. Models of radioactive decay and aneural network are taken from a previous study [7]. The sys-tems biology model is a differential equation model of a MAPkinase cascade taken from [17]. In all models, we find that theeigenvalues of the FIM are roughly geometrically distributedforming a hierarchy in parameter space, with each successivedirection significantly less important. Eigenvalues are normal-ized to unit stiffest value; only the first 10 decades are shown.This means that inferring the parameter combination whoseeigenvalue is smallest shown would require ∼ 1010 times moredata than the stiffest parameter combination. Conversely,this means that the least important parameter combinationis√

1010 times less important for understanding system be-havior. This is a much larger range in eigenvalues than thatpredicted by Wishart statistics (black line marked random),the naive expectation for least squares problems.

The diffusion equation is the canonical example of acontinuum limit. It governs behavior whenever smallparticles undergo stochastic motion. Given translationinvariance in space and time, it subsumes complex micro-scopic collisions into an equation with only three termswhich describe the time evolution of the particle densityρ: ∂tρ(r, t) = D∇2ρ− ~v · ∇ρ+Rρ, where D is the diffu-sion constant ~v is the drift and R is the particle creationrate. Microscopic parameters describing the particles andtheir environment enter into this continuum descriptiononly through their effects on the terms in this equation.To see this, consider a microscopic model of stochasticmotion on a discrete 1-dimensional lattice of sites, with

2N + 1 parameters θµ, for −N ≤ µ ≤ N which describethe probability that in a discrete time step a particle willhop from site j to site j + µ (figure 2 inset). At initialtime, all particles are at the origin, ρ0(j) = δj,0. Theobservables, ~x ≡ ρt(j), are the densities of particles atsome later time t.

After a single time step the distribution of particlesis given by ρ1(j) = θj . This distribution depends inde-pendently on all of its parameters, thus the FIM is theidentity, gµν = δµν (supplementary text). After a singletime step, there is no parameter hierarchy—each param-eter is measured independently. When particles take sev-eral time steps before their positions are observed, someparameter combinations become easier to measure: fewercoarsened observations achieve the same accuracy. Otherparameter combinations become harder to measure, re-quiring exponentially more observations (supplementarytext). At late times, the particle creation rate, R, be-comes easier to measure as the mean particle numberchanges exponentially with time. The next eigenvalue,the drift, also becomes easier to measure as time passes.The diffusion constant itself becomes harder to measureas time passes, and further eigenvectors, describing theskew, kurtosis and higher moments of the final distribu-tion become harder and harder to measure as more timesteps are made, each with a higher negative power of t(see figure 2 and supplementary text). This gives an in-formation theoretic explanation for the wide applicabilityof the diffusion equation. Any system with stochastic mo-tion and conservation of particle number will have a driftterm dominate if it is present (for example, for a smallparticle falling through honey under gravity, in which wemight neglect diffusion). If drift is constrained to be zero,by symmetry for example, then the diffusion constantwill dominate in the continuum limit. Since the diffusionconstant cannot be removed for stochastic systems, thereis never a need for higher terms to enter into a contin-uum description. These results quantify a widely heldintuition: one cannot infer microscopic parameters, suchas the bond angle of a water molecule, from a diffusionmeasurement, and conversely it would also be unneces-sary to have such knowledge to quantitatively understandthe coarse behavior of diffusing particles in water.

Continuum models like the diffusion equation arisewhen fluctuations are only large on the micro scale.Their success can be said to rely on the largeness andslowness of observables when compared with the natu-ral scale of fluctuations. However, RG methods clarifythat system behavior can be universal even when fluc-tuations are large on all scales, as occurs near criticalpoints. The Ising model is the simplest model which ex-hibits nontrivial thermodynamic critical behavior. Nearits critical point, the Ising model predicts fractal do-mains whose statistics are universal, quantitatively de-scribing the spatial structure of magnetic fluctuationsin ferromagnets, the density fluctuations near a liquid-gas transition and the composition fluctuations near aliquid-liquid miscibility transition [18, 19]. Consider a

3

FIG. 2: We consider a hopping model on a 1-D lattice, withseven parameters describing the probability that a particlewill remain at its current step or move to one of its six near-est neighbors in a discrete time step. We calculate the FIMfor this model, for observations taken after a given number oftime steps, for the case where all parameters take the valueaµ = 1/7. Top row shows the resulting densities plotted attimes t = 1, 3, 5, 7. Bottom plot shows the eigenvalues of theFIM versus number of steps. After a single time step, theFIM is the identity, but as time progresses, the spectrum ofthe FIM develops a hierarchy spanning many orders of magni-tude. The second eigenvector measures a net rate of particlecreation, R. The next eigenvector measures a net drift in thedensity, ~v. The third eigenvector corresponds to parametercombinations that change the diffusion constant, D. Eachof the above will dominate a continuum description if thoseabove it are constrained to be 0 (or are otherwise small).Further eigenvectors describe parameter combinations thatdo not affect these macroscopic parameters, but instead mea-sure the kurtosis, skew, and higher moments of the resultingdensity.

two dimensional square lattice Ising model where at ev-ery site a ‘spin’ takes a value of si,j = ±1. Observablesare spin configurations (~x = {si,j}) or subsets of spinconfigurations (~xn, as defined below). The Ising modelassigns to each spin configuration a probability given byits Boltzmann weight, Pθ(~x) = e−Hθ(~x)/Z. The model isparametrized through it’s Hamiltonian Hθ(~x) = θµΦµ(~x)where θµ are parameters describing a field θ0 whichmultiplies Φ0(~x) =

∑i,j si,j , or, a coupling between

spins and one of their nearby neighbors, θαβ , multiply-ing Φαβ(~x) =

∑i,j si,jsi+α,j+β (see inset of figure 3 and

supplementary text).

At the microscopic level, all spins are observable and

the Ising FIM is a sum of 2 and 4-spin correlation func-tions that can be readily calculated using Monte-Carlotechniques ( [9] and supplementary text). Near the criti-cal point, it has two ‘relevant’ eigenvectors with eigenval-ues that diverge like the specific heat and magnetic sus-ceptibility [10, 12]. These two large eigenvalues have noanalog in the diffusion equation, and reflect the presenceof fluctuations at scales much larger than the microscopicscale (here this scale is the lattice constant: the distancebetween neighboring sites). The remaining eigenvaluesall take a characteristic scale given by the system size, inunits of the lattice constant (supplementary text). Theclustering of the remaining eigenvalues is reminiscent ofthe spectrum seen in the diffusion equation when viewedat its microscopic (time) scale. When observables are mi-croscopic spin configurations, the nearest neighbor Isingmodel is a poor description of a binary liquid, and evenof a ferromagnet.

To coarsen the Ising model, the observables are re-stricted to a subset of lattice sites chosen via checker-board decimation procedure (figure 3 top row inset fig-ures). The FIM of equation 8 is now measured using asour observables only those sites in a sub-lattice decimatedby a factor 2n, ~xn = {si,j}{i,j} in n. For example, after1 level of decimation, this corresponds to the black siteson the checkerboard, while after 2 steps, only sites {i, j}where i and j are even remain. Importantly, the distri-bution is still drawn from the ensemble defined by theoriginal Hamiltonian defined on the full lattice. The cal-culation is implemented using compatible Monte-Carlo( [1] and supplementary text).

The results from Monte-Carlo are presented for a64× 64 system at its critical point in figure 3. The irrel-evant and marginal eigenvalues of the metric continue tobehave much as the eigenvalues of the metric in the dif-fusion equation, becoming progressively less importantunder coarsening with characteristic eigenvalues. How-ever, the large eigenvalues, dominated by singular cor-rections, do not become smaller under coarsening; theyare measured by their collective effects on the large scalebehavior, which is primarily informed by large distancecorrelations. In the supplementary text, we use RG anal-ysis to explain the scaling of the FIM eigenvalues withthe coarse-graining level. The analysis clarifies that ‘rel-evant’ directions in the RG are exactly those whose FIMeigenvalues do not contract on coarsening. They con-trol the large-wavelength fluctuations of the model, andthey dominate the behavior provided that the correlationlength of fluctuations is larger than the observation scale.

We have seen that neither the hopping model nor theIsing model are sloppy at their microscopic scales. It isonly upon coarsening the observables, either by allowingseveral time steps to pass, or by only observing a subset oflattice sites, that a typical sloppy spectrum of parametercombinations emerges. Correspondingly, multiparame-ter models such as in systems biology and other areasof science are sloppy only when fit to experiments thatprobe collective behavior— if experiments are designed

4

FIG. 3: We consider an Ising model of ferromagnetism asdefined in the text, with 13 parameters describing nearestand nearby neighbor couplings (shown in the bottom inset),and magnetic field. Observables are spin configurations of allspins on a sub-lattice (dark sites in the insets of the top panel).Top panel shows one particular spin configuration generatedby our model, suitably blurred for level > 0 to the averagespin conditioned on the observed sub-lattice values. As canbe seen by eye, some information about the configuration ispreserved by this procedure (the typical size of fluctuations,for example), while other information, like the nearest neigh-bor correlation function, is lost. We quantify this by measur-ing the eigenvalues of the FIM of this model as a function ofcoarse-graining level. As this coarsening step only discards in-formation, all of the eigenvalues must be non-increasing withlevel. The two largest eigenvalues, whose eigenvectors mea-sure T −Tc and the applied field h do not shrink substantiallyunder coarsening (supplementary text). Further eigenvaluesshrink by a factor of 2−d−yi in each step, where yi is the ith

RG eigenvalue.

to measure one parameter at a time, no hierarchy can beexpected [23, 24]. In the models examined here, there isa clear distinction between the short time or length scaleof the microscopic theory, and the long time or lengthscale of observables. As we show more formally in thesupplementary text, sloppiness can be precisely tracedto the ratio of these two scales— an important smallvariable. On the other hand, in many other areas of sci-ence such a distinction of scales cannot always be made.As such, those models cannot be coarsened or reducedin the same systematic way using methods readily ap-plicable to physics theories (see also [7]). Nonetheless,owing to their sloppy FIMs, these models share many ofthe striking implications of the continuum limit and RGmethods.

We thank Seppe Kuehn and Stefanos Papanikolaou foruseful comments and discussions. This work was sup-ported by NSF grant DMR 1005479 and a Lewis-SiglerFellowhip (BBM).

[1] P. W. Anderson, Science 177, 393 (1972).[2] E. P. Wigner, Communications on Pure and Applied

Mathematics 13, 1 (1960).[3] U. Alon, Nature 446, 497 (2007).[4] G. J. Stephens, B. Johnson-Kerner, W. Bialek, W. S.

Ryu, PLoS Computational Biology 4 (2008).[5] G. J. Stephens, L. C. Osborne, W. Bialek, Proceedings

Of the National Academy of Sciences 108, 15565 (2011).[6] T. Sanger, Journal of Neuroscience 20, 1066 (2000).[7] M. K. Transtrum, P. Qiu, Submitted (2013).[8] R. N. Gutenkunst, et al., PLoS Comput Biol 3, e189

(2007).[9] J. J. Waterfall, et al., Phys. Rev. Lett. 97, 150601 (2006).

[10] M. K. Transtrum, B. B. Machta, J. P. Sethna, Phys. Rev.

Lett. 104, 060201 (2010).[11] T. Mora, W. Bialek, Journal of Statistical Physics 144,

268 (2011).[12] J. Cardy, Scaling and Renormalization in Statistical

Physics (Cambridge University Press, 1996).[13] S. Amari, H. Nagaoka, Methods of Information Geome-

try , Translations of Mathematical Monographs (Ameri-can Mathematical Society, 2000).

[14] I. Myung, V. Balasubramanian, M. Pitt, Proceedings ofthe National Academy of Sciences 97, 11170 (2000).

[15] V. Balasubramanian, Neural Computation 9, 349 (1997).[16] M. K. Transtrum, B. B. Machta, J. P. Sethna, Phys. Rev.

E 83, 036701 (2011).[17] K. S. Brown, et al., Physical Biology 1, 184 (2004).

5

[18] P. M. Chaikin, T. C. Lubensky, Principles of Con-densed Matter Physics (Cambridge University Press,Cambridge, 1995).

[19] S. L. Veatch, O. Soubias, S. L. Keller, K. Gawrisch, Proc.Natl. Acad. Sci. USA 104, 17650 (2007).

[20] G. E. Crooks, Phys. Rev. Lett. 99 (2007).[21] G. Ruppeiner, Reviews Of Modern Physics 67 (1995).[22] D. Ron, R. Swendsen, A. Brandt, Physical Review Letters

89 (2002).[23] F. P. Casey, et al., Systems Biology, IET 1, 190 (2007).[24] J. F. Apgar, D. K. Witmer, F. M. White, B. Tidor,

Molecular Biosystems 6, 1890 (2010).

6

Supplementary Information

I. INTRODUCTION

This supplement contains relevant background andcomputational details to accompany the main text. Insection II we provide a pedagogical overview of the in-formation theoretic tools that we use to quantify distin-guishability. In section III we apply this formalism toa model of stochastic motion that is described in themain text and provide details of the calculation that un-derlies figure 2 of the main text. We also provide anasymptotic analysis of the scaling of the FIM’s eigen-values in the limit where coarsening has proceeded formany time steps. In sections IV- VII we discuss the Isingmodel. In section IV we carefully define our 13 parame-ter Ising model as briefly described in the main text. Insection V we give an outline of our numerical techniquesfor measuring the FIM, as well as give a scaling argu-ment that explains its spectrum before coarsening. Insection VI we extend this analysis to the coarsened case.In section VII we give details of our Monte-Carlo tech-niques, with emphasis on our implementation of ‘Com-patible Monte-Carlo’ [1].

II. INFORMATION GEOMETRY AND THEFISHER METRIC

How different are two probability distributions, P1(x)and P2(x)? What is the correct measure of distancebetween them? In this section we give an overview ofan information theoretic approach to this question [2–4]. Imagine being given a sequence of independent datapoints {x1, x2, ...xN}, with the task of inferring which ofthe two models would be more likely to have generatedthe data. As probabilities multiply, the probability thatP1 would have generated the data is given by:

∏i

P1(xi) = exp

(∑i

logP1(xi)

)(2)

and by calculating this for each of the two distributionsP1(x) and P2(x), we could see which model would bemore likely to have produced the observed data.

How difficult should one expect this task to be? Pre-suming N to be large we can estimate the probabilitythat a typical string generated by P1 would be producedby P1. To do this we simply take a product similar tothat in equation 2 but with each state x entering into theproduct NP1(x) times:

∏xP1(x)NP1(x) = exp

(N∑xP1(x) logP1(x)

)= exp(−NS1)

(3)

where we note that this gives an alternative definition ofthe familiar entropy S1 of P1 (in nats). We can also askhow likely P2 is to produce a typical ensemble generatedby P1. This is just given by:

∏x

P2(x)NP1(x) = exp

(N∑x

P1(x) logP2(x)

)(4)

We can ask how much more likely a typical ensemblefrom P1 is to have come from P1 rather than from P2.This is given by:

∏x

(P1(x)/P2(x))NP1(x) = exp

(N∑xP1(x) log

(P1(x)P2(x)

))= exp (−NDKL(P1||P2))

(5)This defines the Kullback-Liebler Divergence, the sta-

tistical measure of how distinguishable P1 is from P2 fromits data x [4, 5]:

DKL(P1||P2) =∑x

P1(x) log

(P1(x)

P2(x)

)(6)

This measure has several properties that prevent itfrom being a proper mathematical distance measure,most obviously that it does not necessarily satisfyDKL(P1||P2) = DKL(P2||P1)1 . However, for two ‘close-by’ models DKL does become symmetric. Consider acontinuously parameterized set of models Pθ where θ isa set of N parameters θµ. The infinitesimal Kullback-Liebler divergence between models Pθ and Pθ+∆θ takesthe form2:

DKL(Pθ, Pθ+∆θ) = gµν∆θµ∆θν +O∆θ3 (7)

where gµν is the Fisher Information Matrix (FIM), givenby:

gµν(Pθ) = −∑x

Pθ(x)∂

∂θµ∂

∂θνlogPθ(x) (8)

The quadratic form of the KL-divergence at short dis-tances motivates using the FIM as a metric on param-eter space. This defines a Riemannian manifold3 whereeach point on the manifold specifies a probability distri-bution [3]. The tensor gµν can be shown to have all of

1 A distance measure should also satisfy some sort of general-ized triangle inequality- at the very least D(A,B) + D(B,C) ≥D(A,C) which is also not necessarily satisfied here.

2 It is an interesting exercise to show that there is no term linear in∆θ. The crucial step uses that Pθ is a probability distributionsso that ∂µ

∑x Pθ(x) = 0.

3 Although typical models contain internal singularities, where themetric has eigenvalues that are 0 (see [6, 7]).

7

the necessary requirements to be a metric- it is symmet-ric (derivatives commute) and positive semi-definite (in-tuitively because no model can fit any model better thanthat model fits itself). It also has the correct transforma-tion laws under a reparameterization of the parametersθ. Distance on this manifold is (at least locally) a mea-sure of how distinguishable two models are from theirdata, in dimensionless units of standard deviations. Thisalready gives one important difference between informa-tion geometry and the more familiar use of Riemanniangeometry in General Relativity. In General Relativitydistances are dimensionful, measured in meters. Whilecertain functions of the manifold (notably the Scalar cur-vature) are dimensionless and can appear in interestingways on their own, a distance is only large or small whencompared to some other distance. In information geom-etry, by contrast, distances have an intrinsic meaning-Probability distributions are distinguishable from a typ-ical measurement provided the distance between them isgreater than one. Below we consider the metric for twospecial cases.

A. The metric of a Gaussian model

First, motivated by non-linear least squares we con-sider a model whose output is a vector of data, yi (for1 < i < M). Underlying least squares is the assumptionthat observed data is normally distributed with widthσi around a parameter dependent value, ~y0(θ). As such,the ‘cost’ or sum of squared residuals is proportional tothe log of the probability of the model having producedthe data. We write the probability distribution of data ygiven a set of parameters θ as:

Pθ(~y) ∼ exp

(−∑i

(yi − yi0(θ))2/2σi2

)(9)

Defining the Jacobian between parameters and scaleddata as:

Jiµ =∂

∂θµyi0(θ)

σi(10)

The Fisher information for least squares problems is sim-ply given by4 [6, 7]:

gµν =∑i

JiµJiν (11)

This particular metric has a geometric interpretation:distance is locally the same as that measured by embed-ding the model in the space of scaled data according to

4 This assumes that the uncertainty σi does not depend on theparameters, and that errors are diagonal. Both of these assump-tions seem reasonable for a wide class of models, for exampleif measurement error dominates. The more general case is stilltractable, but less transparent.

the mapping y0(θ) (it is induced by the Euclidian metricin data space). It is exactly this metric that was shown tobe sloppy in seventeen models from the system’s biologyliterature [6–8].

B. The metric of a Stat-Mech Model

Second, we consider the case of an exponential model,familiar from statistical mechanics, defined by a param-eter dependent Hamiltonian that assigns an energy toevery possible configuration, x. (We set the temperatureas well as Boltzmann’s constant to 1) Each parameterθµ controls the relative weighting of some function of theconfiguration, Φµ(x), which together define the probabil-ity distribution on configurations through:

P (x|θ) = exp(−Hθ(x))/ZZ(θ) = exp(−F (θ)) =

∑x

exp(−Hθ(x))

Hθ(x) =∑µθµΦµ(x)

(12)

Though perhaps unfamiliar, typical models can be putinto this form. For example, the 2D Ising model of sec-tion IV has spins si,j = ±1 on a square LxL latticewith the configuration, x = {si,j}, being the state of allspins. The magnetic field, θ0 = h multiplies Φ0({si,j}) =∑i,j si,j , and the nearest neighbor coupling, θ1 = −J

multiplies Φ1({si,j}) =∑i,j si,jsi+1,j + si,jsi,j+1. This

form is chosen for convenience in calculating the metric,which is written [9, 10]5:

gµν = 〈−∂µ∂ν log(P (x))〉= 〈∂µ∂νH(x)〉+ ∂µ∂ν log(z)= ∂µ∂ν log(z) = −∂µ∂νF

(13)

To write the last line we have taken advantage of the factthat the Hamiltonian is linear in parameters θµ so that〈∂µ∂νH(x)〉 = 0. As such, the last line does not trans-form like a metric under an arbitrary reparameterization,but only one that preserves the form given in equation12.

III. A CONTINUUM LIMIT: DIFFUSION

With these definitions in hand, we turn to a specificproblem where information about microscopic details is

5 Several seemingly reasonable metrics can be defined for systemsin statistical mechanics and all give similar results in most cir-cumstances [10]. Most differences occur either for systems notin a true thermodynamic (N large) limit, or for systems near acritical point. As far as we are aware, Crooks [9] was the firstto stress that the one used here can be derived from informationtheoretic principles, perhaps making it the most ‘natural’ choice.In [9] Crooks showed that when using this metric ‘length’ hasan interesting connection to dissipation by way of the Jarzynskiequality [11].

8

lost in a coarse-grained description. A prototypical ex-ample of such a continuum limit is the emergence of thediffusion equation in a system consisting of small parti-cles undergoing stochastic motion. Diffusion effectivelydescribes the motion of a particle provided that there istranslation invariance in time and space and that par-ticle number is conserved. Microscopic parameters thatdescribe details of the medium in which the particle isdiffusing and the molecular details of such an object en-ter into this continuum description only through theireffects on the diffusion constant, or, if it is present, therate of drift. Furthermore, knowing molecular details(for example the bond angle of a water molecule in themedium through which a particle is diffusing) that mightenter into a microscopic description of the motion wouldbe extremely unhelpful in predicting a particle’s diffusionconstant.

To see how this comes about we consider a ‘micro-scopic’ model of stochastic motion on a discrete latticeof sites j. Our model is defined by 2N+1 parameters θµ,for −N ≤ µ ≤ N which describe the probability that ina discrete time step a particle will hop from site j to sitej+µ. We presume that we start our particles from a dis-tribution ρ0(j), and that our measurement data consistsof the number of particles at some later time t, ρt(j).

We first consider taking ‘microscopic’ measurements ofour model parameters, by starting with an initial prob-ability distribution ρ0(j) = δj,0, and observing the dis-tribution after one time step, ρ1(j). This distribution isjust given by:

ρ1(j) = θj (14)

Presuming our measurement uncertainty of the num-ber of particles at each site is Gaussian, with width6

σmeas = 1. we can calculate the Fisher metric on theparameter space using the Least Squares metric definedin equations 10 and 11:

Ji,µ = ∂µρ1(i) = δi,µgµν =

∑i Ji,µJi,ν

= δµν

(15)

This metric has 2N+1 eigenvalues each with value λ = 1.All of the parameters in this model are measurable withequal accuracy. Additionally, if we wanted to understandthe behavior at this microscopic level, there is no reasonto think that a reduced description of the model shouldbe possible; each direction in parameter space is equallyimportant in determining the one step evolution from the

6 We could carry out a more complicated calculation assumingour uncertainty comes from the stochastic nature of the modelitself, but presuming we start with many particles, this approachwould yield similar but less transparent results. Changing themeasurement uncertainty from 1 to σmeas will multiply all cal-culated metrics by a trivial factor of 1/σ2

meas and is omitted forclarity.

origin. We next examine the behavior of the FIM for datathat is in the form of densities measured after multipletime steps.

A. Coarsening the diffusion equation by observingat long times

The molecular timescale is typically much faster thanthe typical timescale of a measurement. We ask how ourability to measure microscopic parameters changes withexperiment time.

To calculate the density of particles at position j andtime t, ρt(j), it is useful to introduce the Fourier trans-form of the hopping rates, as well as the Fourier trans-form of the particle density at time t:

θ̃k =N∑

µ=−Ne−ikµθµ

ρ̃kt =∞∑

j=−∞e−ikjρt(j)

ρt(j) = 12π

π∫−π

dkeikj ρ̃kt

(16)

In a time step the density distribution is convoluted bythe hopping rates. In Fourier space this is simply writtenas7:

ρ̃kt = θ̃kρ̃kt−1 (17)

We choose initial conditions with all particles at the ori-gin ρ0(j) = δj,0, so that:

ρ̃kt = (θ̃k)t

ρt(j) = 12π

π∫−π

dkeikj(θ̃k)t(18)

The Jacobian and metric at time t can now be written:

J tjµ = ∂µρt(j) = t2π

π∫−π

dkeik(j−µ)(θ̃k)t−1

gtµν = t2

2π

π∫−π

dkeik(µ−ν)(θ̃k)t−1(θ̃−k)t−1(19)

The metric now depends on the θ themselves. Presumingthe (positive) hopping rates θµ values sum to 1 with atleast two non-zero, then all of the θ values are less thanone and the late time behavior of gtµν is dominated bysmall k values appearing in the integrand (equation 19).At small values of k:

θ̃k = 1− ikv − k2

2 ∆ +O(k3)

= exp(−ikv −D k2

2 ) +O(k3)v =

∑µ µθ

µ

∆ =∑µ µ

2θµ

D = ∆− v2

(20)

7 This is due to the convolution theorem. See, for example [12]

9

where in going from the first line to the second we notethese two equations are the same to second order in k.Here v is the drift and D is the diffusion constant. Fromthis approximation we can estimate the form of gtµν forlate times. For the case where the drift v = 0:

gtµν ≈ t2

2π

∞∫−∞

dkeik(µ−ν)e−Dtk2

∼ t2

(Dt)1/2e−(µ−ν)2/4Dt

(21)

We can expand this in powers of the small parameter(µ− ν)2/Dt. This gives

gtµν ∼ t2((Dt)−1/2 − (Dt)−3/2(µ− ν)2/4 + · · ·)

= t2∞∑n=0

(−1)n(µ−ν)2n

n!(4Dt)n+1/2

(22)

Each term in the series contributes a single new non-zeroeigenvalue which scales like:

λn ∼ t2(Dt

N2

)−n−1/2

n ≥ 0 (23)

The corresponding eigenvectors are best understoodby considering their projection onto the observables.These are proportional to the left singular vectors of J,vL,n = (1/λn)Jiµv

µn. These are exactly the Hermite poly-

nomials of a gaussian with width 2σ =√Dt. The first

one measures non-conservation of particle number, R, thesecond measures drift, v, and the third measures changesin the diffusion coefficient, D. The next terms are lessfamiliar; those past n = 2 never appear in a continuumdescription, because they are always harder to observethan the diffusion constant by a factor of the ratio ofthe observation scale (

√Dt) to the microscopic scale (N)

raised to a positive integer power. It is not possible forthe diffusion constant, as defined here, to be 0 while anyhigher cumulants are non-zero, explaining why thoughdrift and the diffusion constant both appear in contin-uum limits, the physical parameter that describes thethird cumulant does not. The next eigendirection mea-sures the Skew of the resulting density distribution, whilethe next one measures the distribution’s Kurtosis, and soon. It is worth noting that careful observation of a partic-ular θµ, somewhat analogous to knowing the bond-angleof a water molecule, would give very little insight on therelevant observables. The exact eigenvalues, measured atsteps t = 1 − 7 are plotted in figure 2 of the main textfor an N = 3 (seven parameter) model where θµ = 1/7for all µ.

IV. A CRITICAL POINT: THE ISING MODEL

The success of the continuum limit might be said torest on the ‘boringness’ of the large-scale behavior. Allof the fluctuations in the system are essentially averagedat the scale of typical observations. This fails to be truenear critical points of systems, where fluctuations remain

large up to a characteristic scale ξ which diverges at thecritical point itself. Perhaps surprisingly, even at thesepoints these systems have behavior that is universal. TheIsing model, for example, provides a quantitative descrip-tion of both Ferromagnetic and liquid-gas critical points,describing all of the statistics of the observable fluctu-ations of both systems, even though they have entirelydifferent microscopic components. Just as in diffusion,the observed behavior at these points can then be de-scribed by just a few ‘relevant’ parameters (two in theIsing model; the bond strength and the magnetic field).

The Ising model discussed here takes place on a squarelattice (with lattice sites 1 < i, j < L ), with degrees offreedom si,j taking the values of ±1. The probability ofobserving a particular configuration on the whole lattice(denoted by {si,j}) is defined by a Hamiltonian (H {si,j})that assigns each configuration of spins an energy (seeequation 12).

The usual nearest neighbor Ising Model has two pa-rameters: a coupling strength (J), and a magnetic field(h) through the equation:

H({si,j}) = J∑i,j

sijsij+1 + sijsi+1j + h∑i,j

sij (24)

Here we consider a larger dimensional space of possiblemodels, by including in our Hamiltonian the magneticfield (θh), the usual nearest neighbor coupling term, and12 ‘nearby’ couplings parameterized by θαβ . We addi-tionally allow the vertical and horizontal couplings to bedifferent. In the form of equation 12:

H(x) =∑α,β

θαβΦαβ ({si,j}) + θhΦh ({si,j})

Φαβ ({si,j}) =∑i,j

sijsi+αj+β

Φh ({si,j}) =∑i,j

sij

(25)

We calculate the metric along the line through parameterspace that describes the usual Ising model (where θ01 =θ10 = J and θαβ = 0 otherwise) in zero magnetic field(θh = 0).

V. MEASURING THE ISING METRIC

Using equation 13 we can rewrite the metric in termsof expectation values of observables (where except whennecessary we condense the indexes αβ and h into a singleµ).

gµν = ∂µ∂ν log z = 〈ΦµΦν〉 − 〈Φµ〉 〈Φν〉 (26)

Furthermore, given a configuration x = {si,j} we canreadily calculate Φµ(x), which is just a particular twopoint correlation function (or the total sum of spins for

10

Φh) 8.To estimate the distribution defined in equation 26 we

used the Wolff algorithm [13] to very efficiently generatean ensemble of configurations xp = {si,j}p, for 1 < p <

M for systems with L = 64. We also exactly enumeratedall possible states on lattices up to L = 4 to comparewith our Monte-Carlo results (not shown).

With our ensemble of M lattice configurations, xi, wethus measure:

gµν =1

M2 −M

M∑p,q=1,p6=q

Φµ(xp)Φν(xp)− Φµ(xq)Φν(xp)

(27)

FIG. 4: The eigenvalues of the metric for the enlarged 13parameter Ising model described in the text is plotted alongthe line defined by the usual Ising model with βJ as the onlyparameter, and h = 0. Two parameter combinations becomelarge near the critical point, each diverging with character-istic exponents describing the divergence of the susceptibil-ity and specific heat respectively. The other eigenvalues varysmoothly as the critical point is crossed, and furthermore theyhave a characteristic scale and are neither evenly spaced norwidely distributed in log.

The results are plotted in figure 4. Away from the crit-ical point in the high temperature phase (small βJ) theresults seem somewhat analogous to those we found forthe diffusion equation viewed at its microscopic scale. Allof the parameters that control two spin couplings (θαβ)are roughly as distinguishable as each other, with θh hav-ing different units. However, as the critical point is ap-proached, the system becomes extremely sensitive bothto θh and to a certain combination of the θαβ parameters.This divergence has been previously shown for the con-tinuum Ising universality class [10]. In fact, as we will seein the next section, these two metric eigenvalues diverge

8 Φh ({si,j}) =∑i,j si,j is very simple and efficient to calculate

for a given configuration {si,j}. Φαβ ({si,j}) is only slightlyharder. One defines the translated lattice s′i,j(α, β) = si+α,j+β ,

in terms of which we write Φαβ ({si,j}) =∑i,j si,js

′i,j(α, β).

with the scaling of the susceptibility (χ ∼ ξ7/4, whoseeigenvector is simply θh) and specific heat (C ∼ log(ξ),whose eigenvector is a combination of θαβ proportional tothe gradient of the critical temperature, ∂Tc

∂θαβ), respec-

tively. From an information theoretic point of view, thesetwo parameter combinations seem to become particularlyeasy to measure near the critical point because the sys-tem’s behavior becomes extremely sensitive to them. Thebehavior of these two eigenvalues seems to have no par-allel in the diffusion equation viewed at its microscopicscale.

A. Scaling analysis of the Eigenvalue spectrum

To understand our Monte Carlo results for the eigen-values of the metric, we apply a more standard renor-malization group analysis to our calculation. To do thisit is useful to use the form gµν = −∂µ∂νF (see equa-tion 13), and in particular we focus on the critical re-gion, close to the renormalization group fixed point θ0.After a renormalization group transformation that re-duces lengths by a factor of b the remaining degrees offreedom are described by an effective theory with param-eters θ′ related to the original ones by the relationshipθ′µ − θµ0 = Tµν (θν − θν0 )9 where T has left eigenvectorsand eigenvalues given by eLα,µ and byα . It is convenient to

switch to the so-called scaling variables, uα =∑µ e

Lα,µθ

µ,which have the property that under a renormalizationgroup transformation

u′α = byαuα (28)

It is also convenient to divide our free energy into a sin-gular piece and an analytic piece, so that:

F (θ) = Afs(uα(θ)) +Afa(uα(θ))

fs = ud/2y11 U(r0, ..., rα)

rα = uα/uyα/y11

(29)

where fs are free energy densities, A is the system sizeand where fa and U are both analytic functions of theirarguments. Notice that by construction the rs do nottransform under an RG transformation. The Fisher In-formation can be similarly divided into two parts, yield-

9 θ′µ − θµ0 = Tµν (θν − θν0 ) is strictly true only if the parametersspan the space of possible Ising Hamiltonians, but our analysisholds for gµν on the space of the original parameters providedthe θ′ span all possible models, which we can assume in thisanalysis. Said differently, there is no need for T to be square,and it is sufficient for the analysis presented above to assumethat T is 13 by infinite dimensional.

11

ing:

gµν = gsµν + gaµν = −A∂µ∂νfs −A∂µ∂νfa

gsµν = A∑α,β(∂uα∂θµ

∂uβ∂θν )u

(yα+yβ−d)/y11

∂∂rα

∂∂rβU

= A∑α,β

(∂uα∂θµ∂uβ∂θν )Ms

αβ(u)ξyα+yβ−d

gaµν = A∑α,β

∂uα∂θµ

∂uβ∂θν

∂∂uα

∂∂uβ

fa

= A∑α,β

(∂uα∂θµ∂uβ∂θν )Ma

αβ(u)

(30)

where ξ is the correlation length, which diverges

like u−y11 . Both∑α,β(∂uα∂θµ

∂uβ∂θν )Ma

αβ(u) and∑α,β(∂uα∂θµ

∂uβ∂θν )Ms

αβ(u) are tensors in parameterspace with two lower indices that are expected tovary smoothly as their argument is changed, with nodivergent or singular behavior, and eigenvalues that alltake a characteristic scale. As such, we expect that asthe critical point is approached the matrices eigenvalueswill scale like:

λsi ∼ Aξ2yi−d

λai ∼ A(31)

As the critical point is approached we expect the sin-gular piece to dominate provided 2yi − d ≥ 0 . In the2D Ising model, this is true for the magnetic field, whichas the critical point is approached becomes the largesteigenvector e0 = θh (with yh = 15/8) and for the eigen-vector given by e1 = ∂µu1 whose eigenvalue is y1 = 1(in this case 2yi − d = 0 and there is a logarithmic di-vergence, as with the Ising model’s specific heat). Theremaining eigenvectors of gµν are dominated by analyticcontributions. These analytic contributions, just as inthe diffusion equation viewed at its fundamental scale,cause the corresponding eigenvalues to cluster together ata characteristic scale and not exhibit sloppiness (thoughnot necessarily to be exactly the identity). This analysisagrees with the Monte Carlo results plotted in figure 4.

VI. MEASURING THE ISING METRIC AFTERCOARSENING

The diffusion equation became sloppy only after coars-ening. Viewed at its microscopic scale all parameterscould be inferred with exactly the same precision. How-ever, when observed at a time or length scale much largerthan this microscopic scale a hierarchy of importance de-veloped, with particle non-conservation being most vis-ible, drift being the next most dominant term and thediffusion constant being the next most observable pa-rameter. Further parameters became geometrically lessimportant, justifying the use of an effective continuummodel containing just the first of these parameters witha non-zero value.

What happens in the Ising model? Does a similar hier-archy develop? Do the ‘relevant’ parameters in the Ising

model behave differently under coarsening from the ir-relevant ones? To answer these questions we ask howwell we could infer microscopic parameters of the modelfrom data that is coarsened in space10. In particular,we restrict our measurements to observations of spinsthat remain after an iterative checkerboard decimationprocedure11. In the usual RG picture a new effectiveHamiltonian is constructed that describes the observablebehavior at these lattice sites. Here we instead calculatethe Fisher Information Matrix in the original parame-ters, but only using information remaining at the new,coarsened level.

Specifically, we measure gµν = −〈∂µ∂ν log (P (xn))〉where xn = {si,j}for {i,j} in level n. The levels are defined

as follows: If n is even then {i, j} is in level n iff i/2n/2

and j/2n/2 are both integers. If n is odd than {i, j}is in level n if and only if {i, j} is in level n − 1 and(i + j)/2n/2+1 is an integer. The first level is thus acheckerboard, the second has only even sites, the thirdhas a checkerboard of even sites, etc. We define the map-ping to level n, determined by the configuration of allspins x at level 0, as xn = Cn(x)12. It is useful to writeP (xn) in terms of a restricted partition function :

P (xn) = Z̃(xn)/Z

Z̃(xn) =∑x

exp(−H(x))δ(Cn(x) = xn) (32)

where Z̃(xn) is the coarse-grained partition function con-ditioned on the sub-lattice at level n taking the value xn

while summing over the remaining degrees of freedom.We also introduce notation for an expectation value ofan operator defined at level 0 over configurations whichcoarsen to the same configuration xn

{Q}xn =

∑xQ(x)δ(Cn(x) = xn) exp(−H(x))

Z̃(xn)(33)

10 there is no sense of ‘time’ in the Ising model, since it does notspecify dynamics.

11 We use this checkerboard decimation scheme rather than a blockspin scheme (say) as it is easier to implement the CompatibleMonte-Carlo described below.

12 The mapping Cn(x) here simply discards all of the spins that donot remain at level N , leaving an L/2n/2xL/2n/2 square latticefor even N and a rotated ‘diamond’ lattice for odd N . However,this formalism would also apply to other schemes, such as thecommonly used block-spin procedure.

12

We can now rewrite the metric at level n as:

gnµν = −∂µ∂ν⟨

log (P (xn))⟩

= ∂µ∂ν log(Z)−⟨∂µ∂ν log(Z̃(Cn(x)))

⟩= gµν −

⟨{ΦµΦν

}Cn(x)

⟩+⟨{

Φµ}Cn(x)

{Φν}Cn(x)

⟩=⟨{

Φµ}Cn(x)

{Φν}Cn(x)

⟩−⟨{

Φµ}Cn(x)

⟩⟨{Φν}Cn(x)

⟩(34)

This quantity

⟨{Φµ(x)

}Cn(x)

{Φν(x)

}Cn(x)

⟩can be

measured by taking each member of an ensmble, xq, andgenerating a sub-ensemble of x′q,r according to the distri-bution defined by:

P (x′q,r|xq) =

∑x

exp(−H(x))δ(Cn(x′q,r) = Cn(xq))

Z̃(Cn(xq)))(35)

Techniques for generating this ensemble, using a form of‘compatible Monte-Carlo’ [1] are discussed in section VII.From an ensemble of M configurations xq taken from theensemble of full lattice configurations, and xq,r membersof the ensemble given by P (x′q,r|xq) for each xq we cancalculate:

gnµν

= 1(M)(M ′2−M ′)

q=M r,s=M ′∑q,r,s=1r 6=s

(Φµ(x′q,r)Φν(x′q,s)

− 1M−1

M∑p=1 p 6=q

Φµ(x′q,r)Φν(x′p,s)

)(36)

The results of this Monte Carlo presented for a 64×64system at its critical point in figure 3 of the main text.The irrelevant and marginal eigenvalues of the metriccontinue to behave much as the eigenvalues of the met-ric in the diffusion equation, becoming progressively lessimportant under coarsening with characteristic eigenval-ues. However, the large eigenvalues, dominated by singu-lar corrections, do not become smaller under coarsening,presumably because they are measured by their collec-tive effects on the large scale behavior, which is primarilymeasured from large distance correlations.

A. Eigenvalue spectrum after coarse-graining

To understand the values of the metric we observe af-ter coarsening, we apply a more standard RG-like anal-ysis to our system. We do this by constructing an ef-fective Hamiltonian in a new parameter basis, repeatingour analysis for the metric’s eigenvalues in the coordi-nates of the parameters of that Hamiltonian, and finallytransforming back into our original coordinates. After

coarse-graining for n steps each observation yields the

data xn = {si,j}∣∣∣{i,j} in level n

where only the spins {i, j}remaining at level n are observed. The probability ofobserving xn can be written:

P (xn) =exp (−Hn(xn))

Z(An, un)(37)

where Hn is the effective Hamiltonian after n coarse-graining steps. Hn has new parameters most conve-niently written in terms of the scaling variables definedin equation 28 where we can write unα = byαnuα. In addi-tion, the area of the system is reduced to13 An = b−dnAand ∂unα/∂θ

µ = byα∂uα/∂θµ.

After rescaling the entropy of the model is smaller byan amount ∆Sn from the original model’s entropy. It iscustomary in RG analysis to subtract this constant fromthe Hamiltonian, so as to preserve the free energy of thesystem after rescaling:

Fn = Fn,s + Fn,a + ∆Sn = F s + F a = F (38)

Note that the new model’s Hamiltonian would still belinear in these new parameters, allowing us to use thealgebra of equation 13, if we were to remove the constant∆S from the new Hamiltonian. This would of coursebe an identical model, since the addition of a constant tothe energy does not change any observables. This changeallows us to express the metric for the new observablesin terms of the original parameters, taking

gnµν(θ) = ∂µ∂ν(Fn,s+Fn,a) = ∂µ∂ν(F s+F a−∆S) (39)

After some algebra we see that:

gs,nµν = ∂µ∂νFn,s = ∂µ∂νF

s = gsµνga,nµν = ∂µ∂νF

n,a = b−dnA∂µ∂νfn,a

= b−dnA∂unα∂θµ

∂unβ∂θνM

aαβ(un)

= A∑α,β b

(yα+yβ−d)n(∂uα∂θµ∂uβ∂θν )Ma

αβ(un)

(40)

The singular piece is exactly maintained as the sin-gular part of the free energy is preserved after an RGstep. This means that the singular piece of the freeenergy is exactly the piece which describes informationcarried in long wave-length information. On the otherhand, the analytic piece is smaller by ∂µ∂ν∆Sn. The

matrix (∂uα∂θµ∂uβ∂θν )Ma

αβ(un) should be smoothly varying,as un varies a small amount with n. Importantly, all ofits eigenvalues should continue to take a characteristicvalue. Thus, after n rescalings:

13 we keep our rescaling factor b general here, but in our systemb =√

2

13

λn,si ∼ A(ξ)2yi−d

λn,ai ∼ Abn(2yi−d) (41)

To ensure that the Fisher information is strictly de-creasing in every direction on coarsening 14 gaµν must benegative semidefinite in the subspace of scaling variableswhere 2yi − d > 0. For these relevant directions, withi = 0, 1 λni ∼ Aξ2yi−d−Ab2yi−dn, where the second termonly becomes significant when bn ∼ ξ (when the latticespacing is comparable to the correlation length). For ir-relevant directions, or relevant ones with 0 < 2yi < d(corresponding to i ≥ 2 in the Ising model), the analyticpiece will dominate as the critical point is approached,yielding λi ∼ Ab2yi−d. These results are in quantita-tive agreement with those plotted in figure 3 of the maintext assuming that our variables project onto irrelevantand marginal scaling variables with leading dimensions ofy = 0 (blue line in figure 3 of main text), y = −2 (greenline in figure 3 of the main text) and y = −4 (purple linein figure 3 of the main text) consistent with the theo-retical predictions for the irrelevant eigenvalue spectrummade in [14].

VII. SIMULATION DETAILS

To generate ensembles xp that are used to calculatethe metric before coarsening we use the standard Wolffalgorithm [13], implemented on 64x64 periodic squarelattices. We generate M = 10, 000 − 100, 000 indepen-dent members from each ensemble, and calculate gµν asdescribed above.

To generate members of the ensemble defined by eq. 35we use variations on a method introduced in [1] whichthey termed ‘compatible Monte-Carlo’15. Essentially, aMonte-Carlo chain is run with any move which proposesa switch to a configuration x′p,r for which Cn(x′p,r) 6=Cn(xp) is summarily rejected. Given our mapping,Cn(xp) = Cn(xp,r) this rule is easy to enforce. In thesimplest iteration we can equilibrate using Metropolismoves, but only proposing spins which are not in leveln. We introduce several additional tricks to speed upconvergence which we now describe.

Consider the task of generating a random member x′p,rfor a given xp at level 1. Because the spins which are freeto move only make contact with fixed spins, each one can

14 In each coarsening step gnµν−gn+1µν must be a positive semidefinite

matrix. This is because no parameter combinations can be moremeasurable from a subset of the data available at level n thanfrom its entirety.

15 Ron, Swendsen and Brandt used this technique for entirely differ-ent purposes. They generated large equilibrated ensembles closeto the critical point, essentially by starting from a small ‘coars-ened’ lattice and iteratively adding layers to generate a largeensemble.

be chosen independently. As such, if we choose each ‘free’spin according to its heat bath probability then we arriveat an uncorrelated member xp,r of the ensemble definedby xp.

This trick can be further exploited to exactly calculatethe contribution to a metric element at level 1 from alevel 0 configuration x. In particular, by replacing all ofthe spins that are not in level 1 with their mean fieldvalues, defined by s̃i,j(x) = {si,j}Cn(x) (which we can

calculate in a single step) we can immediately write:

{Φαβ}Cn(x) =∑i,j

s̃i,j(x)s̃i+α,j+β(x)

{Φh}Cn(x) =∑i,j

s̃i,j(42)

As such, it is possible to exactly calculate the level

one quantities{

Φµ

}C1(x)

{Φν

}C1(x)

for any microscopic

configuration x and corresponding checkerboard configu-ration C1(x). We can write the metric at level 1 as

g1µν = 1

M2−M

M∑p,q=1,p6=q

({Φµ}C1(xp) {Φν}C1(xp)

−{Φµ}C1(xp) {Φν}C1(xq)

)(43)

Beyond level 1 it becomes necessary to use compati-ble Monte-Carlo, but we can still take advantage of theindependence of the free spins at level 1. In particular,spins at all levels n ≥ 1 only interact with spins that arealready absent at level 1. We continue to leave the spinsthat are free at level 1 (henceforth the red sites, fromtheir color on a checkerboard) integrated out. This par-tition function is most conveniently written in terms ofthe number of up neighbors, nupi,j that each red site has:

log Z̃(C1(x)) =∑

i,j not in level 1

log (z(nupi,j))

z(nup) = cosh ((βJ)(2− nup))(44)

Additional spins that are not integrated out at leveln are flipped using a heat bath algorithm with the ra-tio of partition functions in an ‘up’ vs ‘down’ configura-tion used to determine the transition probability. Theprobability of a spin (at level ≥ 2) transitioning to ’up’after being proposed from the down state is given byzupi,j/(z

upi,j + zdowni,j ) with

zupi,j =∑

{k,l} n.n. of {i,j}z(nupk,l + 1)

zdowni,j =∏

{k,l} n.n. of {i,j}z(nupk,l)

(45)

Equilibration is extremely fast as their are effectivelyno correlations larger than the spacing between fixedspins at level n. This allows us to generate an ensem-ble of lattice configurations at level 1, conditioned on the

14

system coarsening to an arbitrary configuration at an ar-bitrary level n > 1. As such, for efficiency we slightlymodify equation 36 to

gnµν =

1(M)(M ′2−M ′)

q=M r,s=M ′∑q,r,s=1r 6=s

({Φµ

}c1(x′

q,r)

{Φν

}c1(x′

q,s)

− 1M−1

M∑p=1 p 6=q

{Φµ

}c1(x′

q,r)

{Φν

}c1(x′

p,s)

)(46)

This is used to produce figure 3 for data at level 2 andhigher.

[1] D. Ron, R. Swendsen, A. Brandt, Physical Review Letters89 (2002).

[2] C. E. Shannon, Bell System Technical Journal 27, 379(1948).

[3] S. Amari, H. Nagaoka, Methods of Information Geome-try , Translations of Mathematical Monographs (Ameri-can Mathematical Society, 2000).

[4] T. Cover, J. Thomas, Elements of Information theory(Wiley Interscience, New York, NY, 1991).

[5] S. Kullback, R. Leibler, Annals Of Mathematical Statis-tics 22, 79 (1951).

[6] M. K. Transtrum, B. B. Machta, J. P. Sethna, Phys. Rev.Lett. 104, 060201 (2010).

[7] M. K. Transtrum, B. B. Machta, J. P. Sethna, Phys. Rev.

E 83, 036701 (2011).[8] R. N. Gutenkunst, et al., PLoS Computational Biology

3, 1871 (2007).[9] G. E. Crooks, Phys. Rev. Lett. 99 (2007).

[10] G. Ruppeiner, Reviews Of Modern Physics 67 (1995).[11] C. Jarzynski, Phys. Rev. Lett. 78 (1997).[12] G. Arfken, H. Weber, Mathematical Methods for Physi-

cists, International paper edition (Academic Press,2001).

[13] U. Wolff, Phys. Rev. Lett. 62, 361 (1989).[14] M. Caselle, M. Hasenbusch, A. Pelissetto, E. Vicari,

Journal of Physics A 35, 4861 (2002).