Module #6 – Fundamental Physical Limits of Computing

Module #6 – Fundamental Module #6 – Fundamental Physical Limits of ComputingPhysical Limits of Computing

A Brief SurveyA Brief Survey

Fundamental Physical Limits of ComputingFundamental Physical Limits of Computing

Speed-of-LightLimit

Thoroughly Confirmed

Physical Theories

UncertaintyPrinciple

Definitionof Energy

Reversibility

2nd Law ofThermodynamics

Adiabatic Theorem

Gravity

Theory ofRelativity

QuantumTheory

ImpliedUniversal Facts

Affected Quantities in Information Processing

Communications Latency

Information Capacity

Information Bandwidth

Memory Access Times

Processing Rate

Energy Loss per Operation

A Slightly More Detailed ViewA Slightly More Detailed View

The Speed-of-Light Limit on The Speed-of-Light Limit on Information Propagation Velocity Information Propagation Velocity

What are its implications for future What are its implications for future computer architectures?computer architectures?

Implications for ComputingImplications for Computing• Minimum communications latency!Minimum communications latency!• Minimum memory-access latency!Minimum memory-access latency!• Need for Processing-in-Memory architectures!Need for Processing-in-Memory architectures!• Mesh-type topologies are optimally scalable!Mesh-type topologies are optimally scalable!

– Hillis, Vitanyi, Bilardi & PreparataHillis, Vitanyi, Bilardi & Preparata

• Together w. 3-dimensionality of space implies:Together w. 3-dimensionality of space implies:– No network topology with No network topology with ((nn33) ) connectivityconnectivity

(# nodes reachable in (# nodes reachable in nn hops) is scalable! hops) is scalable!– Meshes w. 2-3 dimensions are optimally scalable.Meshes w. 2-3 dimensions are optimally scalable.

• Precise number depends on reversible computing theory!Precise number depends on reversible computing theory!

How Bad it Is, AlreadyHow Bad it Is, Already• Consider a 3.2 GHz processor (off today’s shelf)Consider a 3.2 GHz processor (off today’s shelf)

– In 1 cycle a signal can propagate In 1 cycle a signal can propagate at mostat most::• cc/(3.2 GHz) = 9.4 cm/(3.2 GHz) = 9.4 cm

– For a 1-cycle round-trip to cache memory & back:For a 1-cycle round-trip to cache memory & back:• Cache location can be Cache location can be at mostat most 4.7 cm away! 4.7 cm away!

– Electrical signals travel at ~0.5 Electrical signals travel at ~0.5 cc in typical materials in typical materials• In practice, a 1-cycle memory can be at most 2.34 cm In practice, a 1-cycle memory can be at most 2.34 cm

away!away!

• Already Already logics in labs at 100 GHz speeds! logics in labs at 100 GHz speeds!– E.g.E.g., superconducting logic technology: , superconducting logic technology: RSFQRSFQ– 1-cycle round trips only within 1.5 mm!1-cycle round trips only within 1.5 mm!

• Much smaller than a typical chip diameter!Much smaller than a typical chip diameter!

• As As ff, architectures must be increasingly , architectures must be increasingly locallocal..

• Avg. time to randomly access anyAvg. time to randomly access anyone of one of nn bits of storage (accessible bits of storage (accessibleinformation) scales as information) scales as ((nn1/31/3).).

• This This willwill remain true in remain true in allall future technologies! future technologies!– Quantum mechanics gives a minimum size for bitsQuantum mechanics gives a minimum size for bits

• Esp. assuming temperature & pressure are limited.Esp. assuming temperature & pressure are limited.– Thus Thus nn bits require a bits require a ((nn)-volume region of space.)-volume region of space.– Minimum diameter of this region is Minimum diameter of this region is ((nn1/31/3). ). – At lightspeed, random access takes At lightspeed, random access takes ((nn1/31/3) time!) time!

• Assuming a non-negative curvature region of spacetime.Assuming a non-negative curvature region of spacetime.• Of course, specific memory technologies (or a suite of Of course, specific memory technologies (or a suite of

available technologies) may scale even worse than this!available technologies) may scale even worse than this!

(n1/3)

Latency Scaling w. Memory SizeLatency Scaling w. Memory Size

n bits

Scalability & Maximal ScalabilityScalability & Maximal Scalability• A multiprocessor architecture & accompanying A multiprocessor architecture & accompanying

performance model is performance model is scalablescalable if: if:– it can be “scaled up” to arbitrarily large problem sizes, and/or it can be “scaled up” to arbitrarily large problem sizes, and/or

arbitrarily large numbers of processors, without the predictions arbitrarily large numbers of processors, without the predictions of the performance model breaking down.of the performance model breaking down.

• An architecture (& model) is An architecture (& model) is maximally scalablemaximally scalable for a for a given problem ifgiven problem if– it is scalable, it is scalable, andand if no other scalable architecture can claim if no other scalable architecture can claim

asymptotically superior performance on that problemasymptotically superior performance on that problem

• It is It is universally maximally scalableuniversally maximally scalable (UMS) if it is (UMS) if it is maximally scalable on maximally scalable on allall problems! problems!– I will briefly mention some characteristics of architectures that I will briefly mention some characteristics of architectures that

are universally maximally scalableare universally maximally scalable

Shared Memory isn’t ScalableShared Memory isn’t Scalable• Any implementation of shared memory Any implementation of shared memory requires requires

communicationcommunication between nodes. between nodes.• As the # of nodes increases, we get:As the # of nodes increases, we get:

– Extra contention for any shared BWExtra contention for any shared BW– Increased latency (inevitably).Increased latency (inevitably).

• Can hide communication delays Can hide communication delays to a limited to a limited extent,extent, by by latency hiding:latency hiding: – Find other work to do during the latency delay slot.Find other work to do during the latency delay slot.– But, the amount of “other work” available is limited by But, the amount of “other work” available is limited by

node storage capacity, parallizability of the set of node storage capacity, parallizability of the set of running applications, running applications, etc.etc.

Unit-Time Message Passing Unit-Time Message Passing Isn’t ScalableIsn’t Scalable

• Model: Any node can pass a message to any other Model: Any node can pass a message to any other in a single constant-time interval (independent of in a single constant-time interval (independent of the total number of nodes)the total number of nodes)

• Same scaling problems as shared memory!Same scaling problems as shared memory!• Even if we assume BW contention (traffic) isn’t a Even if we assume BW contention (traffic) isn’t a

problem, unit-time assumption is still a problem.problem, unit-time assumption is still a problem.– Not possible for all Not possible for all NN, given speed-of-light limit!, given speed-of-light limit!– Need cube root of Need cube root of NN asymptotic time, at minimum. asymptotic time, at minimum.

Many Interconnect Topologies aren’t ScalableMany Interconnect Topologies aren’t Scalable

• Suppose we don’t require a node can talk to Suppose we don’t require a node can talk to anyany other in 1 time unit, but only to other in 1 time unit, but only to selectedselected others. others.

• Some such schemes Some such schemes stillstill have scalability have scalability problems, problems, e.g.e.g.::– Hypercubes Hypercubes – Binary trees, fat treesBinary trees, fat trees– Butterfly networksButterfly networks

• Any topology in which the number of unit-time Any topology in which the number of unit-time hops to reach any one of N nodes is of order less hops to reach any one of N nodes is of order less than Nthan N1/31/3 is necessarily doomed to failure! is necessarily doomed to failure!– Caveat: Except in negative-curvature spacetimes!Caveat: Except in negative-curvature spacetimes!

Only Meshes (or subgraphs Only Meshes (or subgraphs thereof) Are Scalablethereof) Are Scalable

See papers by Hillis, Vitanyi, Bilardi & PreparataSee papers by Hillis, Vitanyi, Bilardi & Preparata• 1-D meshes1-D meshes

– linear chain, ring, star (w. fixed # of arms)linear chain, ring, star (w. fixed # of arms)

• 2-D meshes2-D meshes– square grid, hex grid, cylinder, 2-sphere, 2-torus, …square grid, hex grid, cylinder, 2-sphere, 2-torus, …

• 3-D meshes3-D meshes– crystal-like lattices w. various symmetriescrystal-like lattices w. various symmetries– Caveat:Caveat:

• Scalability in 3rd dimension is limited by Scalability in 3rd dimension is limited by energy/information I/O considerations!energy/information I/O considerations!

Amorphousarrangements

in 3d, w. localcomms., are

also ok

Ideally Scalable ArchitecturesIdeally Scalable Architectures

CPU

Local memory hierarchy(optimal fixed size)

Processing Node

CPU


Processing Node

CPU


Processing Node

CPU


Processing Node

CPU


Processing Node

CPU


Processing Node

Claim: A 2- or 3-D mesh multiprocessor with a fixed-size memory hierarchy per node is an optimal scalable computer systems design (for any application).

Mesh interconnection network

Landauer’s PrincipleLandauer’s Principle• Low-level physics is Low-level physics is reversiblereversible

– Means, the time-evolution of a state is bijectiveMeans, the time-evolution of a state is bijective– Deterministic looking backwards in timeDeterministic looking backwards in time

• as well as forwardsas well as forwards

• Physical information (like energy) is Physical information (like energy) is conservedconserved– Cannot be created or destroyed, only reversibly rearranged Cannot be created or destroyed, only reversibly rearranged

and modifiedand modified– Implies the 2nd Law of Thermodynamics:Implies the 2nd Law of Thermodynamics:

• Entropy (unknown info.) in a closed, unmeasured system can Entropy (unknown info.) in a closed, unmeasured system can only increase (as we lose track of the state)only increase (as we lose track of the state)

• Irreversible bit “erasure” really just moves the bit into Irreversible bit “erasure” really just moves the bit into surroundings, increasing entropy & heatsurroundings, increasing entropy & heat

Scaling in 3rd Dimension?Scaling in 3rd Dimension?• Computing based on ordinary Computing based on ordinary irreversibleirreversible bit bit

operations only scales in 3d operations only scales in 3d up to a pointup to a point..– Discarded information & associated energy must be Discarded information & associated energy must be

removed thru surface. Energy flux limited.removed thru surface. Energy flux limited.– Even a Even a singlesingle layer of circuitry in a high-performance layer of circuitry in a high-performance

CPU can barely be kept cool today!CPU can barely be kept cool today!• Computing with Computing with reversiblereversible, adiabatic operations , adiabatic operations

does better:does better:– Scales in 3d, up to a pointScales in 3d, up to a point– Then with Then with square rootsquare root of further increases in thickness, of further increases in thickness,

up to a point. (Scales in 2.5 dimensions!)up to a point. (Scales in 2.5 dimensions!)– Scales to much larger thickness than irreversible!Scales to much larger thickness than irreversible!

Universal Maximum ScalabilityUniversal Maximum Scalability• Existence proof for universally maximally Existence proof for universally maximally

scalable (UMS) architectures:scalable (UMS) architectures:– Physics itselfPhysics itself is a universal maximally scalable is a universal maximally scalable

“architecture” because “architecture” because anyany real computer real computer isis merely a merely a special case of a physical system.special case of a physical system.

• Obviously, no restricted class of real computers can beat the Obviously, no restricted class of real computers can beat the performance scalability of physical systems in general.performance scalability of physical systems in general.

– Unfortunately, physics doesn’t give us a very simple Unfortunately, physics doesn’t give us a very simple or convenient programming model.or convenient programming model.

• Comprehensive expertise at “programming physics” means Comprehensive expertise at “programming physics” means mastery of mastery of allall physical engineering disciplines: chemical, physical engineering disciplines: chemical, electrical, mechanical, optical, electrical, mechanical, optical, etc.etc.

– We’d like an easier programming model than this!We’d like an easier programming model than this!

Simpler UMS ArchitecturesSimpler UMS Architectures• (I propose) any practical UMS architecture will (I propose) any practical UMS architecture will

have the following features:have the following features:– Processing elements characterized by constant Processing elements characterized by constant

parameters (independent of # of processors)parameters (independent of # of processors)• Makes it easy to scale multiprocessors to large capacities.Makes it easy to scale multiprocessors to large capacities.

– Mesh-type message-passing interconnection network, Mesh-type message-passing interconnection network, arbitrarily scalable in 2 dimensionsarbitrarily scalable in 2 dimensions

• w. limited scalability in 3rd dimension.w. limited scalability in 3rd dimension.– Processing elements that can be operated in an Processing elements that can be operated in an

arbitrarily reversible way, at least, up to a point.arbitrarily reversible way, at least, up to a point.• Enables improved 3-d scalability in a limited regimeEnables improved 3-d scalability in a limited regime

– (In long term) Have capability for quantum-coherent (In long term) Have capability for quantum-coherent operation, for extra perf. on some probs.operation, for extra perf. on some probs.

Limits on Amount of Limits on Amount of Information ContentInformation Content

Some Quantities of InterestSome Quantities of Interest• We would like to know if there are limits on:We would like to know if there are limits on:

– Information densityInformation density• = Bits per unit volume= Bits per unit volume• Affects physical size and thus propagation delayAffects physical size and thus propagation delay

across memories and processors. Also affects cost.across memories and processors. Also affects cost.– Information fluxInformation flux

• = Bits per unit area per unit time= Bits per unit area per unit time• Affects cross-sectional bandwidth, data I/O rates, rates of Affects cross-sectional bandwidth, data I/O rates, rates of

standard-information input & effective-entropy removalstandard-information input & effective-entropy removal– Rate of computationRate of computation

• = Number of distinguishable-state changes per unit time= Number of distinguishable-state changes per unit time• Affects rate of information processing achievable in individual Affects rate of information processing achievable in individual

devicesdevices

Bit Density: No classical limitBit Density: No classical limit• In classical (continuum) physics, even a In classical (continuum) physics, even a singlesingle

particle has a real-valued position+momentumparticle has a real-valued position+momentum– All such states are considered physically distinctAll such states are considered physically distinct– Each position & momentum coordinate in general requires Each position & momentum coordinate in general requires

an an infiniteinfinite string of digits to specify: string of digits to specify:• xx = 4.592181291845019587661625618991009… meters = 4.592181291845019587661625618991009… meters• pp = 2.393492301938881726153514427394001… kg m/s = 2.393492301938881726153514427394001… kg m/s

– Even the smallest system contains an infinite amount of Even the smallest system contains an infinite amount of information! information! No limit to bit density. No limit to bit density.

– This picture is the basis for various This picture is the basis for various analog computinganalog computing models studied by some theoreticians.models studied by some theoreticians.

• Wee problem: Wee problem: Classical physics is dead wrong!Classical physics is dead wrong!

The Quantum “Continuum”The Quantum “Continuum”• In QM, still In QM, still uncountably many uncountably many describabledescribable

states (mathematically possible wavefunctions)states (mathematically possible wavefunctions)– Can theoretically take infinite info. to describeCan theoretically take infinite info. to describe

• But, not all this info has physical relevance!But, not all this info has physical relevance!– States are only physically States are only physically distinguishabledistinguishable when their when their

state vectors are state vectors are orthogonalorthogonal..– States that are only indistinguishably different can only States that are only indistinguishably different can only

lead to indistinguishably different consequences lead to indistinguishably different consequences (resulting states)(resulting states)

• due to linearity of quantum physicsdue to linearity of quantum physics– There is There is no physical consequenceno physical consequence from presuming an from presuming an

infinite # of bits in one’s wavefunction!infinite # of bits in one’s wavefunction!

Quantum Particle-in-a-BoxQuantum Particle-in-a-Box• Uncountably manyUncountably many

continuouscontinuouswavefunctions?wavefunctions?

• No, can expressNo, can expresswave as a vectorwave as a vectorover countablyover countablymany orthogonalmany orthogonalnormal modesnormal modes..– Fourier transformFourier transform

• High-frequencyHigh-frequencymodes have highermodes have higherenergy (energy (E=hfE=hf); a); alimit on average limit on average energy impliesenergy impliesthey have low they have low probability.probability.

Ways of Counting StatesWays of Counting StatesThe entire field of quantum statistical mechanics is The entire field of quantum statistical mechanics is

all about this, but here are some simple ways:all about this, but here are some simple ways:• For a system w. a constant # of particles:For a system w. a constant # of particles:

– # of states = numerical volume of the position-# of states = numerical volume of the position-momentum configuration space (momentum configuration space (phase spacephase space))

• When measured in units where When measured in units where hh=1.=1.• Exactly approached in the macroscopic limit.Exactly approached in the macroscopic limit.

– Unfortunately, # of particles is not usually constant!Unfortunately, # of particles is not usually constant!• Quantum field theory bounds:Quantum field theory bounds:

– Smith-Lloyd bound. Still ignores gravity.Smith-Lloyd bound. Still ignores gravity.• General relativistic bounds:General relativistic bounds:

– Bekenstein bound, holographic bound.Bekenstein bound, holographic bound.

Smith-Lloyd BoundSmith-Lloyd Bound

• Based on counting modes of quantum fields.Based on counting modes of quantum fields.– SS = entropy, = entropy, MM = mass, = mass, VV = volume = volume– qq = number of distinct particle types = number of distinct particle types

• Lloyd’s bound is tighter by a factor of Lloyd’s bound is tighter by a factor of • Note:Note:

– Maximum entropy density scales with only the 3/4 Maximum entropy density scales with only the 3/4 power of mass-energy density!power of mass-energy density!

• E.g.E.g., Increasing entropy density by a factor of 1,000 , Increasing entropy density by a factor of 1,000 requires increasing energy density by 10,000×.requires increasing energy density by 10,000×.

4/3

4/1

4/1

603

16

2

V

Mcq

V

S

22

Smith ‘95Lloyd ‘00

Whence this scaling relation?Whence this scaling relation?• Note that in the field theory limit, Note that in the field theory limit, SS EE3/43/4..

– Where does the ¾ power come from?Where does the ¾ power come from?

• Consider a typical mode in field spectrumConsider a typical mode in field spectrum– Note that the minimum size of aNote that the minimum size of a

given wavelet is ~its wavelength given wavelet is ~its wavelength ..

• # of distinguishable wave-packet location states # of distinguishable wave-packet location states in a given volume in a given volume 1/ 1/33

– Each such state carries just a little entropyEach such state carries just a little entropy• occupation number of that state (# of photons in it)occupation number of that state (# of photons in it)

1/1/33 particles each energy particles each energy 1/1/, , 1/1/4 4 energyenergy• SS1/1/33 EE1/1/44 SSEE3/43/4

Whence the distribution?Whence the distribution?• Could the use of more particles (with less energy Could the use of more particles (with less energy

per particle) yield greater entropy?per particle) yield greater entropy?– What frequency spectrum (power level or particle What frequency spectrum (power level or particle

number density as a function of frequency) gives the number density as a function of frequency) gives the largest # states?largest # states?

– Note Note a minimum particle energy in finite-sized box a minimum particle energy in finite-sized box

• No. The Smith-Lloyd bound is based on the No. The Smith-Lloyd bound is based on the blackbodyblackbody radiation spectrum. radiation spectrum.– We know this spectrum has the maximum info. content We know this spectrum has the maximum info. content

among abstract states, b/c it’s the equilibrium state!among abstract states, b/c it’s the equilibrium state!• Empirically verified in hot ovens, Empirically verified in hot ovens, etc.etc.

Examples w. Smith-Lloyd BoundExamples w. Smith-Lloyd Bound• For systems at the density of water (1 g/cmFor systems at the density of water (1 g/cm33), ),

composed only of photons:composed only of photons:– Smith’s example: 1 mSmith’s example: 1 m33 box holds 6×10 box holds 6×103434 bits bits

• = 60 kb/Å= 60 kb/Å33

– Lloyd’s example: 1 liter “ultimate laptop”, 2×10Lloyd’s example: 1 liter “ultimate laptop”, 2×103131 b b• = 21 kb/Å= 21 kb/Å33

• Pretty high, but what’s wrong with this picture?Pretty high, but what’s wrong with this picture?– Example requires very high temperature+pressure!Example requires very high temperature+pressure!

• Temperature around 1/2 billion Kelvins!!Temperature around 1/2 billion Kelvins!!• Photonic pressure on the order of 10Photonic pressure on the order of 101616 psi!! psi!!

– ““Like a miniature piece of the big bang.” -LloydLike a miniature piece of the big bang.” -Lloyd– Probably not feasible to implement any time soon!Probably not feasible to implement any time soon!

More Normal TemperaturesMore Normal Temperatures• Let’s pick a more reasonable temperature: Let’s pick a more reasonable temperature:

1356 K (melting point of copper):1356 K (melting point of copper):– The entropy density of light is only 0.74 bits/The entropy density of light is only 0.74 bits/mm33!!

• Less than the bit density in a DRAM today!Less than the bit density in a DRAM today!– Bit size is comparable to avg. wavelength of optical-Bit size is comparable to avg. wavelength of optical-

frequency light emitted by melting copperfrequency light emitted by melting copper

• Lesson: Lesson: Photons are not a viable nanoscale info. Photons are not a viable nanoscale info. storage medium at ordinary temperatures.storage medium at ordinary temperatures.– They simply aren’t dense enough!They simply aren’t dense enough!

• CPUs that do logic with optical photons can’t have CPUs that do logic with optical photons can’t have their logic devices packed very densely.their logic devices packed very densely.

Entropy Density of SolidsEntropy Density of Solids• Can easily calculate from standard empirical Can easily calculate from standard empirical

thermochemical data. thermochemical data. – E.g.E.g. see CRC Handbook of Chemistry & Physics. see CRC Handbook of Chemistry & Physics.

• Obtain entropy by integrating heat capacity ÷ Obtain entropy by integrating heat capacity ÷ temperature, as temperature increases…temperature, as temperature increases…– Example result, for copper:Example result, for copper:

• Has one of the highest entropy densities among pure Has one of the highest entropy densities among pure elements, at atmospheric pressure.elements, at atmospheric pressure.

• @ room temperature: 6 bits/atom, 0.5 b/Å@ room temperature: 6 bits/atom, 0.5 b/Å33

• At boiling point: 1.5 b/ÅAt boiling point: 1.5 b/Å33

– Cesium has one of the highest #bits/atom at room Cesium has one of the highest #bits/atom at room temperature, about 15. temperature, about 15.

• But, only 0.13 b/ÅBut, only 0.13 b/Å33

– Lithium has a high #bits/mass, 0.7 bits/amu.Lithium has a high #bits/mass, 0.7 bits/amu.

Related toconductivity?

1012×denser

thanits light!

General-Relativistic BoundsGeneral-Relativistic Bounds• Note: the Smith-Lloyd bound does not take into Note: the Smith-Lloyd bound does not take into

account the effects of general relativity.account the effects of general relativity.• Earlier bound from Bekenstein: Derives a limit on Earlier bound from Bekenstein: Derives a limit on

entropy from black-hole physics:entropy from black-hole physics:SS < (2 < (2ER ER / / cc) nats) nats

EE = total energy of system = total energy of systemRR = radius of the system (min sphere) = radius of the system (min sphere)

• Limit only attained by black holes!Limit only attained by black holes!– Black holes have 1/4 nat entropy per square Planck Black holes have 1/4 nat entropy per square Planck

length of surface (event horizon) area.length of surface (event horizon) area.• Absolute minimum size of a nat: 2 Planck lengths, squareAbsolute minimum size of a nat: 2 Planck lengths, square

4×1039 b/Å3

average ent. dens.of a 1-m radius

black hole!(MassSaturn)

The Holographic BoundThe Holographic Bound• Based on Bekenstein black-hole bound.Based on Bekenstein black-hole bound.• The information content The information content II within within anyany surface of surface of

area area AA (independent of its energy content!) is (independent of its energy content!) is II ≤ ≤ AA/(2/(2PP))22 nats nats

PP is the Planck length (see lecture on units) is the Planck length (see lecture on units)

• Implies that any 3D object (of any size) is Implies that any 3D object (of any size) is completely definable via a flat (2D) “hologram” completely definable via a flat (2D) “hologram” on its surface having Planck-scale resolution.on its surface having Planck-scale resolution.– This information is all entropy only in the case a This information is all entropy only in the case a

black hole with event horizon=that surface.black hole with event horizon=that surface.

Holographic Bound ExampleHolographic Bound Example• The age of the universe is 13.7 Gyr The age of the universe is 13.7 Gyr ±1% [WMAP].±1% [WMAP].

– Radius of observed part would thus be 13.7 Glyr…Radius of observed part would thus be 13.7 Glyr…• But, due to expansion, it is actually closer to 40 Glyr today.But, due to expansion, it is actually closer to 40 Glyr today.

• The universe is “flat,” so Euclidean formulas apply:The universe is “flat,” so Euclidean formulas apply:– The surface area of the observable universe is: The surface area of the observable universe is:

• AA = 4 = 4ππrr22 = 4 = 4ππ(40 Glyr)(40 Glyr)22 = 1.80×10 = 1.80×105454 m m22

– The volume of observable universe is:The volume of observable universe is:• VV = (4/3) = (4/3)ππrr33 = (4/3) = (4/3)ππ(40 Glyr)(40 Glyr)33 = 2.27×10 = 2.27×108080 m m33

• Now, we can calculate the universe’s total info. Now, we can calculate the universe’s total info. content, and its average information density!content, and its average information density!– II = = AAn/4n/4PP

22 = ( = (ππrr22//PP22) n = 1.72×10) n = 1.72×10123123 n = 2.49×10 n = 2.49×10123123 b b

– II//VV = 1.10×10 = 1.10×104343 b/m b/m33 = 0.01 b/fm = 0.01 b/fm33 = 1b/(.22fm) = 1b/(.22fm)33

• A proton is ~1 fm in radius. Very close to 1 b/proton-volume!A proton is ~1 fm in radius. Very close to 1 b/proton-volume!

Do Black Holes Destroy Information?Do Black Holes Destroy Information?• Currently, it seems that no one completely understands Currently, it seems that no one completely understands

exactly exactly howhow information is preserved during black hole information is preserved during black hole accretion, for later re-emission in the Hawking radiation.accretion, for later re-emission in the Hawking radiation.– Perhaps via infinite time dilation at event horizon?Perhaps via infinite time dilation at event horizon?

• Some researchers have claimed that black holes Some researchers have claimed that black holes must be doing something irreversible in their must be doing something irreversible in their interior (destroying information).interior (destroying information).– However, the arguments for this may not be valid.However, the arguments for this may not be valid.– Recent string theory calculations contradict this claim.Recent string theory calculations contradict this claim.

• The issue seems not yet fully resolved, but I have The issue seems not yet fully resolved, but I have many references on it if you’re interested.many references on it if you’re interested.

Implications of Info. Density LimitsImplications of Info. Density Limits• There is a minimum size for a bit-device.There is a minimum size for a bit-device.

– thus there is a minimum communication latency to thus there is a minimum communication latency to randomly access a memory containing randomly access a memory containing nn bits bits

• as we discussed earlier.as we discussed earlier.– There is also a minimum cost per bit, if there is a There is also a minimum cost per bit, if there is a

minimum cost per unit of matter/energy.minimum cost per unit of matter/energy.

• Implications for communications bandwidth Implications for communications bandwidth limits…limits…– coming upcoming up


– Information densityInformation density• = Bits per unit volume= Bits per unit volume• Affects physical size and thus propagation delayAffects physical size and thus propagation delay

across memories and processors. Also affects cost.across memories and processors. Also affects cost.– Information fluxInformation flux


standard-information input & effective entropy removalstandard-information input & effective entropy removal– Rate of computationRate of computation


devicesdevices

Communication LimitsCommunication Limits• LatencyLatency (propagation-time delay) limit from (propagation-time delay) limit from

earlier, due to speed of light.earlier, due to speed of light.– Teaches us scalable interconnection technologiesTeaches us scalable interconnection technologies

• BandwidthBandwidth (information rate) limits: (information rate) limits:– Classical information-theory limit (Shannon)Classical information-theory limit (Shannon)

• Limit, per-channel, given signal bandwidth & SNR.Limit, per-channel, given signal bandwidth & SNR.– Limits based on field theory (Smith/Lloyd)Limits based on field theory (Smith/Lloyd)

• Limit given only area and power.Limit given only area and power.• Applies to I/O, cross-sectional bandwidths in parallel Applies to I/O, cross-sectional bandwidths in parallel

machines, and entropy removal rates.machines, and entropy removal rates.

Hartley-Shannon LawHartley-Shannon Law• The maximum information rate (capacity) of a The maximum information rate (capacity) of a

single wave-based communication channel is:single wave-based communication channel is:CC = = BB log (1+ log (1+SS//NN))

– BB = bandwidth of channel, in frequency units = bandwidth of channel, in frequency units– SS = signal power level = signal power level– NN = noise power level = noise power level

• Law not sufficiently powerful for our purposes!Law not sufficiently powerful for our purposes!– Does not tell us Does not tell us how manyhow many effective channels are effective channels are

possible, given available power and/or area.possible, given available power and/or area.– Does not give us Does not give us anyany limit if we are allowed to increase limit if we are allowed to increase

bandwidth or decrease noise arbitrarily.bandwidth or decrease noise arbitrarily.

Density & FluxDensity & Flux• Note that any time you have:Note that any time you have:

– a limit a limit on density (per volume) of something, on density (per volume) of something,– & a limit & a limit vv on its propagation velocity, on its propagation velocity,

• this automatically implies:this automatically implies:– a limit a limit FF = = vv on the on the fluxflux

• by which I mean amount per time per areaby which I mean amount per time per area

• Note also we always have a limit Note also we always have a limit cc on velocity! on velocity!– At speeds near At speeds near cc must account for relativistic effects must account for relativistic effects– Slower velocities Slower velocities vv<<cc may also be relevant: may also be relevant:

• electron saturation velocity, in various materialselectron saturation velocity, in various materials• velocity of air or liquid coolant in a cooling systemvelocity of air or liquid coolant in a cooling system

• Thus density limit Thus density limit implies flux limit implies flux limit FF==cc

v

Cross-section

Relativistic EffectsRelativistic Effects• For normal matter (bound massive-particle For normal matter (bound massive-particle

states) moving at a velocity states) moving at a velocity vv approaching approaching cc::– Entropy density increases by a factor 1/Entropy density increases by a factor 1/

• Due to relativistic length contractionDue to relativistic length contraction– But, energy density increases by factor 1/But, energy density increases by factor 1/22

• Both Both length contraction length contraction & & mass amplification!mass amplification! entropy density scales up only w. square root entropy density scales up only w. square root

(1/2 power) of energy density from high velocity(1/2 power) of energy density from high velocity

• Note that light travels at Note that light travels at cc already, already,• & its entropy density scales with energy density & its entropy density scales with energy density

to the 3/4 power. to the 3/4 power. Light wins as Light wins as vvcc..– If you want to maximize entropy/energy fluxIf you want to maximize entropy/energy flux

Max. Entropy Flux Using LightMax. Entropy Flux Using Light

• Where:Where:FFSS = = entropy fluxentropy flux

FFEE = = energy fluxenergy flux

SBSB = Stefan-Boltzmann constant, = Stefan-Boltzmann constant, 22kkBB44/60/60cc2233

• This is derived from the same field-theory This is derived from the same field-theory arguments as the density bound.arguments as the density bound.

• Again, the blackbody spectrum optimizes the Again, the blackbody spectrum optimizes the entropy flux given the energy fluxentropy flux given the energy flux– It is the equilibrium spectrum!It is the equilibrium spectrum!

434134

ESBS FF Smith ‘95

Entropy Flux ExamplesEntropy Flux Examples• Consider a 10 cm wide, flat, square wireless Consider a 10 cm wide, flat, square wireless

tablet with a 10 W power supply.tablet with a 10 W power supply.– What’s its maximum rate of bit transmission?What’s its maximum rate of bit transmission?

• Independent of spectrum used, noise floor, Independent of spectrum used, noise floor, etc.etc.

• Answer: Answer: – Energy flux 10 W/(2·(10 cm)Energy flux 10 W/(2·(10 cm)22) (use both sides)) (use both sides)– Smith’s formula gives 2.2×10Smith’s formula gives 2.2×102121 bps bps

• What’s the rate What’s the rate per square nanometerper square nanometer surface? surface?– Only 109 kbps! (ISDN speed, in a 100 GHz CPU?)Only 109 kbps! (ISDN speed, in a 100 GHz CPU?)– 100 Gbps/nm100 Gbps/nm22 nearly 1 GW power! nearly 1 GW power!

Light is not informationally dense enough for high-bandwidth communication between densely packed nanometer-scale devices at reasonable power levels!!!

Entropy Flux w. Atomic Entropy Flux w. Atomic MatterMatter• Consider liquid copper (~1.5 b/ÅConsider liquid copper (~1.5 b/Å33) moving ) moving

along at a leisurely 10 cm/s…along at a leisurely 10 cm/s…– BW=1.5x10BW=1.5x102727 bps through the 10-cm wide square! bps through the 10-cm wide square!

• A million times higher BW than with 10W light!A million times higher BW than with 10W light!– 150 Gbps/nm150 Gbps/nm22 entropy flux! entropy flux!

• Plenty for nano-scale devices to talk to their neighborsPlenty for nano-scale devices to talk to their neighbors– Most of this entropy is in the conduction electrons...Most of this entropy is in the conduction electrons...

• Less conductive materials have much less entropyLess conductive materials have much less entropy

• Lesson:Lesson:– For maximum bandwidth density at realistic power For maximum bandwidth density at realistic power

levels, encode information using states of matter levels, encode information using states of matter (electrons) rather than states of radiation (light).(electrons) rather than states of radiation (light).

Exercise: Kinetic energy flux?


– Infropy densityInfropy density• = Bits per unit volume= Bits per unit volume• Affects physical size and thus propagation delayAffects physical size and thus propagation delay

across memories and processors. Also affects cost.across memories and processors. Also affects cost.– Infropy fluxInfropy flux


standard-information input & effective entropy removalstandard-information input & effective entropy removal– Rate of computationRate of computation


devicesdevices

Computation Speed LimitsComputation Speed Limits

The Margolus-Levitin BoundThe Margolus-Levitin Bound• The maximum rate The maximum rate at which a system can at which a system can

transition between distinguishable (orthogonal) transition between distinguishable (orthogonal) states is:states is:

4( 4(EE EE00)/)/hh– where:where:

• EE = average energy (expectation value of energy over all = average energy (expectation value of energy over all states, weighted by their probability)states, weighted by their probability)

• EE00 = energy of lowest-energy or = energy of lowest-energy or groundground state of system state of system• hh = Planck’s constant (converts energy to frequency) = Planck’s constant (converts energy to frequency)

• Implication for computing:Implication for computing:– A circuit node can’t switch between 2 logic states A circuit node can’t switch between 2 logic states

faster than this frequency determined by its energy.faster than this frequency determined by its energy.

This is for pops,rate of nops ishalf as great.

Example of Frequency Bound Example of Frequency Bound • Consider Lloyd’s 1 liter, 1 kg “ultimate laptop”Consider Lloyd’s 1 liter, 1 kg “ultimate laptop”

– Total gravitating mass-energy Total gravitating mass-energy EE of 9 of 910101616 J J– Gives a limit of 5Gives a limit of 510105050 bit-operations per second! bit-operations per second!– If laptop contains 2If laptop contains 210103131 bits (photonic maximum), bits (photonic maximum),

• each bit can change state at a frequency of 2.5each bit can change state at a frequency of 2.510101919 Hz Hz (25 EHz)(25 EHz)

– 12 billion times higher-frequency than today’s 2 GHz Intel 12 billion times higher-frequency than today’s 2 GHz Intel processorsprocessors

– 250 million times higher-frequency than today’s 100 GHz 250 million times higher-frequency than today’s 100 GHz superconducting logicsuperconducting logic

• But, the Margolus-Levitin limit may be far from But, the Margolus-Levitin limit may be far from achievable in practice!achievable in practice!

More Realistic EstimatesMore Realistic Estimates• Most of the energy in complex stable structures is not Most of the energy in complex stable structures is not

accessibleaccessible for computational purposes... for computational purposes...– Tied up in the rest masses of atomic nuclei,Tied up in the rest masses of atomic nuclei,

• Which form anchor points for electron orbitalsWhich form anchor points for electron orbitals– mass & energy of “core” atomic electrons,mass & energy of “core” atomic electrons,

• Which fill up low-energy states not involved in bonding, Which fill up low-energy states not involved in bonding, – & of electrons involved in atomic bonds& of electrons involved in atomic bonds

• Which are needed to hold the structure togetherWhich are needed to hold the structure together

• Conjecture:Conjecture: Can obtain tighter valid quantum Can obtain tighter valid quantum bounds on info. densities & state-transition rates by bounds on info. densities & state-transition rates by considering only the considering only the accessibleaccessible energy. energy.– Energy whose state-information is manipulable.Energy whose state-information is manipulable.

More Realistic ExamplesMore Realistic Examples• Suppose the following system is accessible:Suppose the following system is accessible:

1 electron confined to a (10 nm)1 electron confined to a (10 nm)33 volume, at an volume, at an average potential of 10 V above ground state.average potential of 10 V above ground state.– Accessible energy: 10 eVAccessible energy: 10 eV– Accessible-energy density: 10 eV/(10 nm)Accessible-energy density: 10 eV/(10 nm)33

– Maximum entropy in Smith bound: 1.4 bits?Maximum entropy in Smith bound: 1.4 bits?• Not clear yet whether bound is applicable to this case.Not clear yet whether bound is applicable to this case.

– Maximum rate of change: 9.7 PHzMaximum rate of change: 9.7 PHz• 5 million × typical frequencies in today’s CPUs5 million × typical frequencies in today’s CPUs• 100,000 × frequencies in today’s superconducting logics100,000 × frequencies in today’s superconducting logics

Summary of Fundamental LimitsSummary of Fundamental Limits

Documents

Module #6 – Fundamental Physical Limits of Computing