Module #4 – Information, Entropy,Module #4 – Information, Entropy,Thermodynamics, and ComputingThermodynamics, and Computing

A Grand SynthesisA Grand Synthesis

Information and EntropyInformation and Entropy

A Unified Foundation for bothA Unified Foundation for bothPhysics and ComputationPhysics and Computation

What is information?What is information?• Information, most generally, is simply Information, most generally, is simply that which distinguishes one that which distinguishes one

thing from another.thing from another. – It is the It is the identity identity of a thing itself. of a thing itself. – Part or all of an Part or all of an identificationidentification, or a , or a descriptiondescription of the thing. of the thing.

• But, we must take care to distinguish between the following:But, we must take care to distinguish between the following:– A A body body of info.:of info.: The complete identity of a thing. The complete identity of a thing.– A A piecepiece of info.: of info.: An incomplete description, part of a thing’s identity. An incomplete description, part of a thing’s identity.– A A patternpattern of info.: of info.: Many separate pieces of information contained in Many separate pieces of information contained in

separate things may have identical separate things may have identical patternspatterns, or content., or content.• Those pieces are Those pieces are perfectly correlatedperfectly correlated, we may call them , we may call them copiescopies of each other. of each other.

– An An amountamount of info.: of info.: A quantification of A quantification of how largehow large a given body or piece of a given body or piece of information. information. Measured in logarithmic units.Measured in logarithmic units.

– A A subjectsubject of info.: of info.: The thing that is identified or described by a given body The thing that is identified or described by a given body or piece of information. or piece of information. May be abstract, mathematical, or physical.May be abstract, mathematical, or physical.

– An An embodimentembodiment of info.: of info.: A physical subject of information. A physical subject of information.• We can say that a body, piece, or pattern of information is We can say that a body, piece, or pattern of information is containedcontained or embodied or embodied

in its embodiment.in its embodiment.– A A meaningmeaning of info.: of info.: A semantic interpretation, tying the pattern to A semantic interpretation, tying the pattern to

meaningful/useful characteristics or properties of the thing described.meaningful/useful characteristics or properties of the thing described.– A A representationrepresentation of info.: of info.: An encoding of some information within some An encoding of some information within some

other (possibly larger) piece of info contained in something else.other (possibly larger) piece of info contained in something else.

A Thing

A Pieceof Information

Describes,in contained in

A Bodyof Information

Completelydescribes, isembodied by

Is a part of

An Amountof Information Quant-

ifiedby

Quantifiedby

Information Concept MapInformation Concept Map

A PhysicalThingMay be

AnotherPiece orBody ofInfor-mation

May be

representedby

A Pattern ofInformation

Instanceof

What is knowledge?What is knowledge?A physical entity A physical entity AA can be said to can be said to knowknow a piece (or body) of a piece (or body) of

information information I I about a thing about a thing TT, and that piece is , and that piece is considered part of considered part of AA’s ’s knowledge Kknowledge K, if and only if:, if and only if:

• AA has ready, immediate access to a physical system has ready, immediate access to a physical system SS that contains some physical information that contains some physical information PP which which AA can can observe and that includes a representation of observe and that includes a representation of II,,– E.g.E.g., , SS may be part of may be part of AA’s brain, wallet card, or laptop’s brain, wallet card, or laptop

• AA can readily and immediately decode can readily and immediately decode P’sP’s representation representation of of II and manipulate it into an explicit form and manipulate it into an explicit form

• AA understands the meaning of understands the meaning of II – how it relates to – how it relates to meaningful properties of meaningful properties of TT. Can apply . Can apply II purposefully. purposefully.

Physical InformationPhysical Information• Physical informationPhysical information is simply information that is is simply information that is

contained in a physical system.contained in a physical system.– We may speak of a body, piece, pattern, amount, subject, We may speak of a body, piece, pattern, amount, subject,

embodiment, meaning, or representation of physical embodiment, meaning, or representation of physical information, as with information in general.information, as with information in general.

– Note that Note that allall information that we can manipulate ultimately information that we can manipulate ultimately must be (or be represented by) physical information!must be (or be represented by) physical information!

• In our quantum-mechanical universe, there are two In our quantum-mechanical universe, there are two very different categories of physical information:very different categories of physical information:– Quantum informationQuantum information is is allall info. embodied in the quantum info. embodied in the quantum

state of a physical system.state of a physical system. Can’t all be measured or copied!Can’t all be measured or copied!– Classical informationClassical information is just a piece of info. that picks out a is just a piece of info. that picks out a

particular basis state, once a basis is already given.particular basis state, once a basis is already given.

Amount of InformationAmount of Information• An An amountamount of information can be conveniently quantized of information can be conveniently quantized

as a as a logarithmiclogarithmic quantity. quantity.– This measures the This measures the numbernumber of independent, fixed-capacity of independent, fixed-capacity

physical systems needed to encode the information.physical systems needed to encode the information.• Logarithmically defined values are Logarithmically defined values are inherently inherently dimensionaldimensional

((notnot dimensionless, dimensionless, i.e.i.e. pure-number) quantities. pure-number) quantities.– The pure number result must be paired with a The pure number result must be paired with a unitunit which is which is

associated with the base of the logarithm that was used.associated with the base of the logarithm that was used. log log aa = (log = (logbb aa) log-) log-bb-units = (log-units = (logcc aa) log-) log-cc-units-units

log- log-cc-unit / log--unit / log-bb-unit = log-unit = logbb cc– The log-2-unit is called the The log-2-unit is called the bitbit, the log-10-unit the , the log-10-unit the decadedecade, the, the

log-16-unit the log-16-unit the nibblenibble, the log-256-unit the , the log-256-unit the bytebyte..– Whereas, the log-e-unit (widely used in physics) is called the Whereas, the log-e-unit (widely used in physics) is called the natnat

• The The natnat is also known as is also known as Boltzmann’s constant kBoltzmann’s constant kBB (e.g. in Joules/K) (e.g. in Joules/K)• A.k.a. A.k.a. the ideal gas constant Rthe ideal gas constant R (may be expressed in kcal/mol/K) (may be expressed in kcal/mol/K)

Defining Logarithmic UnitsDefining Logarithmic Units

cb

c

cbc

cxbxc

cxbxx

xcx

bbcbx

b

b

ccb

cb

cbb

xcxcxx cbcbcb

logunit--log

unit--log

unit)--log()unit--log()(log

unit)--log()(log)unit--log()(log)(log

unit)--log()(logunit)--(log)(loglog

)(log)(loglog

)( )(log)(logloglogloglog

2lnbit 1

2ln2logunit-e-log

unit-2-log

nat

bitbit

B

eB

k

k

Forms of InformationForms of Information• Many alternative Many alternative mathematicalmathematical forms may be used to forms may be used to

represent patterns of information about a thing.represent patterns of information about a thing.• Some important examples we will visit:Some important examples we will visit:

– A string of text describing some or all properties orA string of text describing some or all properties or characteristics possessed by the thing. characteristics possessed by the thing.

– A set or ensemble of alternative possible states (or consistent,A set or ensemble of alternative possible states (or consistent, complete descriptions) of the thing. complete descriptions) of the thing.

– A probability distribution or probability density function over A probability distribution or probability density function over a set of possible states of the thing. a set of possible states of the thing.

– A quantum state vector, A quantum state vector, i.e.i.e., wavefunction giving a complex, wavefunction giving a complex valued amplitude for each possible quantum state of the thing. valued amplitude for each possible quantum state of the thing.

– A mixed state (a probability distribution over orthogonal states).A mixed state (a probability distribution over orthogonal states). – (Some string theorists suggest octonions may be needed!)(Some string theorists suggest octonions may be needed!)

Confusing Terminology AlertConfusing Terminology Alert• Be aware that in the following discussion I will often Be aware that in the following discussion I will often

shift around quickly, as needed, between the following shift around quickly, as needed, between the following related concepts:related concepts:– A A subsystemsubsystem BB of a given abstract system of a given abstract system AA. . – A A state spacestate space SS of all possible states of of all possible states of BB. . – A A state variablestate variable XX (a statistical “random variable”) of (a statistical “random variable”) of AA

representing the state of subsystem representing the state of subsystem BB within its state space within its state space SS..– A A setset TTSS of some of the possible states of of some of the possible states of BB..– A statistical “A statistical “eventevent” ” EE that the subsystem state is one of those that the subsystem state is one of those

in the state in the state TT..– A specific A specific statestate ssSS of the subsystem. of the subsystem.– A A valuevalue xx = = ss of the random variable of the random variable XX indicating that the indicating that the

specific state is specific state is ss..

Preview: Some SymbologyPreview: Some Symbology

Unknowninformation

Knowninformation

iNcompressible and/or Non-uNcomputable

iNformation

total Information(of any kind)

physicalEntropy

Unknown Info. Content of a SetUnknown Info. Content of a Set• A.k.a., amount of unknown information content.A.k.a., amount of unknown information content.

– The amount of information required to specify or pick out an The amount of information required to specify or pick out an element of the set, assuming that its members are all equally element of the set, assuming that its members are all equally likely to be selected.likely to be selected.

• An assumption we will see how to justify later.An assumption we will see how to justify later.

• The The unknownunknown information content Uinformation content U((SS) associated ) associated with a set with a set SS is defined as is defined as UU((SS) :) :≡ log |≡ log |SS||..– Since Since UU((SS)) is defined logarithmically, it always comes with is defined logarithmically, it always comes with

attached logarithmic units such as bits, nats, decades, attached logarithmic units such as bits, nats, decades, etc.etc.

• E.g.E.g., the set {a, b, c, d} has an unknown information , the set {a, b, c, d} has an unknown information content of 2 bits.content of 2 bits.

Probability and ImprobabilityProbability and Improbability• I assume you already know a bit about I assume you already know a bit about

probability theory!probability theory!• Given any probability Given any probability ℘℘(0,1], the associated (0,1], the associated

improbability improbability ℑℑ((℘℘) is defined as 1/) is defined as 1/℘℘..– There is a “1 in There is a “1 in ℑℑ((℘℘)” chance of an event occurring )” chance of an event occurring

which has probability which has probability ℘℘..– E.g.E.g. a probability of 0.01 implies an improbability a probability of 0.01 implies an improbability

of 100, of 100, i.e.i.e., a “1 in 100” chance of the event., a “1 in 100” chance of the event.• We can naturally extend this to also define the We can naturally extend this to also define the

improbability improbability ℑℑ((EE) of an event ) of an event EE having having probaprobability bility ℘℘((EE) by: ) by: ℑℑ((EE) :≡ ) :≡ ℑℑ((℘℘((EE))))

Information Gain from an EventInformation Gain from an Event• We define the We define the information gaininformation gain GGII((EE) from an ) from an

event event EE having improbability having improbability ℑℑ((EE) as:) as:GGII((EE) :≡ log ) :≡ log ℑℑ((EE) = log 1/) = log 1/℘℘((EE) = −log ) = −log ℘℘((EE))

• Why? Consider the following argument: Why? Consider the following argument: – Imagine picking event Imagine picking event EE from a set from a set SS which has which has

| |SS| = | = ℑℑ((EE) equally-likely members. ) equally-likely members. – Then, Then, EE’s improbability of being picked is ’s improbability of being picked is ℑℑ((EE), ), – While the unknown information content of While the unknown information content of SS was was

UU((SS) = log |) = log |SS| = log | = log ℑℑ((EE). ). – Thus, Thus, log log ℑℑ((EE)) unknown information must have unknown information must have

become known when we found out that become known when we found out that EE was was actually picked.actually picked.

Unknown Information Content Unknown Information Content (Entropy) of a Probability Distribution(Entropy) of a Probability Distribution

• Given a probability distribution Given a probability distribution ℘℘::SS→[0,1], define the →[0,1], define the unknown unknown information content of information content of ℘℘ as the as the expected information gainexpected information gain over all the over all the singleton events singleton events EE = { = {ss} } SS..– It is therefore the average information needed to pick out a single element.It is therefore the average information needed to pick out a single element.

• The below formula for the entropy of a probability distribution was The below formula for the entropy of a probability distribution was known to the thermodynamicists Boltzmann and Gibbs in the 1800’s!known to the thermodynamicists Boltzmann and Gibbs in the 1800’s!– Claude Shannon rediscovered/rederived it many decades later.Claude Shannon rediscovered/rederived it many decades later.

SsSs

SsSs

I

ssss

sssGs

SsEEGSU

)(log)(})({/1log)(

})({log)(})({)(

]}{|)([Ex):(

INote the −

1 2 3 4 5 6 7 8 9 10

0

0.2

0.4

0.6

0.8

1

Probability

State Index

Sample Nonuniform vs. Uniform Probability Distributions

1 2 3 4 5 6 7 8 9 10

0

20

40

60

80

Improb- ability

(1 out of N)

State Index

Improbability (Inverse Probability)

1 2 3 4 5 6 7 8 9 10 S1

0 1 2 3 4 5 6 7

Log Base 2 of Improb.

State Index

Log Improbability (Information of Discovery)

1 2 3 4 5 6 7 8 9 10

0 0.1

0.2

0.3

0.4

0.5

Bits

State Index

Boltzmann-Gibbs-Shannon Entropy (Expected Log Improbability)

Visualizing Boltzmann-Gibbs-Shannon EntropyVisualizing Boltzmann-Gibbs-Shannon Entropy

Information Content of a Physical SystemInformation Content of a Physical System• The (The (total amount oftotal amount of)) information content I information content I((AA) of an ) of an

abstract physical system abstract physical system AA is the is the unknown information unknown information contentcontent of the mathematical object of the mathematical object DD used to define used to define AA..– If If DD is (or implies) only a set is (or implies) only a set SS of (assumed equiprobable) states, of (assumed equiprobable) states,

then we have: then we have: II((AA) = ) = UU((SS) = log |) = log |SS|.|.– If If DD implies a probability distribution implies a probability distribution ℘℘::SS over a set over a set SS (of (of

distinguishable states), then: distinguishable states), then: II((AA) = ) = UU((℘℘::SS) = −) = −℘℘ii log log ℘℘ii..

• We would expect to gain We would expect to gain II((AA) information if we measured ) information if we measured AA (using basis set (using basis set SS) to find its exact ) to find its exact actualactual state state ssSS.. we say that amount we say that amount II((AA) of information is ) of information is contained incontained in AA..

• Note that the information content depends on how broad Note that the information content depends on how broad (how abstract) the system’s description (how abstract) the system’s description DD is! is!

Information Capacity & EntropyInformation Capacity & Entropy• The information capacity of a system is also the amount The information capacity of a system is also the amount

of information about the actual state of the system that we of information about the actual state of the system that we do do notnot know, given know, given onlyonly the system’s definition. the system’s definition.– It is the amount of physical information that we can say is It is the amount of physical information that we can say is inin the the

state of the system.state of the system.– It is the amount of It is the amount of uncertaintyuncertainty we have about the state of the we have about the state of the

system, if we know system, if we know onlyonly the system’s definition. the system’s definition.– It is also the quantity that is traditionally known as the It is also the quantity that is traditionally known as the

(maximum) (maximum) entropyentropy SS of the system. of the system.• Entropy was originally defined as the ratio of heat to temperature.Entropy was originally defined as the ratio of heat to temperature.

– The importance of this quantity in thermodynamics (the observed fact that it The importance of this quantity in thermodynamics (the observed fact that it never decreases) was first noticed by Rudolph Clausius in 1850.never decreases) was first noticed by Rudolph Clausius in 1850.

• Today we know that entropy is, physically, really nothing other than Today we know that entropy is, physically, really nothing other than (unknown, incompressible) information!(unknown, incompressible) information!

Known vs. Unknown InformationKnown vs. Unknown Information• We, as modelers, define what we mean by “the We, as modelers, define what we mean by “the

system” in question using some abstract description system” in question using some abstract description DD..– This implies some information content This implies some information content II((AA) for the abstract ) for the abstract

system system AA described by described by DD..• But, we will often wish to model a scenario in which But, we will often wish to model a scenario in which

some entity some entity EE (perhaps ourselves) has (perhaps ourselves) has moremore knowledge knowledge about the system about the system AA than is implied by its definition. than is implied by its definition.– E.g.E.g., scenarios in which , scenarios in which EE has prepared has prepared AA more specifically, more specifically,

or has measured some of its properties.or has measured some of its properties.– Such Such EE will generally have a will generally have a more specificmore specific description of description of AA

and thus would quote a and thus would quote a lowerlower resulting resulting II((AA) or entropy.) or entropy.• We can capture this by distinguishing the information We can capture this by distinguishing the information

in in AA that is that is knownknown byby EE from that which is from that which is unknown.unknown.– Let us now see how to do this a little more formally.Let us now see how to do this a little more formally.

Subsystems (More Generally)Subsystems (More Generally)• For a system For a system AA defined by a state set defined by a state set SS, ,

– anyany partition partition PP of of SS into subsets can be considered a subsystem into subsets can be considered a subsystem BB of of AA..• The subsets in the partition The subsets in the partition PP can be considered the “states” of the subsystem can be considered the “states” of the subsystem BB..

One subsystem

of A

Another subsytem of A In this example,the product of thetwo partitions formsa partition of Sinto singleton sets.We say that this isa complete set ofsubsystems of A.In this example, the two subsystemsare also independent.

Pieces of InformationPieces of Information• For an abstract system For an abstract system AA defined by a state set defined by a state set

SS, , anyany subset subset TTSS is a possible piece of is a possible piece of information about information about AA..– Namely it is the information “The actual state of Namely it is the information “The actual state of AA

is some member of this set is some member of this set TT.”.”• For an abstract system For an abstract system AA defined by a defined by a

probability distribution probability distribution ℘℘::SS, any probability , any probability distribution distribution ℘′℘′::SS such that such that ℘℘=0 → =0 → ℘′℘′=0 and =0 and UU((℘′℘′)<)<UU((℘℘) is another possible piece of ) is another possible piece of information about information about AA..– That is, any distribution that is consistent with and That is, any distribution that is consistent with and

more informative than more informative than AA’s very definition.’s very definition.

Known Physical InformationKnown Physical Information• Within any universe (closed physical system) Within any universe (closed physical system) WW

described by distribution described by distribution ℘℘, we say entity , we say entity EE (a subsystem (a subsystem of of WW) ) knows knows a piece a piece PP of the physical information of the physical information contained in system contained in system AA (another subsystem of (another subsystem of WW) iff ) iff ℘℘ implies a implies a correlationcorrelation between the state of between the state of EE and the state and the state of of AA, and this correlation is meaningfully accessible to , and this correlation is meaningfully accessible to EE..– Let us now see how to make this definition more precise.Let us now see how to make this definition more precise.

The Universe W

Entity(Knower)

E

The PhysicalSystem A

Correlation

What What isis a correlation, anyway? a correlation, anyway?• A concept from statistics:A concept from statistics:

– Two abstract systems Two abstract systems AA and and BB are are correlatedcorrelated or or interdependentinterdependent when the entropy of the combined when the entropy of the combined system system SS((ABAB) is less than that of ) is less than that of SS((AA)+)+SS((BB).).

– I.e.I.e., something is known about the combined state of , something is known about the combined state of ABAB that cannot be represented as knowledge about the that cannot be represented as knowledge about the state of either state of either AA or or BB by itself. by itself.

• E.g.E.g. AA,,BB each have 2 possible states 0,1 each have 2 possible states 0,1– They each have 1 bit of entropy.They each have 1 bit of entropy.– But, we might also know that But, we might also know that A=BA=B, so the entropy of , so the entropy of

ABAB is 1 bit, not 2. (States 00 and 11.) is 1 bit, not 2. (States 00 and 11.)

Marginal ProbabilityMarginal Probability• Given a joint probability distribution ℘Given a joint probability distribution ℘XYXY over a over a

sample space sample space S S that is a Cartesian product that is a Cartesian product SS = = XX × × YY, we define the , we define the projection of projection of ℘℘XYXY onto onto

XX, or the , or the marginal probability of X marginal probability of X ((under the under the distribution distribution ℘℘XYXY), written ℘), written ℘XX, as , as ℘℘XX((xxXX) = ∑) = ∑yyYY ℘ ℘XYXY((xx,,yy). ). – Similarly define the marginal probability of Similarly define the marginal probability of YY. .

• May often just write (℘May often just write (℘ xx) or ℘) or ℘xx to mean ℘ to mean ℘XX((xx).).

℘x

S = X×Y

X=x →

Conditional ProbabilityConditional Probability• Given a distribution ℘Given a distribution ℘XYXY : : XX × × YY, we define the , we define the

conditional probability of X given Y (under conditional probability of X given Y (under ℘℘XYXY)), written ℘, written ℘XX||YY, as the relative probability of , as the relative probability of XYXY versus versus YY. That is, . That is, ℘℘XX||YY((xx,,yy) (≝ ℘) (≝ ℘ xy/yxy/y) = ℘) = ℘XYXY((xx,,yy) / ℘) / ℘YY((yy),),and similarly for ℘and similarly for ℘YY||XX. .

• We may also write (℘We may also write (℘ xx||yy), or ℘), or ℘||yy((xx), or even ), or even just ℘just ℘xx||yy to mean ℘ to mean ℘XX||YY((xx,,yy). ).

• Bayes’ ruleBayes’ rule is the observation is the observation that with this definition, that with this definition, ℘ ℘xx||yy = ℘ = ℘yy||xx ℘ ℘xx / ℘ / ℘yy. .

S = X×Y

Y=y → ℘x,y

X=x

℘y

℘x

Mutual ProbabilityMutual Probability• Given a distribution ℘Given a distribution ℘XYXY : : XX × × YY as above, the as above, the mutual probability mutual probability

ratio ratio ℛℛX:YX:Y((x,yx,y) or just ) or just ℛℛx:yx:y = ℘ = ℘xyxy/℘/℘xx℘℘yy. . – Represents the factor by which the prob. of either outcomeRepresents the factor by which the prob. of either outcome

( (XX = = xx or or YY = = yy) gets boosted when we learn the other. ) gets boosted when we learn the other. – Notice that Notice that ℛℛx:yx:y = ℘ = ℘xx||yy / ℘ / ℘xx = ℘ = ℘yy||xx / ℘ / ℘yy, that is it is the relative, that is it is the relative

probability of probability of xx||yy versus versus xx, or , or yy||xx versus versus yy. . – If the two variables represent independent subsystems, then theIf the two variables represent independent subsystems, then the

mutual probability ratio is always 1. mutual probability ratio is always 1. • No change in one distribution from measuring the other.No change in one distribution from measuring the other.

• WARNING:WARNING: Some authors define something they call “mutual Some authors define something they call “mutual probability” as the probability” as the reciprocalreciprocal of the definition given here. of the definition given here.– This seems somewhat inappropriate, given the name. This seems somewhat inappropriate, given the name. – In my definition, if the mutual probability ratio is In my definition, if the mutual probability ratio is greatergreater than 1, than 1,

then the probability of then the probability of xx increasesincreases when we learn when we learn yy. . • In theirs, the opposite is true.In theirs, the opposite is true.

– The traditional definition should perhaps be instead called the The traditional definition should perhaps be instead called the mutual mutual improbability ratio.improbability ratio.

• Mutual improbability ratio: Mutual improbability ratio: ℛℛℑ,ℑ,xx::yy = = ℑℑxyxy//ℑℑxxℑℑyy = = ℘℘xx℘℘yy/℘/℘xyxy..

Marginal, Conditional, Mutual EntropiesMarginal, Conditional, Mutual Entropies• For each of the derived probabilities defined For each of the derived probabilities defined

previously, we can define a corresponding previously, we can define a corresponding informational quantity.informational quantity.– Joint probability Joint probability ℘℘XYXY → →

Joint entropy Joint entropy SS((XYXY) = ) = SS((℘℘XYXY))– Marginal probability Marginal probability ℘℘XX → →

Marginal entropy Marginal entropy SS((XX)) = S = S((℘℘XX))– Conditional probability Conditional probability ℘℘XX||YY → →

Conditional entropy Conditional entropy SS((XX||YY) = ) = ExExyy[[SS((℘℘||yy((xx))]))]– Mutual probability ratio Mutual probability ratio ℛℛXX::YY → →

Mutual information Mutual information II((XX::YY) = ) = ExExx,yx,y[log [log ℛℛx:yx:y]]• Expected reduction in entropy of Expected reduction in entropy of XX from finding out from finding out YY..

More on Mutual InformationMore on Mutual Information• Demonstration that the reduction in entropy of Demonstration that the reduction in entropy of

one variable given the other is the same as the one variable given the other is the same as the expected mutual probability ratio expected mutual probability ratio ℛℛxx::yy..

.log

logloglog

loglogloglog

log)()|()():(

:

|

||

|

xyyxxy

xy yx

xyxy

xy xy

yxxy

xy yx

xxy

xyyxxyxxy

xyyxxy

xxx

xyyxxyXSYXSXSYXI

Known Information, More FormallyKnown Information, More Formally• For a system defined by probability distribution For a system defined by probability distribution ℘℘ that includes that includes two two

subsystems subsystems AA,,BB with respective state variables with respective state variables XX,,YY having mutual having mutual information information II℘℘((XX::YY),),– The The total information content of Btotal information content of B is is II((BB) = ) = UU((℘℘YY))..– The The amount of information in B that is known by A amount of information in B that is known by A is is KKAA((BB) = ) = II℘℘((XX::YY))..– The The amount of information in B that is unknown by Aamount of information in B that is unknown by A is is

UUAA((BB) = ) = UU((℘℘YY) − ) − KKAA((BB) = ) = SS((YY) − ) − II((XX::YY) = ) = SS((YY||XX))..– The The amount of entropy in B from A’s perspective amount of entropy in B from A’s perspective is is

SSAA((BB) = ) = UUAA((BB) = ) = SS((YY||XX))..• These definitions are based on These definitions are based on allall the correlations that are present the correlations that are present

between between AA and and BB according to our global knowledge according to our global knowledge ℘℘..– However, a However, a real real entity entity AA may not know, understand, or be able to utilize may not know, understand, or be able to utilize allall

the correlations that are actually present between him and the correlations that are actually present between him and BB..– Therefore, generally Therefore, generally more more of of BB’s physical information will be ’s physical information will be effectivelyeffectively

entropy, from entropy, from AA’s perspective, than is implied by this definition.’s perspective, than is implied by this definition.• We will explore some corrections to this definition later.We will explore some corrections to this definition later.

• Later, we will also see how to sensibly extend this definition to the Later, we will also see how to sensibly extend this definition to the quantum context.quantum context.

Maximum Entropy vs. EntropyMaximum Entropy vs. EntropyTotal information content I = Maximum entropy Smax =logarithm of # states consistent with system’s definition

Unknown information UA

= Entropy SA

(as seen by observer A)

Known information KA = I − UA

= Smax − SA

as seen by observer AUnknown information UB

= Entropy SB

(as seen by observer B)

A Simple ExampleA Simple Example• A A spinspin is a type of simple is a type of simple

quantum system having only 2 quantum system having only 2 distinguishable states.distinguishable states.– In the In the zz basis, the basis states are basis, the basis states are

called “up” (called “up” (↑) and “down” (↓).↑) and “down” (↓).• In the example to the right, we In the example to the right, we

have a compound system have a compound system composed of 3 spins.composed of 3 spins. it has 8 distinguishable states.it has 8 distinguishable states.

• Suppose we know that the 4 Suppose we know that the 4 crossed-out states have 0 crossed-out states have 0 amplitude (0 probability).amplitude (0 probability).– Due to prior preparation or Due to prior preparation or

measurement of the system.measurement of the system.• Then the system contains:Then the system contains:

– One bit of known informationOne bit of known information• in spin #2in spin #2

– and two bits of entropy and two bits of entropy • in spins #1 & #3in spins #1 & #3

Entropy, as seen from the Entropy, as seen from the InsideInside• One problem with our previous definition of knowledge-dependent One problem with our previous definition of knowledge-dependent

entropy based on mutual information is that it is only well-defined entropy based on mutual information is that it is only well-defined for an for an ensembleensemble or probability distribution of observer states, or probability distribution of observer states, notnot for a single observer state.for a single observer state.– However, as observers, we always find However, as observers, we always find ourselvesourselves in a in a particular particular state, state, notnot in in

an ensemble!an ensemble!• Can we obtain an alternative definition of entropy that works for Can we obtain an alternative definition of entropy that works for

(and can be used by) observers who are in individual states also?(and can be used by) observers who are in individual states also?– While still obeying the 2While still obeying the 2ndnd law of thermodynamics? law of thermodynamics?

• Zurek proposed that entropy Zurek proposed that entropy SS should be defined to include not only should be defined to include not only unknown information unknown information UU, but also , but also incompressibleincompressible information information NN..– By definition, incompressible information (even if it is known) cannot be By definition, incompressible information (even if it is known) cannot be

reduced, therefore the validity of the 2reduced, therefore the validity of the 2ndnd law can be maintained. law can be maintained.• Zurek proposed using a quantity called Zurek proposed using a quantity called Kolmogorov complexityKolmogorov complexity to to

measure the amount of incompressible information.measure the amount of incompressible information.– Size of shortest program that computes the information – intractable to find!Size of shortest program that computes the information – intractable to find!– However, we can instead use However, we can instead use effectiveeffective (practical) incompressibility, from the (practical) incompressibility, from the

point of view of a particular observer, to yield a definition of the point of view of a particular observer, to yield a definition of the effective effective entropy for that observer, for all practical purposes.entropy for that observer, for all practical purposes.

Two Views of EntropyTwo Views of Entropy• Global view: Global view: Probability distribution, from “outside”, of observer+observee Probability distribution, from “outside”, of observer+observee

“system” leads to “expected” entropy of “system” leads to “expected” entropy of BB as seen by as seen by AA, and total system , and total system entropy.entropy.

• Local view: Local view: Entropy of Entropy of BB according to according to AA’s specific knowledge of it, plus ’s specific knowledge of it, plus incompressible size of incompressible size of AA’s representation of that knowledge, yields total entropy ’s representation of that knowledge, yields total entropy associated with associated with BB, from , from AA’s perspective.’s perspective.

Entity(Knower)

A

The PhysicalSystem B

Conditional Entropy SB|A = Expected entropy of B, from A’s perspective;

Joint distribution ℘AB →Total entropy S(AB).

Mutual information IA:B

Physical System B

Amount ofunknown info in B, from A’s

perspectiveSingle “actual”distribution ℘B

over states of B

Entity (knower) A

NB

Amount ofincompressible

info. about B represented

within A

U(℘B)

SA(B) = U(℘B) + NB

Joint dist. ℘AB

Example Comparing the Two ViewsExample Comparing the Two Views• Example:Example:

– Suppose object Suppose object BB contains 1,000 randomly-generated bits of information. contains 1,000 randomly-generated bits of information. (Initial entropy: (Initial entropy: SSBB = 1,000 b.) = 1,000 b.)

– Suppose observer Suppose observer AA reversibly measures and stores (within itself) a copy of reversibly measures and stores (within itself) a copy of one-fourth (250 b) of the information in one-fourth (250 b) of the information in BB..

• Global view:Global view:– The total information content of The total information content of BB is is II((BB) = 1000 b.) = 1000 b.– The mutual information The mutual information IIAA::BB = 250 b. (Shared by both systems.) = 250 b. (Shared by both systems.)– BB’s entropy conditioned on ’s entropy conditioned on AA: : SS((B|AB|A) = ) = II((BB))−−II((A:BA:B) = 750 b.) = 750 b.– Total entropy of joint distribution Total entropy of joint distribution SS((ABAB) = 1,000 b.) = 1,000 b.

• Local view:Local view:– AA’s specific new dist. over ’s specific new dist. over B B implies entropy implies entropy SS((℘℘BB) =) = 750 b of unknown info. 750 b of unknown info.– AA also contains also contains IIAA = 250 b of known but incompressible information about = 250 b of known but incompressible information about BB..– There is a total of There is a total of SSAA((BB) = 750 b + 250 b = 1,000 b of unknown or ) = 750 b + 250 b = 1,000 b of unknown or

incompressible information (entropy) still in the combined system.incompressible information (entropy) still in the combined system.– 750 b of this info is only “in” 750 b of this info is only “in” BB, whereas 250 b of it is shared between , whereas 250 b of it is shared between AA++BB..

750 bunknown by A

250 bknownby A

System BObserver A250 b

incompr.informat.

Re: B

Objective Entropy?Objective Entropy?• In all of this, we have defined entropy as a somewhat In all of this, we have defined entropy as a somewhat subjectivesubjective or or

relativerelative quantity: quantity:– Entropy of a subsystem depends on an observer’s state of knowledge about Entropy of a subsystem depends on an observer’s state of knowledge about

that subsystem, such as a probability distribution.that subsystem, such as a probability distribution.• Wait a minute… Doesn’t physics have a more Wait a minute… Doesn’t physics have a more objectiveobjective, observer-, observer-

independent definition of entropy?independent definition of entropy?– Only insofar as there are “preferred” states of knowledge that are most Only insofar as there are “preferred” states of knowledge that are most

readily achieved in the lab.readily achieved in the lab.• E.g.E.g., knowing of a gas only its chemical composition, temperature, pressure, , knowing of a gas only its chemical composition, temperature, pressure,

volume, and number of molecules.volume, and number of molecules.– Since such knowledge is practically difficult to improve upon using present-Since such knowledge is practically difficult to improve upon using present-

day macroscale tools, it serves as a uniform standard.day macroscale tools, it serves as a uniform standard.• However, in nanoscale systems, a significant fraction of the However, in nanoscale systems, a significant fraction of the

physical information that is present in one subsystem is subject to physical information that is present in one subsystem is subject to being known, or not, by another subsystem (depending on design).being known, or not, by another subsystem (depending on design). How a nanosystem is designed & how we deal with information recorded How a nanosystem is designed & how we deal with information recorded

at the nanoscale may at the nanoscale may vastlyvastly affect how much of the nanosystem’s internal affect how much of the nanosystem’s internal physical information effectively is or is not entropy (for practical purposes).physical information effectively is or is not entropy (for practical purposes).

Conservation of InformationConservation of Information• Theorem:Theorem: The total physical information The total physical information

capacity (maximum entropy) of any closed, capacity (maximum entropy) of any closed, constant-volume physical system (with a fixed constant-volume physical system (with a fixed definition) is unchanging in time.definition) is unchanging in time.– This follows from quantum calculations yielding This follows from quantum calculations yielding

definite, fixed numbers of distinguishable states for definite, fixed numbers of distinguishable states for all systems of given size and total energy.all systems of given size and total energy.

• We will learn about these bounds later.We will learn about these bounds later.

• Before we can do this, let us first see how to Before we can do this, let us first see how to properly define entropy for quantum systems.properly define entropy for quantum systems.

Some Categories of InformationSome Categories of InformationRelative to any given entity, we Relative to any given entity, we

can make the following can make the following distinctions (among others):distinctions (among others):

• A particular A particular piecepiece of information of information may be:may be:– Known vs. UnknownKnown vs. Unknown

• Known information vs. entropyKnown information vs. entropy– Accessible vs. Inaccessible:Accessible vs. Inaccessible:

• Measurable vs. unmeasurableMeasurable vs. unmeasurable• Controllable vs. uncontrollableControllable vs. uncontrollable

– Stable vs. UnstableStable vs. Unstable• Against degradation to entropyAgainst degradation to entropy

– Correlated vs. UncorrelatedCorrelated vs. Uncorrelated• Also, the Also, the factfact of the of the

correlation can be known or correlation can be known or unknownunknown

• The The detailsdetails of correlation can of correlation can be known or unknown be known or unknown

• The details can be easy or The details can be easy or difficult to discoverdifficult to discover

– Wanted vs. UnwantedWanted vs. Unwanted• Entropy is Entropy is usuallyusually unwanted unwanted

– Except when you’re chilly!Except when you’re chilly!• Information may often be Information may often be

unwanted, too unwanted, too – E.g.E.g., if it’s in the way, and not , if it’s in the way, and not

usefuluseful• A particular A particular patternpattern of information of information

may be:may be:– Standard vs. NonstandardStandard vs. Nonstandard

• With respect to some given coding With respect to some given coding conventionconvention

– Compressible vs. IncompressibleCompressible vs. Incompressible• Either absolutely, or effectivelyEither absolutely, or effectively• Zurek’s definition of entropy: Zurek’s definition of entropy:

unknown unknown or incompressibleor incompressible info. info.

We will be using these various We will be using these various distinctions throughout the distinctions throughout the later material…later material…

Quantum InformationQuantum Information

Generalizing classical information Generalizing classical information theory concepts to fit quantum realitytheory concepts to fit quantum reality

Density OperatorsDensity Operators• For any given state |For any given state |, the probabilities of all the basis , the probabilities of all the basis

states states ssii are determined by an Hermitian operator or are determined by an Hermitian operator or

matrix matrix (called the (called the densitydensity matrix): matrix):

• Note that the diagonal elements Note that the diagonal elements ii,,ii are just the are just the

probabilities probabilities of the basis states of the basis states ii..– The off-diagonal elements are called “coherences”.The off-diagonal elements are called “coherences”.

• They describe the entanglements that exist between basis states.They describe the entanglements that exist between basis states.

• The density matrix describes the state |The density matrix describes the state | exactlyexactly!!– It (redundantly) expresses It (redundantly) expresses allall of the quantum info. in | of the quantum info. in |. .

nnn

n

ijji

cccc

cccc

cc**

1

1*

1*1

*, ][][

Mixed StatesMixed States• Suppose the Suppose the onlyonly thing one knows about the true state thing one knows about the true state

of a system that it is chosen from a of a system that it is chosen from a statistical ensemblestatistical ensemble or or mixturemixture of state vectors of state vectors vvii (called “pure” states), each (called “pure” states), each with a derived density matrix with a derived density matrix ii, and a probability , and a probability ℘℘ii. . – In such a situation, in which one’s knowledge about the true In such a situation, in which one’s knowledge about the true

state is expressed as probability distribution over pure states, state is expressed as probability distribution over pure states, we say the system is “in” a we say the system is “in” a mixed statemixed state..

• Such a situation turns out to be Such a situation turns out to be completely described, completely described, for all physical purposesfor all physical purposes, by simply the expectation, by simply the expectationvalue (weighted average) of the value (weighted average) of the vviis’ density matrices:s’ density matrices:

– Note:Note: Even if there were Even if there were uncountablyuncountably many many vvii going into the going into the calculation, the situation remains calculation, the situation remains fully fully described by described by O(O(nn22) ) complex numbers, where complex numbers, where nn is the number of basis states! is the number of basis states!

iii ][Ex

Von Neumann EntropyVon Neumann Entropy• Suppose our probability distribution over states comes Suppose our probability distribution over states comes

from the diagonal of a density matrix from the diagonal of a density matrix ..– But, we will generally also have additional information But, we will generally also have additional information

about the state hidden in the coherences.about the state hidden in the coherences.• The off-diagonal elements of the density matrix.The off-diagonal elements of the density matrix.

– The Shannon entropy of the distribution along the diagonal The Shannon entropy of the distribution along the diagonal will generally depend on the basis used to index the matrix.will generally depend on the basis used to index the matrix.

• However, any density matrix can be (unitarily) rotated However, any density matrix can be (unitarily) rotated into another basis in which it is perfectly into another basis in which it is perfectly diagonaldiagonal!!– This means, all its off-diagonal elements are This means, all its off-diagonal elements are zerozero..– The Shannon entropy of the diagonal distribution is always The Shannon entropy of the diagonal distribution is always

minimizedminimized in the diagonal basis, and so this minimum is in the diagonal basis, and so this minimum is selected as being the selected as being the truetrue basis-independent entropy of the basis-independent entropy of the mixed quantum state mixed quantum state ..

• It is called the It is called the von Neumannvon Neumann entropy. entropy.

V.N. entropy, more formallyV.N. entropy, more formally

nnnn

n

n

21

22221

11211

nn'00

0'0

00'

' 22

11

n

S

iiiii

iiiii

ln

ln

'ln'

'ln'Tr

lnTr)(

• The trace Tr M just means the

sum of M’s diagonal elements.

• The ln of a matrix M just denotes the inverse function to eM. See the logm[] function in Matlab

• The exponential eM of a matrix M is defined via the Taylor-series expansion ∑i≥0 Mi/i!

(Shannon S)

(Boltzmann S)

Quantum Information & SubsystemsQuantum Information & Subsystems• A density matrix for a particular subsystem may be A density matrix for a particular subsystem may be

obtained by “tracing out” the other subsystems.obtained by “tracing out” the other subsystems.– Means, summing over state indices for all systems not selected.Means, summing over state indices for all systems not selected.

• This process This process discardsdiscards information about any quantum information about any quantum correlations that may be present between the subsystems!correlations that may be present between the subsystems!– Entropies of the density matrices so obtained will generally sum Entropies of the density matrices so obtained will generally sum

to > that of the original system. (Even if original state was pure!)to > that of the original system. (Even if original state was pure!)• Keeping this in mind, we may make these definitions:Keeping this in mind, we may make these definitions:

– The The unconditionedunconditioned or or marginal quantum entropy marginal quantum entropy SS((AA)) of of subsystem subsystem AA is the entropy of the is the entropy of the reducedreduced density matrix density matrix ρρAA..

– The The conditioned quantum entropyconditioned quantum entropy SS((AA||BB) = ) = SS((ABAB)−)−SS((AA))..• Note: this may be Note: this may be negativenegative! (In contrast to the classical case.)! (In contrast to the classical case.)

– The The quantum mutual information quantum mutual information II((AA::BB) = ) = SS((AA)+)+SS((BB)−)−SS((ABAB))..• As in the classical case, this measures the amount of quantum information As in the classical case, this measures the amount of quantum information

that is shared between the subsystemsthat is shared between the subsystems– Each subsystem “knows” this much information about the other.Each subsystem “knows” this much information about the other.

Tensors and Index NotationTensors and Index Notation• A A tensortensor is nothing but a generalized matrix that may have is nothing but a generalized matrix that may have more than onemore than one row row

and/or column index.and/or column index. Can also be defined recursively as a Can also be defined recursively as a matrix of tensorsmatrix of tensors..• Tensor Tensor signaturesignature: An (: An (rr,,cc) tensor has ) tensor has rr row indices and row indices and cc column indices. column indices.

– Convention: Row indices are shown as subscripts, and column indices as superscripts.Convention: Row indices are shown as subscripts, and column indices as superscripts.

• Tensor Tensor productproduct: An (: An (ll,,kk) tensor ) tensor TT times an ( times an (nn,,mm) tensor ) tensor UU is a ( is a (ll++nn,,kk++mm) tensor ) tensor VV formed from all products of an element of formed from all products of an element of TT times an element of times an element of UU::

• Tensor Tensor tracetrace: The : The tracetrace of an ( of an (rr,,cc) tensor ) tensor TT with respect to index # with respect to index #k k (where 1 (where 1 ≤ ≤ k k ≤ r≤ r,,cc)) is given by contracting (summing over) the is given by contracting (summing over) the kkthth row index together with the row index together with the kkthth column index: column index:

2

2

2

2

2

2

2

221

21 11

01

10

00

cr

cr

cr

crcc

rr tt

tttT

m

n

k

l

mk

nl

ccrr

ccrr

ccccrrrr utvUTV

1

1

1

1

11

11:

Ii

cicccrirrr

cicccrirrr

ccrrk

mkk

nkk

mkk

nkk

m

nttt

111

111

111

111

1

1::Tr

(I is the set of legal values of indices rk and ck) →

Example: a (2,2)tensor T in which all 4indices take on values from the set {0,1}:

1111

1011

0111

0011

1110

1010

0110

0010

1101

1001

0101

0001

1100

1000

0100

0000

1111

1011

1110

1010

0111

0011

0110

0010

1101

1001

1100

1000

0101

0001

0100

0000

tttt

tttt

tttt

tttt

tt

tt

tt

tt

tt

tt

tt

tt

Quantum Information ExampleQuantum Information Example• Consider the state Consider the state vvABAB = |00 = |00+|11+|11 of compound system of compound system ABAB. .

– Let Let ρρABAB = = vvvv††..

– Note that the reduced density matrices Note that the reduced density matrices ρρAA= = ρρBB are fully classical: are fully classical:

• Let’s look at the quantum entropies:Let’s look at the quantum entropies:– The joint entropyThe joint entropy SS((ABAB) =) = SS((ρρABAB) = 0 bits) = 0 bits. . (Because (Because vvABAB is a pure state.) is a pure state.)– The unconditioned entropy of subsystem The unconditioned entropy of subsystem AA is is SS((AA) = ) = SS((ρρAA) = 1 bit) = 1 bit..– The entropy of The entropy of AA conditioned on conditioned on BB is is SS((A|BA|B) = ) = SS((ABAB)−)−SS((AA) = −1 bit!) = −1 bit!– The mutual information between themThe mutual information between them II((AA::BB) = ) = SS((AA)+)+SS((BB)−)−SS((ABAB) = 2 ) = 2

bits!bits!

21

21

1111

1010

0111

0010

1101

1000

0101

0000

11

01

10

00

0

0Tr

k

kkk

kk

k

kkk

kkkj

kiABBAA

A

B

B

B

B

B

B

B

B

ji

ji

ji

ji

AB vv 11

01

10

00

21

21

21

21

21

21

21

21

00

0000

0000

00

000

0

|00 |01 |10 |11

AB AB

Quantum vs. Classical Mutual Info.Quantum vs. Classical Mutual Info.• 2 classical bit-systems have a mutual information of 2 classical bit-systems have a mutual information of at most one at most one

bitbit, , – Occurs if they are perfectly correlated, Occurs if they are perfectly correlated, e.g.e.g.,{00, 11},{00, 11}– Each bit considered Each bit considered by itselfby itself appears to have 1 bit of entropy. appears to have 1 bit of entropy.

• But taken together, there is “really” only 1 bit of entropy shared between themBut taken together, there is “really” only 1 bit of entropy shared between them– A measurement of either extracts that one bit of entropy,A measurement of either extracts that one bit of entropy,

• Leaves it in the form of 1 bit of incompressible information (to the measurer).Leaves it in the form of 1 bit of incompressible information (to the measurer).– The real joint entropy is 1 bit less than the “apparent” total entropy.The real joint entropy is 1 bit less than the “apparent” total entropy.

• Thus, the mutual information is 1 bit.Thus, the mutual information is 1 bit.

• 2 quantum bit-systems (qubits) can have a mutual info. of 2 quantum bit-systems (qubits) can have a mutual info. of two bits!two bits!– Occurs in maximally Occurs in maximally entangledentangled states, such as |00 states, such as |00+|11+|11..– Again, each qubit considered Again, each qubit considered by itselfby itself appears to have 1 bit of entropy. appears to have 1 bit of entropy.

• But taken together, there is But taken together, there is nono entropy in this pure state. entropy in this pure state.– A measurement of either qubit leaves us with A measurement of either qubit leaves us with nono entropy, rather than 1 bit! entropy, rather than 1 bit!

• If done right… see next slide.If done right… see next slide.– The real joint entropy is thus The real joint entropy is thus 22 bits less than the “apparent” total entropy. bits less than the “apparent” total entropy.

• Thus the mutual information is (by definition) 2 bits.Thus the mutual information is (by definition) 2 bits.• BothBoth of the “apparent” bits of entropy vanish if either qubit is measured. of the “apparent” bits of entropy vanish if either qubit is measured.

• Used in a communication tech. called Used in a communication tech. called quantum superdense codingquantum superdense coding..– 1 qubit’s worth of prior entanglement between two parties can be used to 1 qubit’s worth of prior entanglement between two parties can be used to

pass 2 bits of classical information between them using pass 2 bits of classical information between them using only 1 qubitonly 1 qubit!!

Classical

Classical

Qu

antu

mQ

uan

tum

Why the Difference?Why the Difference?• Entity Entity AA hasn’t yet measured hasn’t yet measured B B and and CC, which (, which (AA knows) are knows) are

initially correlated with each other, quantumly or classically:initially correlated with each other, quantumly or classically:

• AA has measured has measured BB and is now correlated with both and is now correlated with both BB and and CC::

• AA can use his new knowledge to uncompute (compress away) can use his new knowledge to uncompute (compress away) the bits from both the bits from both BB and and CC, restoring them to a standard state:, restoring them to a standard state:

11000

111000

0010

)( 11000

111000

0010 )(

Order:ABC

000

•• KKnowing he is in state |0nowing he is in state |0+|1+|1, , AA can unitarily rotate can unitarily rotate himselfhimself back to back to state |0state |0. Look ma, no entropy!. Look ma, no entropy!

•• AA, being in a mixed state, still holds , being in a mixed state, still holds a bit of information that is either a bit of information that is either unknown (external view) or unknown (external view) or incompressible (incompressible (AA’s internal view), ’s internal view), and thus is entropy, and can never and thus is entropy, and can never go away (by the 2go away (by the 2ndnd law of thermo.). law of thermo.).

Thermodynamics and ComputingThermodynamics and Computing

Proving the 2nd law of thermodynamicsProving the 2nd law of thermodynamics• Closed systems evolve via unitary transforms Closed systems evolve via unitary transforms UUtt11tt22..

– Unitary transforms just change the basis, so they Unitary transforms just change the basis, so they do not do not changechange the system’s “true” (von Neumann) entropy. the system’s “true” (von Neumann) entropy.

Theorem:Theorem: Entropy is constantEntropy is constant in all closed systems in all closed systems undergoing an undergoing an exactlyexactly-known unitary evolution.-known unitary evolution.– However, if However, if UUtt11tt22 is ever is ever at allat all uncertain, or we disregard uncertain, or we disregard

some of our information about the state, we get a some of our information about the state, we get a mixturemixture of of possible resulting states, with provably possible resulting states, with provably ≥≥ effective entropy. effective entropy.

Theorem Theorem ((2nd law of thermodynamics2nd law of thermodynamics): ): Entropy Entropy may increase but never decreasesmay increase but never decreases in closed systems in closed systems– It can increase if the system undergoes interactions whose It can increase if the system undergoes interactions whose

details are not completely known, or if the observer discards details are not completely known, or if the observer discards some of his knowledge.some of his knowledge.

Maxwell’s DemonMaxwell’s Demon• A longstanding “paradox” in thermodynamics:A longstanding “paradox” in thermodynamics:

– Why exactly can’t you beat the 2Why exactly can’t you beat the 2ndnd law, reducing the law, reducing the entropy of a system via measurements?entropy of a system via measurements?

– There were many attempted There were many attempted resolutions, all with flaws, until…resolutions, all with flaws, until…

• Bennett@IBM (‘82) noted…Bennett@IBM (‘82) noted…– The information resulting fromThe information resulting from

the measurement must bethe measurement must bedisposed of somewhere…disposed of somewhere…

– The entropy is still present inThe entropy is still present inthe demon’s memory, until hethe demon’s memory, until heexpels it into the environment.expels it into the environment.

Entropy & MeasurementEntropy & Measurement• To clarify a widespread misconception:To clarify a widespread misconception:

– The entropy (when defined as just unknown information) in an The entropy (when defined as just unknown information) in an otherwise-closed system otherwise-closed system BB cancan decrease (from the point of view of decrease (from the point of view of another entity another entity AA) if ) if AA performs a reversible or performs a reversible or non-demolition non-demolition measurement of B’s state.measurement of B’s state.

• Actual quantum non-demolition measurements have been Actual quantum non-demolition measurements have been empirically empirically demonstrateddemonstrated in carefully controlled experiments. in carefully controlled experiments.

• But, such a decrease does But, such a decrease does notnot violate the 2nd law! violate the 2nd law!• There are several alternative viewpoints as to why:There are several alternative viewpoints as to why:

– (1)(1) System System BB isn’t perfectly closed – the measurement requires an isn’t perfectly closed – the measurement requires an interaction! interaction! BB’s entropy has been ’s entropy has been movedmoved away, not deleted. away, not deleted.

– (2)(2) The entropy of the The entropy of the combinedcombined, , closed closed ABAB system does system does notnot decrease decreasefrom the point of view of an outside from the point of view of an outside entity C not measuring AB.entity C not measuring AB.

– (3)(3) From From AA’s point of view, entropy’s point of view, entropydefined as defined as unknownunknown++incompressibleincompressibleinformationinformation (Zurek) has not decreased. (Zurek) has not decreased.

0/10A B

0/10/1A B

C

C

Standard StatesStandard States• A certain state (or state set) of a system may be A certain state (or state set) of a system may be

declared by convention to be declared by convention to be “standard”“standard” within some within some context.context.– E.g.E.g. gas at standard temperature & pressure in physics gas at standard temperature & pressure in physics

experiments.experiments.– Another example: Newly allocated regions of computer Another example: Newly allocated regions of computer

memory are often standardly initialized to all 0’s.memory are often standardly initialized to all 0’s.• Information that a system is just in the/a standard state Information that a system is just in the/a standard state

can be considered can be considered null informationnull information..– It is not very informative…It is not very informative…

• There are more There are more nonstandardnonstandard states than states than standardstandard ones ones– Except in the case of isolated 2-state systems.Except in the case of isolated 2-state systems.

– However, pieces of information that are in standard states can However, pieces of information that are in standard states can still be useful as “clean slates” on which newly measured or still be useful as “clean slates” on which newly measured or computed information can be recorded.computed information can be recorded.

Computing InformationComputing Information• ComputingComputing, in the most general sense, is just the , in the most general sense, is just the

time-evolution of any physical system.time-evolution of any physical system.– Interactions between subsystems may cause Interactions between subsystems may cause

correlations to exist that didn’t exist previously.correlations to exist that didn’t exist previously.• E.g.E.g. bits bits aa=0,=0,bb interact, assigning interact, assigning a=ba=b• a a changes from a known, standard value (null information changes from a known, standard value (null information

with zero entropy) to a value that correlates with with zero entropy) to a value that correlates with bb– When systems When systems AA,,BB interact in such a way that the interact in such a way that the

state of state of AA is changed in a way that depends on the is changed in a way that depends on the state of state of BB, ,

• we say that the information in we say that the information in AA is “being computed” is “being computed”

UncomputingUncomputing Information Information• When some piece of information has been When some piece of information has been

computed using a series of known interactions, computed using a series of known interactions, – it will often be possible to perform another series of it will often be possible to perform another series of

interactions that will:interactions that will:• undoundo the effects of some or all of the earlier interactions, the effects of some or all of the earlier interactions, • and and uncomputeuncompute the pattern of information the pattern of information

– restoring it to a standard state, if desiredrestoring it to a standard state, if desired

• E.g.E.g., if the original interactions that took place , if the original interactions that took place were thermodynamically were thermodynamically reversiblereversible (did not (did not increase entropy) thenincrease entropy) then– performing the original series of interactions, performing the original series of interactions,

inverted, is one way to restore the original state.inverted, is one way to restore the original state.• There will generally be other ways also.There will generally be other ways also.

Effective EntropyEffective Entropy• For any given entity For any given entity AA, the , the effective entropyeffective entropy SSeffeff in a in a

given system given system BB is that part of the information in is that part of the information in BB that that AA cannot reversibly cannot reversibly uncomputeuncompute (for (for whateverwhatever reason). reason).

• Effective entropy also obeys a 2nd law.Effective entropy also obeys a 2nd law.– It always increases. It’s the incompressible info.It always increases. It’s the incompressible info.

• The law of The law of increase of effective entropyincrease of effective entropy remains true remains true for an combined system for an combined system ABAB in which entity in which entityAA measures system measures system BB, , eveneven from fromAA’s ’s ownown point of view! point of view!– No “outside” entity No “outside” entity CC need be need be

postulated, unlike the case forpostulated, unlike the case fornormal “unknown info” entropy.normal “unknown info” entropy.

0/10A B

0/10/1A B

Advantages of Effective EntropyAdvantages of Effective Entropy• (Effective) entropy, defined as (Effective) entropy, defined as non-reversibly-non-reversibly-

uncomputable informationuncomputable information, subsumes the following:, subsumes the following:– Unknown information:Unknown information: Can’t be reversibly uncomputed, Can’t be reversibly uncomputed,

because we don’t know what its pattern is.because we don’t know what its pattern is.• We don’t have any other info that is correlated with it.We don’t have any other info that is correlated with it.

– Known but incompressible information:Known but incompressible information: Can’t be Can’t be reversibly uncomputed because it’s incompressible.reversibly uncomputed because it’s incompressible.

• To reversibly uncompute it would be to compress it.To reversibly uncompute it would be to compress it.– Inaccessible information:Inaccessible information: Also can’t be uncomputed, Also can’t be uncomputed,

because we can’t get to it!because we can’t get to it!• E.g.E.g., a signal of known info. sent away into space at , a signal of known info. sent away into space at cc..

• This simple yet powerful definition is, I submit, the This simple yet powerful definition is, I submit, the “right” way to understand entropy.“right” way to understand entropy.

Reversibility of PhysicsReversibility of Physics• The universe is (apparently) a closed system.The universe is (apparently) a closed system.• Closed systems always evolve via unitary transforms!Closed systems always evolve via unitary transforms!

– ApparentApparent wavefunction collapse wavefunction collapse doesn’tdoesn’t contradict this contradict this (established by work of Everett, Zurek, (established by work of Everett, Zurek, etc.etc.))

• The time-evolution of the concrete state of the universe The time-evolution of the concrete state of the universe (or any closed subsystem) is therefore (or any closed subsystem) is therefore reversiblereversible::– By which (here) we mean invertible (bijective)By which (here) we mean invertible (bijective)– Deterministic looking backwards in timeDeterministic looking backwards in time– Total info. content Total info. content II & # of poss. states does not decrease & # of poss. states does not decrease

• It It cancan increase, though, if the volume is increasing increase, though, if the volume is increasing• Thus, information cannot be destroyed!Thus, information cannot be destroyed!

– It can only be It can only be invertiblyinvertibly manipulated & transformed! manipulated & transformed!• However, it However, it cancan be mixed up with other info, lost track of, sent away be mixed up with other info, lost track of, sent away

into space, into space, etc.etc.– Originally-uncomputable information can thereby become (effective) Originally-uncomputable information can thereby become (effective)

entropy.entropy.

Arrow of Time “Paradox”Arrow of Time “Paradox”• An apparent but false paradox, asking:An apparent but false paradox, asking:

– ““If physics is reversible, how is it possible that entropy can increase only in If physics is reversible, how is it possible that entropy can increase only in one time direction?”one time direction?”

• This question results from misunderstandings of the meaning & This question results from misunderstandings of the meaning & implications of implications of reversible reversible in this context.in this context.– First, reversibility (here meaning reverse-determinism) does First, reversibility (here meaning reverse-determinism) does notnot imply imply time-time-

reversal symmetryreversal symmetry.. • Which would mean that physics isWhich would mean that physics is unchanged unchanged under negation of time coordinate.under negation of time coordinate.• In a reversible system, the time-reversed dynamics does In a reversible system, the time-reversed dynamics does notnot have to be identical to have to be identical to

the forward-time dynamics, the forward-time dynamics, justjust deterministic. deterministic.– However, it happens that the Standard Model However, it happens that the Standard Model is is essentiallyessentially time-reversal symmetric time-reversal symmetric

» If we simultaneously negate charges, and reflect one space coordinate. If we simultaneously negate charges, and reflect one space coordinate. » This is more precisely called “CPT” (charge-parity-time) symmetry.This is more precisely called “CPT” (charge-parity-time) symmetry.

– I have heard that General Relativity is too, but I’m not quite sure yet…I have heard that General Relativity is too, but I’m not quite sure yet…– But anyway, But anyway, eveneven when time-reversal symmetry when time-reversal symmetry is is present, if the present, if the initial stateinitial state is is

defined to have a defined to have a lowlow max. entropy (# of poss. states), there is only room for max. entropy (# of poss. states), there is only room for entropy to increase in entropy to increase in oneone time direction: time direction: awayaway from the initial state. from the initial state.

• As the universe expands, the volume and maximum entropy of a given region ofAs the universe expands, the volume and maximum entropy of a given region of space increases. Thus, entropy increases in space increases. Thus, entropy increases in thatthat time direction. time direction.

• If you simulate a reversible and time-reversal symmetric dynamics on a computer,If you simulate a reversible and time-reversal symmetric dynamics on a computer, state complexity (practically-incompressible info., thus entropy) still state complexity (practically-incompressible info., thus entropy) still empiricallyempirically increases only in one direction (away from a simple initial state). increases only in one direction (away from a simple initial state).

– There is a simple combinatorial explanation for this behavior, namely…There is a simple combinatorial explanation for this behavior, namely…– There are always a greater number of more-complex than less-complex states to go to!There are always a greater number of more-complex than less-complex states to go to!

CRITTERS Cellular AutomatonCRITTERS Cellular Automaton• A A cellular automatoncellular automaton (CA) is a (CA) is a

discrete, local dynamical system.discrete, local dynamical system.• The CRITTERS CA uses the The CRITTERS CA uses the

“Margolus neighborhood” “Margolus neighborhood” technique.technique.– On even steps, the black 2On even steps, the black 2×2 ×2

blocks are updatedblocks are updated– On odd steps, the red blocks are On odd steps, the red blocks are

updatedupdated

• CRITTERS’ update rules:CRITTERS’ update rules:– A block with 2 A block with 2 11’s is unchanged.’s is unchanged.– A block with 3 A block with 3 11’s is rotated 180’s is rotated 180° °

and complemented.and complemented.– Other blocks are complemented.Other blocks are complemented.

• This rule, as given, is not time-This rule, as given, is not time-reversal symmetric,reversal symmetric,– But if you complement all cells But if you complement all cells

after each step, it becomes so.after each step, it becomes so.

(Plus all rotatedversions of thesecases.)

Margolus Neighborhood

Movie at http://www.ai.mit.edu/people/nhm/crit.AVI

EquilibriumEquilibrium• Due to the 2Due to the 2ndnd law, the entropy of any closed, constant- law, the entropy of any closed, constant-

volume system volume system (with not-precisely-known interactions)(with not-precisely-known interactions) increases increases until it approaches its maximum entropy until it approaches its maximum entropy II = log = log NN. . – But the But the raterate of approach to equilibrium varies greatly, of approach to equilibrium varies greatly,

depending on the precise scenario being modeled.depending on the precise scenario being modeled.• Maximum-entropy states are called Maximum-entropy states are called equilibriumequilibrium states. states.• We saw earlier that entropy is maximized by uniform We saw earlier that entropy is maximized by uniform

probability distributions.probability distributions. Theorem:Theorem: ( (Fundamental assumption of statistical Fundamental assumption of statistical

mechanics.mechanics.) Systems at equilibrium have an ) Systems at equilibrium have an equal equal probabilityprobability of being in each of their possible states. of being in each of their possible states.– Proof:Proof: The Boltzmann distribution is the one with the The Boltzmann distribution is the one with the

maximum entropy! Thus, it is the equilibrium state. maximum entropy! Thus, it is the equilibrium state. ■■– This holds for states of equal total energy…This holds for states of equal total energy…

Other “Boltzmann Distributions”Other “Boltzmann Distributions”• Consider a system Consider a system AA described in a basis in which not all basis states are assigned described in a basis in which not all basis states are assigned

the same energy.the same energy.– E.g.E.g., choose a basis consisting of energy eigenstates., choose a basis consisting of energy eigenstates.

• Suppose we know of a system Suppose we know of a system A A (in addition to its basis set) (in addition to its basis set) onlyonly that our that our expectation of its average energy expectation of its average energy EE if measured to have a certain value if measured to have a certain value EE00::

• Due to conservation of energy, if Due to conservation of energy, if EE==EE00 initially, this must remain true, so long initially, this must remain true, so long as as AA is a closed system. is a closed system.

• Jaynes (1957) showed that for a system at temperature Jaynes (1957) showed that for a system at temperature TT, the maximum entropy , the maximum entropy probability distribution probability distribution ℘℘ that is consistent with this constraint is one in which: that is consistent with this constraint is one in which:

• This same distribution was derived earlier, but in a less general scenario, byThis same distribution was derived earlier, but in a less general scenario, by Boltzmann. Boltzmann.

• Thus, at equilibrium, systems will have this distribution over state sets that do not Thus, at equilibrium, systems will have this distribution over state sets that do not all have the same energy.all have the same energy.– Does not contradict the uniform Boltzmann distribution from earlier, because that was Does not contradict the uniform Boltzmann distribution from earlier, because that was

a distribution over specific distinguishable states that are all individually consistent a distribution over specific distinguishable states that are all individually consistent with our description (in this case, that all have energy with our description (in this case, that all have energy EE00). ).

0:::][:ˆ: EEEEEEEi iiii iiii Ex

kTEi

ie /

What is energy, anyway?What is energy, anyway?• Related to the constancy of physical law.Related to the constancy of physical law.• Noether’s theoremNoether’s theorem (1905) relates conservation laws to (1905) relates conservation laws to

physical symmetries.physical symmetries.– The conservation of energy (1The conservation of energy (1stst law of thermo.) can be law of thermo.) can be

shown to be a direct consequence of the time-symmetry of shown to be a direct consequence of the time-symmetry of the laws of physics.the laws of physics.

• We saw that We saw that energy eigenstatesenergy eigenstates are those state vectors are those state vectors that remain constant (except for a phase rotation) over that remain constant (except for a phase rotation) over time. (Eigenvectors of the time. (Eigenvectors of the UUtt matrix.) matrix.)– Equilibrium states are statistical mixtures of theseEquilibrium states are statistical mixtures of these– The eigenvalue gives the energy of the eigenstateThe eigenvalue gives the energy of the eigenstate– = the rate of phase-angle accumulation of that state= the rate of phase-angle accumulation of that state

• Later, we will see that energy can also be viewed as Later, we will see that energy can also be viewed as the the rate of (quantum) computing that is occurring rate of (quantum) computing that is occurring within a physical system.within a physical system.

“Noether” rhymes with “mother”

Aside on Noether’s theoremAside on Noether’s theorem(Of no particular use in this course, but fun to know anyway…)(Of no particular use in this course, but fun to know anyway…)Virtually all of physical law can be reconstructed as a necessary consequence of Virtually all of physical law can be reconstructed as a necessary consequence of

various fundamental symmetries of the dynamics.various fundamental symmetries of the dynamics.These exemplify the general principle that the dynamical behavior itself should These exemplify the general principle that the dynamical behavior itself should

naturally be independent of all the arbitrary choices that we make in setting up naturally be independent of all the arbitrary choices that we make in setting up our mathematical representations of states.our mathematical representations of states.

• Translational symmetry (arbitrariness of position of origin) implies:Translational symmetry (arbitrariness of position of origin) implies:– Conservation of momentum!Conservation of momentum!

• Symmetry under rotations in space (no preferred direction) implies:Symmetry under rotations in space (no preferred direction) implies:– Conservation of angular momentum!Conservation of angular momentum!

• Symmetry of laws under Lorentz boosts, and accelerated motions:Symmetry of laws under Lorentz boosts, and accelerated motions:– Implies special & general relativity!Implies special & general relativity!

• Symmetry of electron wavefunctions (state vectors, or density matrices) under Symmetry of electron wavefunctions (state vectors, or density matrices) under rotations rotations in the complex planein the complex plane (arbitrariness of phase angles) implies: (arbitrariness of phase angles) implies:– For uniform rotations over all spatial points: For uniform rotations over all spatial points:

• We can derive the conservation of electric charge!We can derive the conservation of electric charge!– For spatially nonuniform (gauge) rotations: For spatially nonuniform (gauge) rotations:

• Can derive the existence of photons, and all of Maxwell’s equations!!Can derive the existence of photons, and all of Maxwell’s equations!!– Add relativistic gauge symmetries for other types of particles and interactions:Add relativistic gauge symmetries for other types of particles and interactions:

• Can get QED, QCD and the Standard Model!Can get QED, QCD and the Standard Model! • Discrete symmetries have various implications as well...Discrete symmetries have various implications as well...

Temperature at EquilibriumTemperature at Equilibrium• Recall the # of states of a compound system AB is the Recall the # of states of a compound system AB is the

product of the # of states of A and of B.product of the # of states of A and of B. the total information the total information II(AB) = (AB) = II(A)+(A)+II(B)(B)

• Combining this with the 1Combining this with the 1stst law of thermo. law of thermo. (conservation of energy) one can show (conservation of energy) one can show (Stowe sec. 9A)(Stowe sec. 9A) that that two subsystems at equilibrium with each other (so two subsystems at equilibrium with each other (so II==SS) share a property ) share a property S/S/EE– Assuming no mechanical or diffusive interactionsAssuming no mechanical or diffusive interactions

• TemperatureTemperature is then defined as the reciprocal of this is then defined as the reciprocal of this quantity, quantity, E/E/S. S. (Units: energy/entropy.)(Units: energy/entropy.)– Energy needed per increase in entropyEnergy needed per increase in entropy

Generalized TemperatureGeneralized Temperature• Any increase in the entropy of a system Any increase in the entropy of a system atat maximum entropy maximum entropy

implies an increase in that system’s total information content,implies an increase in that system’s total information content,– since total information content since total information content is the same thing asis the same thing as maximum entropy. maximum entropy.

• But, a system that is But, a system that is notnot at its maximum entropy is nothing at its maximum entropy is nothing other than just the very same system,other than just the very same system,– only in a situation where some of its state information just happens to be only in a situation where some of its state information just happens to be

known by the observer!known by the observer!• And, note that the And, note that the totaltotal information content itself does information content itself does notnot

depend on the observer’s knowledge about the system’s state,depend on the observer’s knowledge about the system’s state,– only on the very only on the very definitiondefinition of the system. of the system.

adding adding EE energy even to a energy even to a non-equilibrium systemnon-equilibrium system must must increase its increase its totaltotal information information II by the very by the very samesame amount, amount, SS!!

• So, So, II//EE in any non-equilibrium system equals in any non-equilibrium system equals SS//EE of the of the same system, same system, if it were at equilibriumif it were at equilibrium. . So, redefine So, redefine TT==EE//II..

System @temperature T

E energy

I = E/T information

Information “erasure”Information “erasure”• Suppose we have access to a subsystem containing one Suppose we have access to a subsystem containing one

bit of information (which may or may not be entropy).bit of information (which may or may not be entropy).• Suppose we now want to “erase” that bitSuppose we now want to “erase” that bit

– I.e.I.e., restore it unconditionally to a standard state, , restore it unconditionally to a standard state, e.g.e.g. 0 0• So we can later compute some new information in that location.So we can later compute some new information in that location.

– But the information/entropy in that bit physically But the information/entropy in that bit physically cannotcannot just just be irreversibly destroyed.be irreversibly destroyed.

• We can We can onlyonly ever do physically reversible actions, ever do physically reversible actions, e.g.e.g., , – Move/swap the information Move/swap the information outout of the bit of the bit

• Store it elsewhere, or let it dissipate awayStore it elsewhere, or let it dissipate away• If you lose track of the information, it becomes entropy!If you lose track of the information, it becomes entropy!

– IfIf it wasn’t already entropy. it wasn’t already entropy. Important to remember Important to remember– Or, reversibly Or, reversibly transformtransform the bit to the desired value the bit to the desired value

• Requires uncomputing the old valueRequires uncomputing the old value– based on other knowledge redundant with that old valuebased on other knowledge redundant with that old value

Energy Cost of Info. “Erasure”Energy Cost of Info. “Erasure”• Suppose you wish to “erase” (get rid of) 1 bit of Suppose you wish to “erase” (get rid of) 1 bit of

unwanted (“garbage”) information by disposing of it in unwanted (“garbage”) information by disposing of it in an external system at temperature an external system at temperature TT..TT ≈ 300 K terrestially, ~2.73 K in cosmic ≈ 300 K terrestially, ~2.73 K in cosmic μμwave backgroundwave background

• Adding that much information to the external system Adding that much information to the external system will require adding at least will require adding at least EE = (1 bit)· = (1 bit)·TT energy to your energy to your “garbage dump.”“garbage dump.”– This is true by the very This is true by the very definitiondefinition of temperature! of temperature!– In natural log units, this is In natural log units, this is kkBBTT ln 2 energy. ln 2 energy.

• @ room temperature: ~18 meV; @ 2.73 K: ~0.16 meV.@ room temperature: ~18 meV; @ 2.73 K: ~0.16 meV.

• Landauer@IBM (1961) first proved this relation Landauer@IBM (1961) first proved this relation between bit-erasure and energy.between bit-erasure and energy.– Though a similar claim was made by von Neumann in 1949.Though a similar claim was made by von Neumann in 1949.

Landauer’s 1961 Principle from basic quantum theoryLandauer’s 1961 Principle from basic quantum theory

…Ndistinct

states

Ndistinct

states

……

2Ndistinctstates

Unitary(1-1)

evolution

Before bit erasure: After bit erasure:

Increase in entropy: S = log 2 = k ln 2. Energy lost to heat: ST = kT ln 2

0s0

0sN−1

…

1s′0

1s′N−1

…

…

0s″0

0s″N−1

0s″N

0s″2N−1

…

Bistable Potential-Energy WellsBistable Potential-Energy Wells• Consider any system having an adjustable, bistable potential Consider any system having an adjustable, bistable potential

energy surface (PES) or “well” in its configuration space.energy surface (PES) or “well” in its configuration space.• The two stable states form a natural The two stable states form a natural bitbit..

– One state represents 0, the other 1.One state represents 0, the other 1.

• Consider now the P.E. well havingConsider now the P.E. well havingtwo adjustable parameters:two adjustable parameters:– (1) Height of the potential energy barrier(1) Height of the potential energy barrier

relative to the well bottomrelative to the well bottom– (2) Relative height of the left and right(2) Relative height of the left and right

states in the well (bias)states in the well (bias)

0 1

(Landauer ’61)

Possible Parameter SettingsPossible Parameter Settings• We will distinguish six qualitatively We will distinguish six qualitatively

different settings of the well’s parameters, different settings of the well’s parameters, as follows… as follows…

Direction of Bias Force

BarrierHeight

One Mechanical ImplementationOne Mechanical Implementation

spring spring

Rightwardbias

Leftwardbias

Barrier up

Barrier down

Barrierwedge

Stateknob

Possible Reversible TransitionsPossible Reversible Transitions• Catalog of all the possible transitions between known Catalog of all the possible transitions between known

states in these wells, both states in these wells, both thermodynamically reversiblethermodynamically reversible & & notnot......

Direction of Bias Force

BarrierHeight

0 0 0

111

10 N

(Ignoring superposition states.)

leak

leak

“1”states

“0”states

Erasing Digital Erasing Digital EntropyEntropy • Note that if the information in a bit-system is Note that if the information in a bit-system is alreadyalready entropy, entropy,

– Then erasing it just Then erasing it just movesmoves this entropy to the surroundings. this entropy to the surroundings.– This can be done with a thermodynamically reversible process, and does This can be done with a thermodynamically reversible process, and does

notnot necessarily necessarily increaseincrease total entropy! total entropy!• However, if/when we take a bit that is known, and irrevocably However, if/when we take a bit that is known, and irrevocably

commit ourselves to thereafter treating it as if it were commit ourselves to thereafter treating it as if it were unknownunknown, , – thatthat is the true irreversible step, is the true irreversible step, – and and thatthat is when the entropy is is when the entropy is

effectively generated!!effectively generated!!

0 0

0 N

1

1

? This state contains 1 bitof physical entropy, but ina stable, “digital” form

In these 3 states, there is no entropy in the digital state; it has all been pushed out into the environment.

This state contains 1 bitof uncomputable information, in a stable, “digital” form

ExtropyExtropy• Rather than repeatedly saying “uncomputable Rather than repeatedly saying “uncomputable

((i.e.i.e., compressible) information,” , compressible) information,” – A cumbersome phrase,A cumbersome phrase,

• let us coin the term “extropy” (and sometimes let us coin the term “extropy” (and sometimes use symbol use symbol XX) for this concept.) for this concept.– Name chosen to connote the opposite of entropy.Name chosen to connote the opposite of entropy.– Sometimes also called “negentropy.”Sometimes also called “negentropy.”

• Since a system’s total information content Since a system’s total information content II = = XX + + SS,, – we have we have XX = = II − − SS..

– We ignore previous meanings of the word We ignore previous meanings of the word “extropy,” promoted by the “Extropians”“extropy,” promoted by the “Extropians”

• A certain “trans-humanist” organization.A certain “trans-humanist” organization.

Work vs. Heat Work vs. Heat • The The total energy Etotal energy E of a system (in a given frame) can be of a system (in a given frame) can be

determined from its total inertial-gravitational mass determined from its total inertial-gravitational mass mm (in that (in that frame) using frame) using EE = = mcmc22..

• We can define the We can define the heat contentheat content HH of the system as that part of of the system as that part of EE whose state information is all entropy.whose state information is all entropy.– I.e.I.e., the part of , the part of EE that is in the subsystem w. all the entropy, and no extropy. that is in the subsystem w. all the entropy, and no extropy.– The state of that energy is unknown and/or incompressible.The state of that energy is unknown and/or incompressible.– For systems at uniform temperature For systems at uniform temperature TT, we have , we have HH = ( = (SS//II))E = STE = ST..

• For lack of a better word, we could also define the For lack of a better word, we could also define the chill contentchill contentof a system as of a system as CC = = EE − − HH. . – ““Chill” is thus any energy whose state information is all extropy.Chill” is thus any energy whose state information is all extropy.– Thus, in principle, chill can be converted into energy in any desired Thus, in principle, chill can be converted into energy in any desired

(standard) state. (standard) state. • We can define We can define work contentwork content WW≤C≤C as that part of the chill that can as that part of the chill that can

actually be practically converted into other forms as needed.actually be practically converted into other forms as needed.– E.g.E.g., gravitational potential energy can be considered work content, but , gravitational potential energy can be considered work content, but

most rest mass-energy is not.most rest mass-energy is not.• Unless we have some antimatter handy!Unless we have some antimatter handy!

Not All Heat is Unusable!Not All Heat is Unusable!• Heat enginesHeat engines can extract work from heat! can extract work from heat!

– by isolating the entropy from theby isolating the entropy from theheat into a low-temperature reservoirheat into a low-temperature reservoir

• using a smaller amount of heat.using a smaller amount of heat.

• Optimal Optimal reversible reversible (Carnot cycle)(Carnot cycle) engines recover a fraction engines recover a fraction ((TTHHTTLL)/)/TTHH of the heat as work. of the heat as work.

• Lowest-Lowest-TT capacious reservoirs: capacious reservoirs:– atmosphere (~300 K) or space (~3 K).atmosphere (~300 K) or space (~3 K).

• We would like to distinguish energy that is We would like to distinguish energy that is potentially recoverable from energy that isn’t...potentially recoverable from energy that isn’t...

Reservoir athigh temp. TH

Reservoir atlow temp. TL

HeatHH=STH

HeatHL=STL

WorkW=S(THTL)

S

S

The Carnot CycleThe Carnot Cycle• In 1822-24, Sadi Carnot analyzed the efficiency In 1822-24, Sadi Carnot analyzed the efficiency

of an ideal heat engine all of whose steps were of an ideal heat engine all of whose steps were thermodynamically thermodynamically reversiblereversible, and furthermore , and furthermore proved that, operating between any two thermal proved that, operating between any two thermal reservoirs at temperatures reservoirs at temperatures TTHH and and TTLL::– AnyAny reversible engine (regardless of internal details) reversible engine (regardless of internal details)

must have the must have the samesame efficiency efficiency ((TTHHTTLL)/)/TTHH..– NoNo engine could have greater efficiency than a engine could have greater efficiency than a

reversible engine w/o producing work from nothing reversible engine w/o producing work from nothing – TemperatureTemperature itself could be defined on an absolute itself could be defined on an absolute

thermodynamic scale based on heat recoverable by thermodynamic scale based on heat recoverable by a reversible engine operating between a reversible engine operating between TTHH and and TTLL

Steps of Carnot CycleSteps of Carnot Cycle• IsothermalIsothermal expansion at expansion at TTHH

• AdiabaticAdiabatic expansion expansion TTHHTTLL

– Latin for “without flow of heat”Latin for “without flow of heat”

• Isothermal compression at Isothermal compression at TTLL

• Adiabatic compression Adiabatic compression TTLLTTHHV

P

TL

TH

Isothermalexpansion(in contact

w. hot body)

Isolatechamber

Adiabaticexpan-

sion

Contact tocold body

Isothermalcompression Isolate

chamber Adiabaticcompres-

sion

Free Energy vs. Spent EnergyFree Energy vs. Spent Energy• If If TT is the temperature of the is the temperature of the lowestlowest-temperature available -temperature available

thermal reservoir of effectively unlimited capacity,thermal reservoir of effectively unlimited capacity,– The The spent energyspent energy EEspentspent in a system is the total entropy in a system is the total entropy SS in the in the

system, times system, times TT::EEspentspent = = SS··TT

– At least this much energy must be committed to the reservoir in At least this much energy must be committed to the reservoir in order to dispose of the entropy order to dispose of the entropy SS..

• The The unspent energyunspent energy EEunspunsp is total energy minus spent is total energy minus spent energy, energy, EEunspunsp = = EE − − EEspentspent..– This is the energy that could be This is the energy that could be convertedconverted into chill, in principle. into chill, in principle.

• Note this may include some of the heat, if body is above temperature Note this may include some of the heat, if body is above temperature TT..– However, not all of the unspent energy may be practically However, not all of the unspent energy may be practically

accessible, accessible, e.g.e.g., rest mass-energy tied up in massive particles., rest mass-energy tied up in massive particles.• We can then define the We can then define the free energyfree energy FF in a system as the in a system as the

part of the unspent energy that is actually realistically part of the unspent energy that is actually realistically accessible for conversion into other forms as needed.accessible for conversion into other forms as needed.

Internal EnergyInternal Energy• Internal energy Internal energy in old-school thermodynamics textbooks is usually defined, in old-school thermodynamics textbooks is usually defined,

somewhat ambiguously, to include:somewhat ambiguously, to include:– Heat contentHeat content (though this itself is usually left undefined) (though this itself is usually left undefined)– Internal kinetic energies (of Internal kinetic energies (of e.g.e.g. internal moving parts) internal moving parts)– Internal potential energies (Internal potential energies (e.g.e.g. chemical energies) chemical energies)– But But notnot the net kinetic/potential energies of the whole system relative to its the net kinetic/potential energies of the whole system relative to its

environment (this is reasonable)environment (this is reasonable)– And (strangely) And (strangely) notnot most of the rest of a system’s total rest mass-energy! most of the rest of a system’s total rest mass-energy!

• However, the supposed distinction between the rest mass-energy and the However, the supposed distinction between the rest mass-energy and the vaguely-defined “internal” energy is somewhat illusory!vaguely-defined “internal” energy is somewhat illusory!– Since relativity teaches us that Since relativity teaches us that allall the energy in a stationary system contributes to the energy in a stationary system contributes to

that system’s rest mass! (that system’s rest mass! (EE = = mcmc22 again) again)• Other authors try to define internal energy as being relative to the “lowest Other authors try to define internal energy as being relative to the “lowest

energy” state of the system,energy” state of the system,– But, lowest energy with respect to But, lowest energy with respect to whatwhat class of transformations? class of transformations?

• Chemical? Nuclear? Annihilation with antimatter? Absorption into a black hole?Chemical? Nuclear? Annihilation with antimatter? Absorption into a black hole?• I say, I say, abolishabolish the traditional vague definition of internal energy from the traditional vague definition of internal energy from

thermodynamics entirely.thermodynamics entirely.– Redefine Redefine internal energy to be a synonym for the total internal energy to be a synonym for the total rest mass-energy rest mass-energy of a system.of a system.

• Not including potential energy of interactions with surroundings.Not including potential energy of interactions with surroundings.– Use the phrase “Use the phrase “accessible internal energyaccessible internal energy” ” EEaccacc when needed to refer to that part of when needed to refer to that part of

the rest mass that we currently know how to extract and convert to other forms.the rest mass that we currently know how to extract and convert to other forms.• The part that is not forever tied up encoding, say, conserved quarks and leptons.The part that is not forever tied up encoding, say, conserved quarks and leptons.

Helmholtz and Gibbs Free EnergiesHelmholtz and Gibbs Free Energies• Helmholtz free energyHelmholtz free energy FFHelmHelm is essentially the same as is essentially the same as

our definition of free energy…our definition of free energy…– Except that the usual definitions of Helmholtz free energy Except that the usual definitions of Helmholtz free energy

depend on some vaguely-defined concept of internal energy.depend on some vaguely-defined concept of internal energy.• The usual definition is The usual definition is FFHelmHelm = = EEintint − − EEspentspent = = EEintint − − STST..

– If we replace this with our clearer concept of the accessible If we replace this with our clearer concept of the accessible part of rest mass-energy, the definitions become the same.part of rest mass-energy, the definitions become the same.

• Gibbs free energyGibbs free energy is just Helmholtz free energy, plus is just Helmholtz free energy, plus the potential energy of interaction with a (presumed the potential energy of interaction with a (presumed much larger) surrounding medium at pressure much larger) surrounding medium at pressure pp..– For a system of volume For a system of volume VV, this energy is , this energy is pVpV, since if the , since if the

system could be adiabatically compressed to zero volume, system could be adiabatically compressed to zero volume, the medium would deliver the medium would deliver pVpV work into it during such work into it during such compression.compression.

P = Potential energy ofinteraction w. surroundings

Breakdown of Energy ComponentsBreakdown of Energy Components

Eint = “Internal” energy = accessible rest energy

Einacc = Inaccessible part of rest

mass-energy (tied up in massive

particles)

K = System’s overall kinetic energy relative to its environ-

ment (in a given reference frame)

EpV = pV = Energy of interaction

w. surrounding medium at pressure

p.

Espent = STenv = Spent energy,

energy needed to expel internal

entropy

E0 = m0c2 = Rest mass-energy(true internal energy)

F = Free

energy

Echill = E−H = “Chill,” energy whose state

information is extropy (compressible).

Etot = E = mc2 = Total energy.

FGibbs = F + pV = Gibbsfree energy

W = Work content, the chill that is accessibleH = STS = Heat content, energy

whose state info. is entropy.

Key Points to RememberKey Points to Remember• Entropy and “extropy” (uncomputable information) Entropy and “extropy” (uncomputable information)

are but two sides of the same coin – information!are but two sides of the same coin – information!– At a fundamental level they are distinguished At a fundamental level they are distinguished only only by the by the

observer’s state of knowledge and available computing observer’s state of knowledge and available computing techniques (available physical manipulations).techniques (available physical manipulations).

• Total amount of information content = log(# states), Total amount of information content = log(# states), entropy can be calculated given a density matrixentropy can be calculated given a density matrix– Special cases: Probability distribution, or state-subsetSpecial cases: Probability distribution, or state-subset

• Total information content is Total information content is conservedconserved in any system in any system that is defined to have a time-independent state space.that is defined to have a time-independent state space.

• Key principles of thermodynamics can be seen to Key principles of thermodynamics can be seen to follow from (quantum) information theory.follow from (quantum) information theory.

• Temperature: Energy per unit of physical information.Temperature: Energy per unit of physical information.

Physics as ComputationPhysics as Computation

Reinterpreting fundamental physical Reinterpreting fundamental physical quantities in computational termsquantities in computational terms

Physics as ComputationPhysics as Computation• We will argue for making the following identities, among others:We will argue for making the following identities, among others:

– EntropyEntropy is the part of physical information that is not uncomputable. is the part of physical information that is not uncomputable.– EnergyEnergy is the rate at which (quantum) computational operations of a given is the rate at which (quantum) computational operations of a given

type are taking place in a system. It is a measure of physical type are taking place in a system. It is a measure of physical computing computing activityactivity..

• Rest mass Rest mass is the rate of updating of internal state information in proper time.is the rate of updating of internal state information in proper time.• Kinetic energyKinetic energy is closely related to the rate of updating of positional is closely related to the rate of updating of positional

information.information.• Potential energyPotential energy is the total rate at which operations that carry out the exchange is the total rate at which operations that carry out the exchange

of virtual particles implementing forces are taking place.of virtual particles implementing forces are taking place.• HeatHeat is that part of the energy that is operating on information that is entropy. is that part of the energy that is operating on information that is entropy.

– (Generalized) (Generalized) TemperatureTemperature is the rate of operations per unit of information.is the rate of operations per unit of information.• II..e.e., ops/bit/t is effectively just “clock frequency!”, ops/bit/t is effectively just “clock frequency!”• Thermodynamic temperatureThermodynamic temperature is the temperature of information that is entropy. is the temperature of information that is entropy.

– The The actionaction of an energy measures the total number of quantum operations of an energy measures the total number of quantum operations performed by that energy. An action is thus an performed by that energy. An action is thus an amount of computationamount of computation..

• The action of the Lagrangian, is the difference between amount of kinetic The action of the Lagrangian, is the difference between amount of kinetic vs.vs. potential computation.potential computation.

– (Relativistic) (Relativistic) momentummomentum measures spatial-translation (“motional”) ops measures spatial-translation (“motional”) ops performed per unit distance traversed.performed per unit distance traversed.

• This definition makes this entire system consistent with special relativity.This definition makes this entire system consistent with special relativity.

Physics as Computing (1 of 2)Physics as Computing (1 of 2)Physical QuantityPhysical Quantity Computational InterpretationComputational Interpretation Computational UnitsComputational Units

EntropyEntropy Physical information that is Physical information that is unknown (or incompressible)unknown (or incompressible)

Information (log #states), Information (log #states), e.g.e.g., nat = , nat = kkBB, bit = , bit = kkBB ln 2 ln 2

ActionAction Number of (quantum) operationsNumber of (quantum) operationscarrying out motion & interactioncarrying out motion & interaction

Operations or ops: Operations or ops: r-op = r-op = , , ππ-op = -op = hh/2/2

Angular Angular MomentumMomentum

Number of operations taken per Number of operations taken per unit angle of rotationunit angle of rotation

ops/angleops/angle(1 r-op/rad = 2 (1 r-op/rad = 2 ππ-ops/-ops/))

Proper TimeProper Time, , DistanceDistance, , TimeTime

Number of Number of internal-updateinternal-update ops, ops, spatial transitionspatial transition ops, ops, totaltotal ops if ops if trajectory is taken by a reference trajectory is taken by a reference system (Planck-mass particle?) system (Planck-mass particle?)

ops, ops, opsops, ops, ops

VelocityVelocity22 Fraction of total ops of systemFraction of total ops of systemeffecting net spatial translationeffecting net spatial translation

ops/ops = dimensionless,ops/ops = dimensionless,max. value 100% (max. value 100% (cc22))

Physics as Computing (2 of 2)Physics as Computing (2 of 2)Physical QuantityPhysical Quantity Computational InterpretationComputational Interpretation Computational UnitsComputational Units

EnergyEnergy Rate of (quantum) computation,Rate of (quantum) computation,total ops total ops ÷ time÷ time

ops/time = ops/ops = ops/time = ops/ops = dimensionlessdimensionless

Rest mass-energyRest mass-energy Rate of internal opsRate of internal ops ops/time = dimensionlessops/time = dimensionless

MomentumMomentum Spatial translation ops/distanceSpatial translation ops/distance ops/dist. = dimensionlessops/dist. = dimensionless

GeneralizedGeneralizedTemperatureTemperature

Update frequency, avg. rate of Update frequency, avg. rate of complete parallel update stepscomplete parallel update steps

ops/time/infoops/time/info= info= info−1−1

HeatHeat Energy in subsystems whose Energy in subsystems whose information is entropyinformation is entropy

ops/time = dimensionlessops/time = dimensionless

ThermalThermalTemperatureTemperature

Generalized temperature of Generalized temperature of subsystems whose information subsystems whose information

is entropyis entropy

ops/time/infoops/time/info= info= info−1−1

The Computational Interpretation The Computational Interpretation of Energyof Energy

Energy as Rate of ComputingEnergy as Rate of Computing

Energy as Rate of Phase RotationEnergy as Rate of Phase Rotation• Consider any quantum system whatsoever.Consider any quantum system whatsoever.

– And consider any eigenvector |And consider any eigenvector |EE (with eigenvalue (with eigenvalue EE) of the Hermitian ) of the Hermitian operator operator HH that is the quantum representation of the system’s Hamiltonian. that is the quantum representation of the system’s Hamiltonian.

• Remember, |Remember, |EE is therefore also an eigenstate of the unitary time- is therefore also an eigenstate of the unitary time-evolution operator evolution operator UU = e = eiiHtHt//..– Specifically, w. the eigen-equation Specifically, w. the eigen-equation UU||EE = (e = (eiiEtEt//)|)|EE..

• Thus, in any wavefunction Thus, in any wavefunction ΨΨ for this system, | for this system, |EE’s amplitude at any ’s amplitude at any time time tt is given by is given by ΨΨ((||EE, , tt) = ) = mm··eeiiEtEt//..– It is a complex number with some magnitude |It is a complex number with some magnitude |ΨΨ|=|=mm, phase-rotating in the , phase-rotating in the

complex plane at the angular frequency complex plane at the angular frequency ωω = ( = (EE//) rad = () rad = (EE//hh) circ = ) circ = ff..• In fact, the In fact, the entiretyentirety of any system’s quantum dynamics can be of any system’s quantum dynamics can be

summed up by saying that each of its energy eigenstates |summed up by saying that each of its energy eigenstates |EE is just is just sitting there, phase-rotating at frequency sitting there, phase-rotating at frequency EE//hh..– Thus we can say, energy Thus we can say, energy isis nothing other than a rate of phase rotation. nothing other than a rate of phase rotation.– And, Planck’s constant And, Planck’s constant hh isis just 1 cycle of phase rotation, just 1 cycle of phase rotation, isis 1 radian. 1 radian.

• But, what happens to states But, what happens to states otherother than energy eigenstates? than energy eigenstates?– Next we’ll see that energy Next we’ll see that energy isis also a rate of rotation of vectors in Hilbert space! also a rate of rotation of vectors in Hilbert space!

Some terminology we’ll need…Some terminology we’ll need…• A A transformationtransformation is any unitary is any unitary UU which can applied which can applied

to the state of a quantum system.to the state of a quantum system.– A transformation is A transformation is effectiveeffective with respect to a given basis with respect to a given basis BB

if the basis states aren’t all eigenstates of if the basis states aren’t all eigenstates of UU..• A A local operationlocal operation is a transformation applied only to a is a transformation applied only to a

spatially compact, closed subsystem.spatially compact, closed subsystem.– But, But, anyany region can be considered closed under sufficiently region can be considered closed under sufficiently

short time intervals.short time intervals.• The The UU of any local theory can be approached arbitrarily closely by of any local theory can be approached arbitrarily closely by

compositions of local operations compositions of local operations onlyonly..

• A A transformation trajectorytransformation trajectory is any decomposition of a is any decomposition of a transformation into a sequence of local operations.transformation into a sequence of local operations.– Approximate, but exact in the limit as time per opApproximate, but exact in the limit as time per op→0.→0.

Operation AnglesOperation Angles• For any short, local operation For any short, local operation UU,,

– Where short means close to the identity matrix,Where short means close to the identity matrix,

• Define Define UU’s ’s operation angleoperation angle θθUU as the following: as the following:θθUU = max = maxvv cos cos−1−1(|(|vv††UvUv|)|)

– Where Where vv ranges over all normalized (| ranges over all normalized (|vv|=1) states of the local |=1) states of the local subsystem in question.subsystem in question.

• In other words, consider each possible “input” state In other words, consider each possible “input” state vv. . – After transformation by After transformation by UU, it rotates to , it rotates to UvUv. . – The inner product with the original The inner product with the original vv is is vv††Uv.Uv.– The magnitude of the inner product is given by the cosine of the The magnitude of the inner product is given by the cosine of the

angle between the original (angle between the original (vv) and final () and final (UvUv) vectors.) vectors.• Just as with dot products between real-valued vectorsJust as with dot products between real-valued vectors..

• Maximizing over Maximizing over vv gives us a definition of the operation gives us a definition of the operation angle of angle of U U that is independent of the actual state that is independent of the actual state vv..– The minimum would not be useful because it is always 0The minimum would not be useful because it is always 0..

• Since the eigenstates of Since the eigenstates of U U do not change in magnitude.do not change in magnitude.

Motivating this DefinitionMotivating this Definition• Notice that our definition of the operation angle of a transformation Notice that our definition of the operation angle of a transformation

does not depend on the actual state of the system.does not depend on the actual state of the system.– Only on the maximum angle of rotation over Only on the maximum angle of rotation over allall states of the system. states of the system.

• Later, we are going to identify the Later, we are going to identify the number of operationsnumber of operations or or amount amount of physical computationof physical computation taking place in a system with our definition taking place in a system with our definition of the operation angle.of the operation angle.– Why is this approach justified?Why is this approach justified?

• Why doesn’t the actual state of the system matter?Why doesn’t the actual state of the system matter?• Consider an ordinary logic gate, such as AND.Consider an ordinary logic gate, such as AND.

– If the inputs are 0,1 and the output is 1, the output is changed, to 0.If the inputs are 0,1 and the output is 1, the output is changed, to 0.• But, if the inputs then change to 0,0, and the output is already 0, the output isn’t But, if the inputs then change to 0,0, and the output is already 0, the output isn’t

changed at all by the gate!changed at all by the gate!– Yet, we say that the gate has still done some work, performed an op,Yet, we say that the gate has still done some work, performed an op,

• It has determined It has determined thatthat the output should not change! the output should not change!• Analogously, even when a given transformation Analogously, even when a given transformation UU does does not not rotate rotate

the actual input state, the system the actual input state, the system still still has done the work of has done the work of determining determining thatthat the actual input isn’t one of the ones that should the actual input isn’t one of the ones that should have rotated by the maximum amount.have rotated by the maximum amount.– Therefore, it is fair to quantify the amount of computational work performed Therefore, it is fair to quantify the amount of computational work performed

by the maximal amount of rotation, over by the maximal amount of rotation, over allall possible input states. possible input states.

Operation Angle of a TrajectoryOperation Angle of a Trajectory• Now, for any transformation trajectory Now, for any transformation trajectory TT = ( = (UUii),),

– (A sequence of small, local unitaries (A sequence of small, local unitaries UUii),),– We can define We can define itsits operation angle operation angle θθTT as simply the as simply the

sum, over the sum, over the UUii’s, of ’s, of theirtheir operation angles: operation angles:

ii

v

iUT

vUv

i

1cosmax:

:

Operation Angle of a TransformationOperation Angle of a Transformation• And now, for And now, for anyany transformation transformation UU (no matter (no matter

how large), of how large), of any any extended quantum system, extended quantum system, – we can define the operation angle of we can define the operation angle of UU as the as the

minimum, over minimum, over allall decompositions decompositions TT = ( = (UUii) of ) of UU, of , of

the operation angle of the operation angle of TT::

ii

vUUU

iU

UUUU

UUUU

vUvi

ii

i

iii

i

iii

1

:)(

:)()(

:)(

cosmaxmin:

min:min:

Primitive Orthogonalizing OperationsPrimitive Orthogonalizing Operations• Define a Define a primitive orthogonalizing operationprimitive orthogonalizing operation or or

poppop or ( or (ππ/2)-op to be any transformation with /2)-op to be any transformation with an operation angle ofan operation angle of

oopp = = ππ/2 rad = 90°/2 rad = 90°..

• If a transformation If a transformation UU with with θθUU = = oopp is applied to is applied to

a local system (a local system (e.g.e.g. qubit), qubit), – then then somesome initial vector initial vector vv of that system must get of that system must get

transformed by transformed by UU to a vector to a vector uu = = UvUv that is that is orthogonal to orthogonal to vv..

• Since only if Since only if uuvv ( (uu††vv = 0) will cos = 0) will cos−1−1(|(|uu††vv|) = 90°.|) = 90°.

Pop ExamplePop Example• Consider a 2-state quantum system w. energy eigenstates |Consider a 2-state quantum system w. energy eigenstates |

00 and | and |EE, with energies 0 & , with energies 0 & E E respectivelyrespectively..• Consider now an initial state Consider now an initial state vv = |0 = |0+|+|EE..• Since the phase of |Since the phase of |EE rotates at frequency rotates at frequency ff==EE//hh, ,

– it will make a complete cycle in the complex plane in it will make a complete cycle in the complex plane in hh//EE time. time.• Thus will make a half-cycle rotation to Thus will make a half-cycle rotation to −|−|EE in time in time tt = = hh/2/2EE..

• Meanwhile, state |0Meanwhile, state |0 does not phase-rotate at all. does not phase-rotate at all.– So, the new state So, the new state uu at time at time tt is is uu = |0 = |0−|−|EE. . – Note this is Note this is orthogonalorthogonal to the old state to the old state vv..

• So the time evolution So the time evolution UU = e = eiiHt/Ht/ has an has an operation angle operation angle θθUU of ≥1 pop. of ≥1 pop.– It turns out that, in fact, it It turns out that, in fact, it isis exactlyexactly 1 pop. 1 pop.

• Note both these states have average energy of:Note both these states have average energy of:

• So we have a system with average energy So we have a system with average energy performing pops at the rate:performing pops at the rate:– Can we do better than this?Can we do better than this?

E

|0

|E

−|E

v = |0+|E

u = |0−|E

−|0

pppop )/ˆ4()/2( oo hEhEf

2/ˆ EE

Margolus-Levitin TheoremMargolus-Levitin Theorem• Theorem:Theorem: No quantum system with average energy No quantum system with average energy EE

(relative to its lowest-energy eigenstate |0(relative to its lowest-energy eigenstate |0) can transition ) can transition between orthogonal states at a rate faster than between orthogonal states at a rate faster than ff = 4 = 4EE//hh. . – See the original paper (in the Limits reading list) for the proof.See the original paper (in the Limits reading list) for the proof.

• Now, notice that whenever a system has an available Now, notice that whenever a system has an available energy eigenstate at (or very close to) energy 2energy eigenstate at (or very close to) energy 2EE,,– (and this will almost always be the case in complex systems,(and this will almost always be the case in complex systems,

• which have near-continuous bands of closely-spaced energy levels)which have near-continuous bands of closely-spaced energy levels)– The state |0The state |0+|2+|2EE is a possible initial state of the system, is a possible initial state of the system,

• And it will, in fact, make transitions at the rate And it will, in fact, make transitions at the rate ff..

• Thus, by our definition of pops, virtually any physical Thus, by our definition of pops, virtually any physical system with expected energy system with expected energy EE really really isis, in fact, , in fact, performing pops at the rate 4performing pops at the rate 4EE//hh..– Thus, energy Thus, energy EE isis simply a rate of pops of 4 simply a rate of pops of 4EE//h.h.

Cycle-length Cycle-length NN orthogonalizing operations orthogonalizing operations• One flaw with thinking of a pop such as |0One flaw with thinking of a pop such as |0+|+|EE → |0→ |0−|−|EE

as a meaningful computational operation is that it as a meaningful computational operation is that it immediately undoes itself!immediately undoes itself!– The state transitions back to |0The state transitions back to |0+|+|EE, also in time , also in time tt..

• It is therefore not a computationally It is therefore not a computationally usefuluseful operation. operation.• A useful computation should be able to transition through a A useful computation should be able to transition through a

very long sequence (very long sequence (vvii) of distinct states before repeating.) of distinct states before repeating.– Margolus and Levitin also show that for a cycle of Margolus and Levitin also show that for a cycle of NN states, the states, the

frequency of orthogonal transitions is frequency of orthogonal transitions is 2(2(NN−1)/−1)/NN times slower times slower than for a cycle with only than for a cycle with only NN=2 states.=2 states.

– Thus, for indefinitely long computations (Thus, for indefinitely long computations (NN→∞), the rate of→∞), the rate of transitions between orthogonal states in such long chains transitions between orthogonal states in such long chains approaches approaches 22EE//hh for systems of average energy for systems of average energy EE..

• We define a We define a cycle-length Ncycle-length N orthogonalizing operationorthogonalizing operation or or cycle-N-opcycle-N-op or or ooNN to be an operation angle of to be an operation angle of 2(2(NN−1)−1)oopp//NN..– A A normal opnormal op (nop, (nop, ππ-op, op) -op, op) oo==oonn is just a cycle-∞-op is just a cycle-∞-op oo∞∞ = 2 = 2oopp. .

• It represents a useful transition in an indefinitely-long computation.It represents a useful transition in an indefinitely-long computation.

Cycle-Cycle-NN-op Example-op Example• Consider Consider nn independent, noninteracting 2-state systems independent, noninteracting 2-state systems SS00……SSnn−1−1..• Let system Let system ii have energy eigenstates |0 have energy eigenstates |0 and | and |EEii, where , where EEii = 2 = 2iiEE00..

• Consider the initial state Consider the initial state ii |0 |0+|+|EEii……– This is a tensor product state over all the subsystems.This is a tensor product state over all the subsystems.– Note that subsystem Note that subsystem ii will have (average) energy will have (average) energy ½·2½·2iiEE00 = = 22ii−1−1EE00..

• The whole system will therefore have a total energy given by:The whole system will therefore have a total energy given by:

• Subsystem Subsystem SSnn−1−1, whose |, whose |EEnn component has energy 2 component has energy 2nn−1−1EE00, will make , will make orthogonal transitions at the rate orthogonal transitions at the rate 22nnEE00//hh..– Note that each of these is also an orthogonal transition of the whole system.Note that each of these is also an orthogonal transition of the whole system.

• The system as a whole implements a binary counter with The system as a whole implements a binary counter with nn bits. bits.– It thus makes It thus makes NN = 2 = 2nn transitions before repeating. transitions before repeating.

• Rate Rate RR = = NENE00//hh, energy , energy EE = ( = (NN−1)−1)EE00/2/2 gives us: gives us:– EE00 = 2 = 2EE/(/(NN−1), −1), RR = 2 = 2NE/NE/((NN−1)/−1)/hh, rather than the 4, rather than the 4EE//hh rate of pops. rate of pops.– Thus, it’s slower by a factor of Thus, it’s slower by a factor of (4(4EE//hh)/[2)/[2NE/NE/((NN−1)/−1)/hh] = 2(] = 2(NN−1)/−1)/NN..

0

1

00

1

00

1

2

122

2

12 EEEE

nn

i

in

i

i

What about smaller angles?What about smaller angles?• We have seen that the pop We have seen that the pop oopp quantifies the minimum operation angle for quantifies the minimum operation angle for

orthogonalizing transformations,orthogonalizing transformations,– While the nop While the nop oo = 2 = 2oopp is the minimum operation angle for an orthogonalizing is the minimum operation angle for an orthogonalizing

operation in an unboundedly-long sequence of such.operation in an unboundedly-long sequence of such.• Is there any sense in which operations with much Is there any sense in which operations with much smallersmaller operation angles operation angles

θθ ≪≪ oopp could still represent “useful” computational ops? could still represent “useful” computational ops?– No, not considered individually… No, not considered individually…

• Consider, for any Consider, for any UU with with θθUU ≪≪ 90°, and any normalized input vector 90°, and any normalized input vector vv, the , the magnitude magnitude mm=|=|aa| of the amplitude | of the amplitude aa = = uvuv of the output vector of the output vector uu = = UvUv given given vv. . – It must be that It must be that mm = | = |vv††Uv| ≈Uv| ≈ 1, since if 1, since if mm were appreciably less than 1, then were appreciably less than 1, then θθUU ≡ ≡ cos cos−1−1

mm would be a significant fraction of 90°. would be a significant fraction of 90°.• Furthermore, for any vectors Furthermore, for any vectors uu,,vv, any subsequent unitary transforms will preserve , any subsequent unitary transforms will preserve

the angle the angle θθ between between uu and and vv..– Since a unitary transform is just a remapping to a different orthonormal basis (like a Since a unitary transform is just a remapping to a different orthonormal basis (like a

change of coordinate system), it does not change the geometric relations between change of coordinate system), it does not change the geometric relations between vectors.vectors.

• Therefore, Therefore, no subsequent operationsno subsequent operations can arrange for the states can arrange for the states uu,,vv to be to be significantly distinguishable (this is just Heisenberg again).significantly distinguishable (this is just Heisenberg again).– Thus, the original operation Thus, the original operation UU makes no significant change to the state, in that if makes no significant change to the state, in that if UU

were omitted, the result of any subsequent transformations would yield a state not were omitted, the result of any subsequent transformations would yield a state not significantly distinguishable from the case where significantly distinguishable from the case where UU was included was included..

Why Energy means Rate of ComputationWhy Energy means Rate of Computation• For any unitary transform For any unitary transform UU (including the time-evolution of any (including the time-evolution of any

quantum system), we will say that the quantum system), we will say that the number of (potentially useful number of (potentially useful physical computational) operations o physical computational) operations oUU making up making up UU is simply the is simply the numeric value of the operation angle numeric value of the operation angle θθUU, when expressed in units, when expressed in units of normal operations of normal operations oo = 180°. = 180°. That is, That is, ooUU :≡ :≡ θθUU//oo = = θθUU/180°./180°.

• Similarly, we will say the Similarly, we will say the amount of physical computationamount of physical computation (or (or computational workcomputational work) ) CCUU performed by the transform performed by the transform UU is the is the dimensionaldimensional quantity given by quantity given by θθUU itself, with its units of operation itself, with its units of operation angle (such as angle (such as oo) taken as also being units of computational work. ) taken as also being units of computational work. Thus, Thus, CCUU = = ooUU oo..– So for example, So for example, CCUU = 10 = 10 oo is the amount of computational work performed by a is the amount of computational work performed by a

transform transform UU that is made up of 10 operations (operation angle of 1800°). that is made up of 10 operations (operation angle of 1800°).• If we further declare the op If we further declare the op oo to be equal to to be equal to hh/2 (that is, we identify/2 (that is, we identify

180° of operation angle –1 useful op—with 180° of phase rotation), 180° of operation angle –1 useful op—with 180° of phase rotation),– then the results of preceding slides tell us that the rate of operationsthen the results of preceding slides tell us that the rate of operations

RRoo :≡ :≡ ooUU//tt = 2 = 2EE//hh = = EE//oo. . – Thus, the rate of physical computation Thus, the rate of physical computation RRcc :≡ :≡ ccUU//tt = = ooUU oo//tt = = EE ! !

• Thus, the energy Thus, the energy E E is is the exact same physical quantitythe exact same physical quantity as the rate of as the rate of physical computation physical computation RRcc!!!!

A Physical Computing ExampleA Physical Computing Example• Consider a subsystem consisting of 1 electron, Consider a subsystem consisting of 1 electron,

with a maximum energy level located 1 with a maximum energy level located 1 VV above the minimum level.above the minimum level.– Maximum energy Maximum energy EE = 1 eV = 1.602 = 1 eV = 1.602×10×10−19−19 J J..

• This subsystem can thus perform “useful” (non-This subsystem can thus perform “useful” (non-cyclic) physical ops at a maximum rate of cyclic) physical ops at a maximum rate of

RRoo = = EE//oo = 2 = 2EE//hh = = 483.6 THz483.6 THz..• This seems high, but it’s “only” ~150,000 times This seems high, but it’s “only” ~150,000 times

the frequencies of today’s fastest (3.2 GHz) the frequencies of today’s fastest (3.2 GHz) Pentium 4’s…Pentium 4’s…– Only ~26 years away if Moore’s Law continues!Only ~26 years away if Moore’s Law continues!

Physical vs. Logical ComputationPhysical vs. Logical Computation• A nop A nop oo also corresponds to a normal (and potentially- also corresponds to a normal (and potentially-

useful) bit-operation (nbop).useful) bit-operation (nbop).– Note a bit is the minimum-sized subsystem we can pick.Note a bit is the minimum-sized subsystem we can pick.

• However, in a real computer, we are generally using a larger However, in a real computer, we are generally using a larger number of physical bits (and ops) to emulate a number of physical bits (and ops) to emulate a smallersmaller number of logical bits/ops.number of logical bits/ops.– E.g., a minimum sized circuit node today has ~10E.g., a minimum sized circuit node today has ~1055 electron states electron states

in the conduction band between low and high voltage levels. in the conduction band between low and high voltage levels.– When we charge the node, we are changing the FermionicWhen we charge the node, we are changing the Fermionic

occupancy numbers (0 vs. 1) of occupancy numbers (0 vs. 1) of allall of these electron states. of these electron states.– Each such change requires at least 1 nop, since a change inEach such change requires at least 1 nop, since a change in

any occupancy number is distinguishable. any occupancy number is distinguishable.– Thus, today we are using at least around 10Thus, today we are using at least around 1055 physical nops to physical nops to

emulate 1 logical bop. emulate 1 logical bop. • Our goal is to make the encoding more efficient…Our goal is to make the encoding more efficient…

Units of Physical Computational WorkUnits of Physical Computational Work• Based on prior discussions…Based on prior discussions…

Name of Unit of Operation Sym. Definition

Operation Angle

Some OtherEquivalences

Primitive op, pop op Local op that maps some state to an orthogonal state. 90° h/4,

Cycle-2 op, 2-op o2 Maps a state to the next in a cycle of 2 orthogonal states. 90° op

Cycle-3 op, 3-op o3 Maps a state to the next in a cycle of 3 orthogonal states. 120° 1⅓ op

Cycle-4 op, 4-op o4 Maps a state to the next in a cycle of 4 orthogonal states. 135° 1½ op

Cycle-5 op, 5-op o5 Maps a state to the next in a cycle of 5 orthogonal states. 144° 1.6 op

Cycle-N op, N-op oN Maps a state to the next in a cycle of N orthogonal states. 180°×(N1)/N 2(N1)op/N

Cycle-∞ op, ∞-op o∞ Maps a state to the next in an infinite seq. of ortho. states. 180° 2 op, limN∞ oN, h/2

Normal/natural op, nop on, o Any op with the same operation angle as an ∞-op 180° o∞

Bit-operation, bop ob A primitive op applied to a 2-state subsystem ≥90° ≥ op

Natural bit-operation, nbop

obn A minimal orthogonalizing op on a digital bit that can be part of an indefinite-length computation

180° on, 2 op

Circle op, cop oc An operation with an operation angle of 360°. 360° 4 op, h, 2

Radian op, rop or An operation with an operation angle of 1 radian 1 radian o/π,

Kinetic vs. Potential EnergyKinetic vs. Potential Energy• Kinetic energy rotates the phase angles of Kinetic energy rotates the phase angles of

momentum eigenstates.momentum eigenstates.– Momentum eigenstates are also eigenstates Momentum eigenstates are also eigenstates

of the kinetic energy component of the of the kinetic energy component of the Hamiltonian.Hamiltonian.

• Kinetic energy is effective with respect to Kinetic energy is effective with respect to the position basisthe position basis

– Carries out changes of position,Carries out changes of position,• Transfers probability mass (and energy) Transfers probability mass (and energy)

between neighboring position states.between neighboring position states.– and ineffective with respect to momentum.and ineffective with respect to momentum.

• does does notnot transfer probability mass (and transfer probability mass (and energy) between momentum states.energy) between momentum states.

• Kinetic energy is Kinetic energy is rate of computation in the rate of computation in the position “subsystem.”position “subsystem.”

– Rate of state changes involving the spatial Rate of state changes involving the spatial translation of particles (quanta).translation of particles (quanta).

• Potential energy rotates the phase angles Potential energy rotates the phase angles of position eigenstates.of position eigenstates.

– Position eigenstates are also eigenstates of Position eigenstates are also eigenstates of the potential energy component of the the potential energy component of the Hamiltonian.Hamiltonian.

• Potential energy is effective with respect Potential energy is effective with respect to the momentum basis,to the momentum basis,

– Carries out changes of momentum (virtual Carries out changes of momentum (virtual particle creation/annihilation),particle creation/annihilation),

• Transfers probability mass (and energy) Transfers probability mass (and energy) between neighboring momentum states.between neighboring momentum states.

– and ineffective with respect to position.and ineffective with respect to position.• Does Does notnot transfer probability mass (and transfer probability mass (and

energy) between position states.energy) between position states.• Potential energy is Potential energy is rate of computation in rate of computation in

the momentum “subsystem.”the momentum “subsystem.”– Rate of state changes involving Rate of state changes involving

creation/annihilation of particles creation/annihilation of particles (increase/decrease in number of quanta).(increase/decrease in number of quanta).• Position and momentum bases are Position and momentum bases are

Fourier transforms (a unitary transform) of each other.Fourier transforms (a unitary transform) of each other.– However, the laws of physics are However, the laws of physics are notnot symmetric with respect to symmetric with respect to

exchanges of position and momentum. exchanges of position and momentum. • Momentum is related to the time-derivative of position (times Momentum is related to the time-derivative of position (times mm).).

– Derivatives and integrals are complementary to, but not symmetrical with each Derivatives and integrals are complementary to, but not symmetrical with each other.other.

Temperature is “Clock Frequency”Temperature is “Clock Frequency”• Now that we know that energy is just rate of computingNow that we know that energy is just rate of computing (rate of (rate of

ops performed)…ops performed)…– Recall that Recall that (generalized) temperature(generalized) temperature TT = = EE//II is just energy per amount of is just energy per amount of

total information content…total information content…• Therefore, temperature is nothing but the Therefore, temperature is nothing but the rate of computingrate of computing per per

unit of informationunit of information..– Measured, for example, in Measured, for example, in oo/b/b··s (ops/bit/sec).s (ops/bit/sec).

• In other words, it is the physical equivalent of a computer’s In other words, it is the physical equivalent of a computer’s clock clock frequencyfrequency or or rate of completerate of complete parallel update stepsparallel update steps..

• For example, consider the meaning of “room temperature”:For example, consider the meaning of “room temperature”:300 K = 0.0259 eV/300 K = 0.0259 eV/kkBB = 12.5 = 12.5 oo·T·THz/n = 8.67 THz/n = 8.67 Too/b/b·s·s

– That is, room temperature That is, room temperature isis nothing other than a rate of parallel physical nothing other than a rate of parallel physical computing of about 9 trillion ops per bit per second (9 THz clock freq.).computing of about 9 trillion ops per bit per second (9 THz clock freq.).

• Note this is only ~3,000 times faster than today (17 years away).Note this is only ~3,000 times faster than today (17 years away).– The digital subsystem of any hypothetical CPU that is manipulating digital The digital subsystem of any hypothetical CPU that is manipulating digital

bits at this average frequency bits at this average frequency mustmust, by definition, be at this temperature!, by definition, be at this temperature!• Superconducting electronics at cryogenic temperatures is doomed to be slower.Superconducting electronics at cryogenic temperatures is doomed to be slower.

• Note that today’s 3.2 GHz clock frequencies require only Note that today’s 3.2 GHz clock frequencies require only ≥≥0.1 K 0.1 K in the processor’s digital subsystem.in the processor’s digital subsystem.

Internal Internal vs.vs. Interaction Temperatures Interaction Temperatures• Two subsystems that are well-isolated from Two subsystems that are well-isolated from

interactions with each other may have different interactions with each other may have different internalinternal temperatures. temperatures.– Internal temperature is the step rate associated with the Internal temperature is the step rate associated with the

accessible internal energy.accessible internal energy.• The updating of the internal state of each subsystem.The updating of the internal state of each subsystem.

• But also, But also, between between any two subsystems, we may also any two subsystems, we may also define an define an interaction temperature.interaction temperature.– This is the step rate associated with the energy of interaction This is the step rate associated with the energy of interaction

(particle exchange) between the subsystems, per bit of (particle exchange) between the subsystems, per bit of information in the information in the interaction subsysteminteraction subsystem..

• The states of the interaction subsystem represent different The states of the interaction subsystem represent different distributions of quanta between the original two subsystems.distributions of quanta between the original two subsystems.

– The interaction energy updates this part of the overall system state.The interaction energy updates this part of the overall system state.

Discrete SchrDiscrete Schröödinger’s Equationdinger’s Equation• We can discretize space to an (undirected) graph of “neighboring” We can discretize space to an (undirected) graph of “neighboring”

locations…. locations….• Then, there is a discrete set of distinguishable states or Then, there is a discrete set of distinguishable states or configurationsconfigurations

(in the position basis) consisting of a function from these locations to (in the position basis) consisting of a function from these locations to the number of quanta of each distinct type at that location:the number of quanta of each distinct type at that location:– For Fermions such as electrons: For Fermions such as electrons:

• For each distinct polarization state: 0 or 1 at each loc.For each distinct polarization state: 0 or 1 at each loc.– For Bosons such as photons: For Bosons such as photons:

• For each distinct polarization state: 0 or more at each loc.For each distinct polarization state: 0 or more at each loc.• We can then derive an undirected graph of neighboring configurations.We can then derive an undirected graph of neighboring configurations.

– Each link corresponds to the transfer of 1 quanta between two neighboring locations,Each link corresponds to the transfer of 1 quanta between two neighboring locations,– Or (optionally) the creation/annihilation of 1 quanta at a location.Or (optionally) the creation/annihilation of 1 quanta at a location.

• The system’s Hamiltonian then takes the following form:The system’s Hamiltonian then takes the following form:– For each configuration For each configuration cc, there is a term , there is a term HHcc

• Gives the “potential” energy (particle rest masses) of that configuration.Gives the “potential” energy (particle rest masses) of that configuration.– For each link between two neighboring configurations For each link between two neighboring configurations cc,,dd corresponding to the corresponding to the

motion of quanta with mass motion of quanta with mass mm, include a term , include a term HHcdcd::• Comes from the kinetic energy term in SchrComes from the kinetic energy term in Schrödinger’s eq.ödinger’s eq.

– For each link corresponding to particle For each link corresponding to particle creation/annihilation, include a tem based on creation/annihilation, include a tem based on fundamental coupling constantsfundamental coupling constants

• Corresponding to strengths of fundamental forces.Corresponding to strengths of fundamental forces.

01

10

2 2

2

mxH cd

1000

0100

0001

0010

c d

Interaction Subsystem ExampleInteraction Subsystem Example• Consider a simple system with two spatially-distinct Consider a simple system with two spatially-distinct

subsystems “left” and “right”, each with 2 locations.subsystems “left” and “right”, each with 2 locations.• Suppose the wholeSuppose the whole

system contains 2system contains 2Bosonic quanta.Bosonic quanta. 20

001100

0200

1010

0101

1001

0020

0110

0011

0002

Leftsubsystem

Rightsubsystem

Graph of Locations

Graph of Configurations

Leftsubsystem

Rightsubsystem

Interactionsubsystem

CORP Device ModelCORP Device Model• Physical degrees of freedom (sub-state-spaces)Physical degrees of freedom (sub-state-spaces)

broken down into broken down into codingcoding and and non-codingnon-coding parts. parts.– Further subdivided as shown below.Further subdivided as shown below.

• Devices are characterized by geometry, delay, & Devices are characterized by geometry, delay, & operatingoperating & & interaction interaction temperatures within & between temperatures within & between devices and their subsystems and subcomponents.devices and their subsystems and subcomponents.

Device

CodingSubsystem

Non-codingSubsystem

LogicalSubsystem

RedundancySubsystem

StructuralSubsystem

ThermalSubsystem

Relativistic Spacetime, Mass, Relativistic Spacetime, Mass, and Momentumand Momentum

A Computational InterpretationA Computational Interpretation

Time as an Amount of ComputationTime as an Amount of Computation• We saw earlier that (average) energy is the amount of computation We saw earlier that (average) energy is the amount of computation

CCUU performed by a given unitary transform performed by a given unitary transform UU, divided by the time , divided by the time t t taken to perform the transformation: taken to perform the transformation:

EE = = CCUU//tt..– We can turn this around, and note that the time that passes is the amount of We can turn this around, and note that the time that passes is the amount of

computation computation CCUU divided by the energy divided by the energy EE:: tt = = CCUU//EE..• So, if we had a natural unit of energy So, if we had a natural unit of energy EE, we could say that the , we could say that the

elapsed time for a given transformation elapsed time for a given transformation UU isis just just another name foranother name for the amount the amount CCUU((EE) of computation that ) of computation that UU would do on any system would do on any system defined to have energy defined to have energy EE, which would be our “clock.” , which would be our “clock.” – One natural choice is to take One natural choice is to take EE = = EE : :≡ ≡ EEPP = ( = (cc55//GG))1/21/2, the , the Planck energyPlanck energy..

• A oft-presumed maximum particle energy in quantum gravity.A oft-presumed maximum particle energy in quantum gravity.• EE ≈ 2 GJ ≈ explosive output of ½ ton of TNT. Call it 1 “blast.” ≈ 2 GJ ≈ explosive output of ½ ton of TNT. Call it 1 “blast.”

– Then, 1 natural op Then, 1 natural op oo gives us a natural “computational” time unit of gives us a natural “computational” time unit of ttoo = = oo//E E ≈ ≈ 1.7×101.7×104343 s s, which we will call the , which we will call the ticktick. .

• Since Since oo = = hh/2 = /2 = ππ, , ttoo = = ππ//EE = = ππttPP where where ttPP = ( = (cc55//GG))1/21/2 is the is the Planck timePlanck time..• HypothesisHypothesis: The tick is the absolute minimum possible time to : The tick is the absolute minimum possible time to

perform a natural op on perform a natural op on anyany physically possible bit-system. physically possible bit-system.– Follows from the assumption that no single particle can have energy greater Follows from the assumption that no single particle can have energy greater

than than EE (in (in anyany reference frame). (See, reference frame). (See, e.g.e.g., , gr-qc/0207085gr-qc/0207085.).)

Distance as Number of Motional OpsDistance as Number of Motional Ops• A system of definite energy (A system of definite energy (e.g.e.g., , EE) performs a ) performs a

definite rate definite rate RR==EE==CC//tt of computation per unit time. of computation per unit time.– If it has a definite speed (If it has a definite speed (e.g.e.g., , cc), then ), then xx = = ctct and it also and it also

performs a definite amount of computation per unit performs a definite amount of computation per unit distance distance traversedtraversed, , CC//xx = = CC//ctct = = EE//cc = = ppPP = 6.525 kg m/s = 6.525 kg m/s ≈≈ 2 2×10×103434oo/m/m

• For a general system, some computational ops may update its For a general system, some computational ops may update its internal state (“internal” ops), internal state (“internal” ops),

– others its position (“motional” ops).others its position (“motional” ops).• We will see, for a system with zero rest mass (We will see, for a system with zero rest mass (e.g.e.g., photon), , photon), allall its its

ops are motional—concerned with updating its position.ops are motional—concerned with updating its position.

• Thus, we can also identify the Thus, we can also identify the distancedistance xx between two between two points in space with the points in space with the amount of computationamount of computation CC that that would be performed by a photon of Planck energy would be performed by a photon of Planck energy EE passing between those points. (Since passing between those points. (Since xx = = CC((cc//EE)).).)

• Consider a system undergoing a certain amount Consider a system undergoing a certain amount CCintint of of

“internal” evolution in its rest frame, without moving.“internal” evolution in its rest frame, without moving.– And also consider a system undergoing an amount And also consider a system undergoing an amount CCmotmot of of

“motional” transformation that just shifts it in space, without “motional” transformation that just shifts it in space, without evolving it internally.evolving it internally.

• Arguably, these transformations rotate the state vector Arguably, these transformations rotate the state vector in mutually orthogonal directions in Hilbert space.in mutually orthogonal directions in Hilbert space.– Thus, concurrent Thus, concurrent small operation-angle increments small operation-angle increments

θθintint and and θθmotmot make a combined operation angle make a combined operation angle

θθtottot á la Pythagoras: á la Pythagoras: θθtottot ≈ ( ≈ (θθintint22++θθmotmot

22))1/21/2..• E.g.E.g. 1° N + 1° E → ~1.414° NE at equator 1° N + 1° E → ~1.414° NE at equator

• Thus, we expect that summing suchThus, we expect that summing suchincrements gives increments gives CCtottot = ( = (CCintint

22++CCmotmot22))1/21/2..

Internal vs. Motional OpsInternal vs. Motional Ops

θmot

θint

Initialstate

Distinctinternalstate

Distinctpositional

state

θtot

Time Dilation from Time Dilation from EEPP Invariance? Invariance?• Interestingly, if we just assume that a particle of our Interestingly, if we just assume that a particle of our

reference energy reference energy EE has this same energy in has this same energy in allall frames, frames,– As in, As in, e.g.e.g., Magueijo & Smolin ’02, , Magueijo & Smolin ’02, gr-qc/0207085gr-qc/0207085,,

• Then, for this particle, since we have rest energyThen, for this particle, since we have rest energyEE00 = = CCintint//tt′ = ′ = EE = = EE = = CCtottot//tt, we must conclude that , we must conclude that tt′ = ′ = tt((CCintint//CCtottot)), but since , but since CCintint = = ((CCtottot

22−−CCmotmot22))1/21/2, we have , we have

that:that:

• But, since we defined time But, since we defined time tt as as CCtottot for all energy- for all energy-EE particles, if we analogously define particles, if we analogously define xx as as CCmotmot for all for all energy-energy-EE particles (regardless of velocity), we have: particles (regardless of velocity), we have:

2

tot

mot

tot

2mot

2tot

tot

int 1'

C

C

C

CCt

C

Ct

ttt

xtt γ11' 2

2

Note this is thecorrect formula forrelativistic timedilation!

What is momentum?What is momentum?• To try to complete our picture, let us first guessTo try to complete our picture, let us first guess

that that momentummomentum pp for any object in a given for any object in a given direction is the direction is the rate of motional opsrate of motional ops,, p = Rp = Rmotmot = C = Cmotmot//tt in that direction… And get: in that direction… And get:

• Oops! TheOops! Thecorrect relativisticcorrect relativisticrelation should berelation should beEE22 = = pp22 + + mm00

2 2 !!– What went wrong?What went wrong?– How do we fix it?How do we fix it?

• Note we can’t fix it by saying Note we can’t fix it by saying that that mm00 = = CCintint//tt, because that, because thatwouldn’t be velocity-invariant.wouldn’t be velocity-invariant.

20

222

2

int22

2

int

2

mot

2

tot

2int

2mot

2tot

γ

γ

mpE

t

CpE

t

C

t

C

t

C

CCC

Wrong!

Computation ComponentsComputation Components• Look, it can’t Look, it can’t really really be right to say that be right to say that CCtottot is is

CCtottot = ( = (CCmotmot22 + + CCintint

22))1/21/2, because amounts of computation , because amounts of computation

(like energy) ought to combine (like energy) ought to combine additivelyadditively……– And recall, the And recall, the CCmotmot & & CCintint amounts that we started with were amounts that we started with were

the amounts of motional and internal computation if the the amounts of motional and internal computation if the system only did system only did one or the otherone or the other, , notnot if doing both at once. if doing both at once.

• So, instead let us just say that So, instead let us just say that CCtottot = = CCmotmot + + CCintint, while , while

keeping the definitions keeping the definitions CCtottot = = EtEt and and CCintint = = EE00tt′.′.– In other words, the In other words, the truetrue amount of motional computation is amount of motional computation is

what is left over when you subtract the internal from the what is left over when you subtract the internal from the total, total, CCmotmot = = CCtottot − − CCintint..

Internal Internal vs.vs. Motional Computation Motional Computation• The The amount of internal computation Camount of internal computation Cintint in a system during in a system during

a transformation is the system’s total amount of a transformation is the system’s total amount of computation, computation, as measured in that system’s rest frameas measured in that system’s rest frame..– Differs from total computation Differs from total computation CCtottot for moving systems (at least for moving systems (at least

given particle energies <<given particle energies <<EE ), since some computation is carrying ), since some computation is carrying out translation of the system, not updating its state.out translation of the system, not updating its state.

• In time In time tt, a moving system’s internal (proper) time , a moving system’s internal (proper) time tt′ = ′ = γγtt..– In its own frame, the moving system thinks it has energy In its own frame, the moving system thinks it has energy

EE00 = = mm00cc22, where , where mm00 is its rest mass. Recall that is its rest mass. Recall that mm00 = = γγmm..

• Thus, the Thus, the amount of internal computationamount of internal computation CCintint is isCCintint = = EE00tt′ = ′ = γγmcmc22··γγtt = = γγ22((mcmc22//tt) = ) = γγ22EE//tt = = γγ22CCtottot..

– So, So, γγ22 is is the fraction of computation that is internal.the fraction of computation that is internal.

• Let the Let the amount of motional computationamount of motional computation CCmotmot be the rest: be the rest:CCmotmot = = CCtottot − − CCintint = = CCtottot(1−(1−γγ22) = ) = ββ22CCtottot..

– Thus, Thus, ββ22 is is the fraction of computation that is motionalthe fraction of computation that is motional..

Momentum, RevisitedMomentum, Revisited• Now, we discover that we can Now, we discover that we can

define momentum (correctly, this define momentum (correctly, this time) as time) as pp = = CCmotmot//xx..– I.e.I.e., the , the motional ops per unit motional ops per unit

distance traverseddistance traversed• in a given direction; more generallyin a given direction; more generally

ppii = ( = (CCmotmot))ii//xxii; where the motional ; where the motional

computation is also a vector, given by computation is also a vector, given by ((CCmotmot))ii = = ββii

22CCtottot..

• When we plug this version into When we plug this version into the relativistic energy-momentum the relativistic energy-momentum relation, everything checks out!relation, everything checks out! Our computational interpretation Our computational interpretation

of energy & momentum is fully of energy & momentum is fully consistent with special relativity!consistent with special relativity!

22

2222

2

tot

int

2

tot

mot

2

int

2

mot2tot

2

int

2

mot

2

tot

2

int

2

mot

2

tot

20

22

1

1

1

C

C

C

C

CCC

t

C

t

C

t

C

t

C

x

C

t

C

mpE

Kinetic Energy vs. Motional EnergyKinetic Energy vs. Motional Energy• Conventional Conventional kinetic energykinetic energy is defined as the difference is defined as the difference

between an object’s total energy when moving and its rest between an object’s total energy when moving and its rest mass-energy; mass-energy; EEkinkin = = EEtottot − − EE00. . We can do the same.We can do the same.

• We can also define We can also define motionalmotional and and internalinternal energies based on energies based on the corresponding types of computation:the corresponding types of computation:– Motional energy Motional energy EEmotmot :≡ :≡ CCmotmot//tt..– ““Internal” energy Internal” energy EEintint :≡ :≡ CCintint//tt. (Not the thermodynamic kind!). (Not the thermodynamic kind!)– Thus, Thus, EEmotmot = = EEtottot − − EEintint..

• Now, are motional and kinetic energies the same?Now, are motional and kinetic energies the same?– And likewise, are internal and rest-mass energies the same?And likewise, are internal and rest-mass energies the same?

• No, because No, because EE00 = = CCmotmot//tt′′, , notnot CCmotmot//tt..– So, So, EEintint = = γγEE00 (this decrease accounts for time dilation), (this decrease accounts for time dilation),– And And EEmotmot = = EEkinkin + (1− + (1−γγ))EE00..

• Motional energy is kinetic energy, plus the missing part of rest energy!Motional energy is kinetic energy, plus the missing part of rest energy!

Motional Energy and MomentumMotional Energy and Momentum• The previous slide implies that imparting kinetic The previous slide implies that imparting kinetic

energy to an object not only adds to its motional energy to an object not only adds to its motional energy, but also converts some of its existing energy, but also converts some of its existing internal energy to a motional form. How much?internal energy to a motional form. How much?– Exercise: [1 point] Show that Exercise: [1 point] Show that EEmotmot//EEkinkin = 1 + = 1 + γγ..

• Thus at non-relativistic speeds (that is, where Thus at non-relativistic speeds (that is, where ββ→→0 0 and thus and thus γ→γ→1), 1), EEmotmot ≈ 2 ≈ 2 EEkinkin..– Slow objects make a ~100% “matching contribution” for Slow objects make a ~100% “matching contribution” for

investments of kinetic energy you put into them!investments of kinetic energy you put into them!• Thus, Thus, EEmotmot ≈ 2(½ ≈ 2(½mvmv22) = ) = mvmv22 = = pvpv..

– Motional energy is just momentum times velocity!Motional energy is just momentum times velocity!– In fact, this relation is In fact, this relation is exactexact even relativistically, since even relativistically, since

EEmotmot = = CCmotmot//tt = = CCmotmot/(/(xx//vv) = () = (CCmotmot//xx))vv = = pvpv..

Connection with Hamilton’s PrincipleConnection with Hamilton’s Principle• Recall that the Langrangian Recall that the Langrangian LL = = pvpv − − HH..• Recall also the Hamiltonian is total energy Recall also the Hamiltonian is total energy EEtottot..

– We are free to include rest-mass energy in it if we want.We are free to include rest-mass energy in it if we want.• Thus, the Lagrangian is nothing other than the negative of Thus, the Lagrangian is nothing other than the negative of

the internal energy! the internal energy! LL = = EEmotmot − − EEtottot = − = −EEintint!!• And so, the action of the Lagrangian is just the negative And so, the action of the Lagrangian is just the negative

of the amount of internal computation! of the amount of internal computation! AALL = ∫ = ∫L dtL dt = −∫ = −∫EEintint dt dt = − = −CCintint..

• Therefore, to extremize the action is to extremize the Therefore, to extremize the action is to extremize the amount of internal computation, amount of internal computation, i.e.i.e., the proper time , the proper time tt′′..– Extreme action Extreme action extreme amt. of internal computation extreme amt. of internal computation

extreme proper time extreme proper time extreme operation angle of internal extreme operation angle of internal computation computation extreme phase angle extreme phase angle phase angle stationary phase angle stationary under small path variations under small path variations nearby paths accumulate similar nearby paths accumulate similar amplitudes amplitudes amplitudes add constructively amplitudes add constructively total amplitude total amplitude has large magnitude has large magnitude path is (locally) the most probable one! path is (locally) the most probable one!

Important Points to RememberImportant Points to Remember• Energy is simply the Energy is simply the rate of physical computingrate of physical computing..

– HeatHeat = Rate of computing in bits that are entropy. = Rate of computing in bits that are entropy.– TemperatureTemperature = Rate of computing per bit of info. = Rate of computing per bit of info.

• It is physical “clock speed”; room temp. It is physical “clock speed”; room temp. ≈ 9 THz.≈ 9 THz.– Rest mass-energyRest mass-energy = Computing rate in rest frame. = Computing rate in rest frame.– MomentumMomentum = Motional computation per unit distance. = Motional computation per unit distance.– Motional energyMotional energy = Motional computing per unit time. = Motional computing per unit time.– Kinetic energyKinetic energy = Total energy minus rest mass-energy. = Total energy minus rest mass-energy.– Internal energyInternal energy = Internal operation rate per unit time. = Internal operation rate per unit time.

• Time dilation factor times rest mass-energy.Time dilation factor times rest mass-energy.

• In the next module, we will see how these & other In the next module, we will see how these & other facts impact the fundamental limits of computing…facts impact the fundamental limits of computing…