Journal of Physics Communications

PAPER • OPEN ACCESS

Relation between Boltzmann and Gibbs entropyand example with multinomial distributionTo cite this article: Paško Županovi and Domagoj Kui 2018 J. Phys. Commun. 2 045002

View the article online for updates and enhancements.

Related contentThermalization and prethermalization inisolated quantum systems: a theoreticaloverviewTakashi Mori, Tatsuhiko N Ikeda, ErikoKaminishi et al.

-

Derivation of the Gibbs entropy with theSchrodinger approachM Santillán, E S Zeron and J L Del Rio-Correa

-

The different paths to entropyL Benguigui

-

Recent citationsRelating the small world coefficient to theentropy of 2D networks and applications inneuromorphic engineeringV Onesto et al

-

This content was downloaded from IP address 200.130.19.195 on 06/03/2020 at 16:22

https://doi.org/10.1088/2399-6528/aab7e1http://iopscience.iop.org/article/10.1088/1361-6455/aabcdfhttp://iopscience.iop.org/article/10.1088/1361-6455/aabcdfhttp://iopscience.iop.org/article/10.1088/1361-6455/aabcdfhttp://iopscience.iop.org/article/10.1088/0143-0807/29/3/022http://iopscience.iop.org/article/10.1088/0143-0807/29/3/022http://iopscience.iop.org/article/10.1088/0143-0807/34/2/303http://iopscience.iop.org/2399-6528/3/9/095011http://iopscience.iop.org/2399-6528/3/9/095011http://iopscience.iop.org/2399-6528/3/9/095011

J. Phys. Commun. 2 (2018) 045002 https://doi.org/10.1088/2399-6528/aab7e1

PAPER

Relation between Boltzmann and Gibbs entropy and example withmultinomial distribution

PaškoŽupanović andDomagoj Kuić1

Faculty of Science, University of Split, R. Boškovića 33, 21000 Split, Croatia1 The author towhomany correspondence should be addressed

E-mail: pasko@pmfst.hr and dkuic@pmfst.hr

Keywords: boltzmann entropy, gibbs entropy, fluctuation entropy, thermodynamic limit, nonequilibrium, critical phenomena

AbstractGeneral relationship betweenmeanBoltzmann entropy andGibbs entropy is established. It is foundthat their difference is equal tofluctuation entropy, which is aGibbs-like entropy ofmacroscopicquantities. The ratio of the fluctuation entropy andmeanBoltzmann, orGibbs entropy vanishes in thethermodynamic limit for a systemof distinguishable and independent particles. It is argued that largefluctuation entropy clearly indicates the limit where standard statistical approach should bemodified,or extended using othermethods like renormalization group.

1. Introduction

Entropy is the fundamental physical quantity in thermodynamics and statistical physics. It was introduced byClausius [1] in thermodynamics. It is a function of themacroscopic state of the system. In contrast to theKelvinand theClausius formulations of the second law in terms of processes, the second law expressed by entropy isformulated using a function of state. Boltzmann gave amicroscopic definition of entropy, as a logarithmof thenumber ofmicroscopic states that share the values of physical quantities of themacroscopic state of the system.Later, Gibbs gave another definition of entropy via probabilities ofmicroscopic states of the system. Bothdefinitions are used in some textbooks on statistical physics [2, 3]. In some textbooks, we find only Boltzmann[4] orGibbs [5]definition of entropy. In these textbooks there is, either, no connection between these twodefinitions, or it is not established in a completely transparent way [2].

The problemof two different definitions of entropy is the subject of several works. The reference [6] putsinto relation theGibbs andBoltzmannH functions. These are given respectively by

H f f d x d p d x d pln , 1G N N N N3

13

13 3ò= ( )

H N f f d x d pln . 2B 1 13

13

1ò= ( )

Here, f tx p x p, , , , ;N N N1 1 ¼( ) is theN-particle probability density function in 6N-dimensional phase space,while f tx p, ;1 1 1( ) is the single particle probability density function in 6-dimensional phase space of one particle.They are connected by the relationship [6]:

f t f t d x d p d x d px p x p x p, ; , , , , ; . 3N N N N N1 1 1 1 13

23

23 3ò= ¼ ( ) ( ) ( )

Jaynes has proven in [6] thatHGHB, where the difference betweenHG andHB is not negligible for any systeminwhich interparticle forces have any observable effect on the thermodynamic properties. Furthermore, basedon the use of theGibbs canonical ensemble, in [6] it is argued that theGibbs entropy SG=−kHG, and notBoltzmann’s SB=−kHB, is the correct statisticalmechanics form that agrees with thermodynamic entropy. Onthe other hand, reference [6] alsomakes a connection between theGibbs entropy SG and the other definition ofentropywhich is also known as Boltzmann’s, as a logarithmof the number ofmicroscopic states that correspondto themacroscopic state of the system S k WlnB = , where k is the Boltzmann constant. In this paper, we

OPEN ACCESS

RECEIVED

1December 2017

REVISED

27 February 2018

ACCEPTED FOR PUBLICATION

19March 2018

PUBLISHED

3April 2018

Original content from thisworkmay be used underthe terms of the CreativeCommonsAttribution 3.0licence.

Any further distribution ofthis workmustmaintainattribution to theauthor(s) and the title ofthework, journal citationandDOI.

© 2018TheAuthor(s). Published by IOPPublishing Ltd

https://doi.org/10.1088/2399-6528/aab7e1https://orcid.org/0000-0001-6399-7600https://orcid.org/0000-0001-6399-7600mailto:pasko@pmfst.hrmailto:dkuic@pmfst.hrhttp://crossmark.crossref.org/dialog/?doi=10.1088/2399-6528/aab7e1&domain=pdf&date_stamp=2018-04-03http://crossmark.crossref.org/dialog/?doi=10.1088/2399-6528/aab7e1&domain=pdf&date_stamp=2018-04-03http://creativecommons.org/licenses/by/3.0http://creativecommons.org/licenses/by/3.0http://creativecommons.org/licenses/by/3.0

consider only these two definitions. For the sake of simplicity, we call themGibbs andBoltzmann entropy,respectively.

For clarity, it is also important tomention the reference [7], where a numerical calculation of the Boltzmannentropy and the so-calledGibbs volume entropy, as candidates for themicrocanonical entropy, is done on smallclusters for two level systems and harmonic oscilators. However, in view of recent discussions [8–12] on thequestionwhich of these two definitions ofmicrocanonical entropy is justified on thermodynamic basis,numerical results of reference [7] are not decisive. TheGibbs volume entropy, and theGibbs entropy defined inthe sense that is used in our paper, coincide only if themicrostates of energy less than or equal to the valueE areonly allowed and they have equal probabilities.

The reference [13] is interesting because it introduces an assumption of some internal structure ofmicrostates. The author P. Attard in this way defines the entropy of amicroscopic state, which is basically thelogarithmof theweight (weight is proportional to but not equal to probability) assigned to themicroscopic state.In the next step, the entropy of themacroscopic state is defined in an analogousway. Than the author claims thatthe entropy of statisticalmechanics is the logarithmof the sumof all weights, which is equal regardless if onesums theweights over allmicroscopic, or over allmacroscopic states. In this way, this (so called total) entropydiffers from theGibbs-Shannon entropy by an additional term that is equal to the average entropy ofmicroscopic states. Thefirst problemwith this approach are the entropies ofmicroscopic states. Disregardingthat this is a departure from the standard approach that assumes zero entropy for amicroscopic state, themainproblemwith it is that theweight of themicroscopic state is not a uniquely defined quantity such as theprobability, but only proportional to it. The second problem concerns with the definition of entropy as thelogarithmof the total weight of allmacroscopic states. In thatway, one can not assign this entropy to anyparticularmacroscopic state. In contrast to our analysis, which shows that if the Boltzmann andGibbs entropiesare defined in the sense used throughout this paper then they lead practically to same results inmost cases,reference [13] denies the validity of the use ofGibbs-Shannon formof entropy in statistical physics.

Attard is not the only authorwho has introduced the entropy of amicroscopic state. This was done earlier byCrooks [14]whodefined, based on information theoretic arguments, the entropy ofmicroscopic state of asystem as p xln- ( ), where x specifies the positions andmomenta of all the particles. According toCrooks, this isthe amount of information required to describe themicroscopic state given that its occurs with a probability p(x). Crooks definition of entropy ofmicroscopic state is based on the state probability which is a uniquelydefined quantity. On the other hand, Attard’s definition [13] relies on the ambiguously definedweight of amicroscopic state. The definition given byCrooks enabled him to derive a general version of the fluctuationtheorem for stochastic,microsocopically reversible dynamics [14].

Furthermore, Seifert [15] has adaptedCrooks definition to the stochastic trajectory of a colloidal particle,using the Fokker-Planck equation that alongwith the initial conditions determines the probability as a functionof the space and time coordinates of a particle. In this way Seifert [15]has derived fluctuation theorem forarbitrary initial conditions and arbitrary time dependent driving over afinite time interval.

In information theory, Shannon had searched for the function that has to describe information on the set ofrandom events whose probabilities are (p1, ..., pn ), under well known consistency conditions [19, 20]. He provedthat the only function that satisfies the imposed conditions is, up to amultiplication factor,

H p plog . 4ni

n

i i1

å= -=

( )

It was named information entropy. It is equal to theGibbs definition of entropy (10). It is interesting to note thattwo completely different approaches lead to the same expression.More aboutGibbs-Shannon definition ofentropy interested reader canfind in references [20, 21].We note here only that Jaynes took the identity betweenthese definitions as the starting point for his principle ofmaximumentropy known asMaxEnt, and then usingthis principle derived the canonical and grand canonical distributions [22], first introduced byGibbs [23]. As animportant example of a successful application of this principle in standard statistical physics, we refer to thepaperwritten by Lee andPresse [24], where they show thatMaxEnt formulation for a small subsystem thatexchanges energy with the heat bath, follows from the partialmaximization of theGibbs-Shannon entropy of thetotal, isolated large system, over the bath degrees of freedom.

The aimof this paper is to establish the relationship between the Boltzmann andGibbs definitions of entropyin a clear way, without putting in question these standard definitions.

The paper is organized in the followingway. In the next section general relationship between Boltzmann andGibbs definition of entropy is established. By doing this, the condition for applicability of standard statisticaldescription is derived.

For pedagogical reasons it is shown, in the third section, that Boltzmann andGibbs definition of entropy leadto the same result for a systemwithN distinguishable and independent particles with two equally probable singleparticle states.

2

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

The result of the third section is generalized in fourth section to the systemofN distinguishable andindependent particles withm single particle states with nonuniformprobabilities.

Finally, in the last section the results are summarized.

2.General relationship between boltzmann and gibbs entropy

Weconsider a generalmacroscopic systemwhich can be in differentmacroscopic states that form together a setA. In order to bemore specific, eachmacroscopic state a AÎ is determined by the values of nmacroscopicquantities (a1, ..., an) defining it. Let us assume that for anymacroscopic state a AÎ the systemof interest can beinW(a) possible configurations (themultiplicity)which are denoted by ia=1,K,W(a). Also, all configurationscorresponding to amacroscopic state a have equal probabilities p p aia = ( ). This is the principle of equal a prioriprobabilities applied to configurations corresponding tomacroscopic state a. Then the probability of amacroscopic state a is equal to

P a W a p a . 5=( ) ( ) ( ) ( )

According to the Boltzmann definition, entropy of a system in amacroscopic state a, apart from themultiplicative factor given by Boltzmann constant k, is equal to

S W aln . 6B = ( ) ( )

Entropy, like other thermodynamic quantities, in equilibrium state assumes itsmean value. This holds in amicrocanonical ensemble, too. Clearly, formulation of the second law for isolated systems based onClausiusinequality relies on the assumption that the relative difference between themost probable, which is at the sametime themaximum, value of entropy, and amean value of the entropy vanishes in the thermodynamic limit,N ¥ andN/V=const. The reason is that, in the thermodynamic limit, themean values ofmacroscopicquantities correspond closely to their values associated to themost probablemacroscopic state. In themicrocanonical ensemble, themost probablemacroscopic state can be realized by overwhelmingly greatestnumberWmax ofmicroscopic configurations of the system, and therefore, in that state the Boltzmann entropy

Wln max ismaximum. In other ensembles, nonuniformprobabilities of individual configurations also come intoplay.One of the results of this paper shows that, in the thermodynamic limit, the relative difference between themost probable value and themean value of Boltzmann entropy vanishes for all systemswhose probability (5) of amacroscopic state is given by themultinomial distribution.

Themean value of the Boltzmann entropy is obtained by averaging SBwith respect to the probability ofmacroscopic statesP(a):

S P a W aln , 7Ba Aå=Î

( ) ( ) ( )

where the summation goes over allmacroscopic states a AÎ . Using (5)we canwrite SB as

S W a p a p a P a P aln ln . 8Ba A a Aå å= - +Î Î

( ) ( ) ( ) ( ) ( ) ( )

Remembering that all configurations belonging to amacroscopic state ahave equal probabilities p a pia=( ) , foria=1,K,W(a), we then have from (8) that

S p p P a P aln ln . 9Ba A i

W a

i ia A1a

a aå å å= - +Î = Î

( ) ( ) ( )( )

Thefirst term in (9) is theGibbs entropy of the systemwhose probability distribution of configurations is pia,a A i W a, 1, ,aÎ = ¼ ( ),

S p pln . 10Ga A i

W a

i i1a

a aå å= -Î =

( )( )

The summation in (10) goes over all configurations of the system. From (9) and (10)we then obtain a generalrelationship between theGibbs entropy SG and themean value of the Boltzmann entropy SB:

S S P a P aln . 11G Ba Aå= -Î

( ) ( ) ( )

The difference between SG and SB is the second term in (11). The interpretation of this term is the following; wesee that it has theGibbs-Shannon form for entropy of a probability distribution. However, unlike theGibbsentropy (10), this is the entropy of a probability distribution P(a) formacroscopic states a AÎ of the system.Relation (11) is derivedwith the proviso on the configuration probability p p aia = ( ), a A i W a, 1, ,aÎ = ¼ ( )and puts themean value of the Boltzmann entropy aswell as Gibbs definition of the entropy in the following

3

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

order S SG B . The exact equality, in contrast to the one defined in the thermodynamic limit discussed infollowing text, holds in a case when all allowed configurations belong only to onemacroscopic state.

The statistical description relies on the assumption that relative fluctuations of themacroscopic quantitiesvanish.Macroscopic states are determined by the values of a set ofmacroscopic quantities. Itmeans that in thethermodynamic limit the probability P(a) of themacroscopic state becomes sharply and overwhelminglydistributed around themost probablemacroscopic state characterized by themean values ofmacroscopicquantities. Therefore, in the thermodynamic limit, the second term in (11) becomes negligible relative to themeanBoltzmann entropy and theGibbs entropy, which can be taken to be equal. The second term in (11) can becalled entropy offluctuations of themacroscopic state, or entropy offluctuations.

However, there are exceptions, when the fluctuations of themacroscopic quantities do not vanish in thethermodynamic limit. The simplest example is the ferromagnetic Isingmodel in the zero externalmagnetic field,where at temperatures below the critical temperatureT

P nW n N

n n2

1

2. 13

N N N N

2 2

= =- +( ) ( )( )

( ) !! !

( )

In probability theory, this is known as the binomial distribution for the number of occurrencesN1 andN2 of twomutually exclusive events, obtained inN1+N2=N repeated independent trials, for the special case when theprobabilities of the two events are equal p p1 2

1

2= = . The binomial distribution can be generalized to the case

when there aremmutually exclusive events or outcomes in each of theN repeated independent trials; thisgeneralization is known as themultinomial distribution, elaborated in appendix A and used throughout of thispaper.

Themultiplicity factor (12) for the binomial distribution is also called the binomial coefficient. In the casewhen there aremmutually exclusive events in each of theN repeated random trials, themultiplicity factor, i.e.the number of possible realizations of such an experiment is given by the generalization, known as themultinomial coefficient (B.1) given in appendix B. For the casewhen the probabilities of all of the total ofmmutually exclusive events are equal p pm m1

1= = = , themaximumof themultinomial coefficient (B.1) and

themaximumof themultinomial distribution (A.1) coincide at N NmN

m1= = = . Analogously, for the

binomial coefficient and binomial distribution, in case when twomutually exclusive events have equalprobabilities p p1 2

1

2= = , themaxima for both coincide at N N N1 2 2= = . The reader can easily check these

simple but very important facts.For small deviations n1=−n and n2=n of the single particle occupation numbers N n

N1 2 1= + and

N nN2 2 2= + , the logarithmof the binomial coefficient is appropriately expanded up to the order n2 from the

maximum situated at N N N1 2 2= = . This is obtained using the analogous expansion (B.9) for the logarithmofthemultinomial coefficient derived in appendix B, and putting, for the case at hand given here,m=2,p p1 2

1

2= = and n1=−n and n2=n:

W n N NN n

Nln 1 ln 2

1

2ln

1

2ln 2

2

2. 14

2

p» + - - - ⎜ ⎟⎛⎝

⎞⎠( ) ( ) ( ) ( )

For a large number of particlesN?1 and small deviations 1nN

2 this approximation is sufficient.Furthermore, by exponentiating equation (14) one obtains

W nN

e22

. 15NnN

2 2

p= -( ) ( )

Then, using equation (13) it is easy to see thatfluctuations n1=−n and n2=n of the single particle occupationnumbers N nN1 2 1= + and N n

N2 2 2= + are approximately Gaussian distributed aroundmost probable values

N N N1 2 2= = :

P nN

e2

. 16nN

2 2

p= -( ) ( )

Furthermore, we can pass on to a continuumapproximation and replace the discrete distribution (13) by acontinuousGaussian form (16), which, as it is written, is properly normalized. The standard deviation, or root-mean-square fluctuation, for theGaussian distribution (16) is

nN

2. 172 = ( )

Interestingly, in the case of a discrete distribution (13), the same result n N22

= for the standard deviation isobtained.One can easily verify this using the expression for the square of standard deviation (A.6) of themultinomial distribution (A.1) in appendix A, for the casem=2, p p1 2

1

2= = .

Again, we emphasize that theGaussian approximation (16) is valid for smallfluctuations 1nN

2 andlarge number of particlesN?1.However, forN?1, largefluctuations around themost probable values ofoccupation numbers N N N1 2 2= = have a very small probability, as is easily seen from (13) and (16). This is due

to themultiplicity factor W n xN2

=( ), which for largeN has a very sharp spike at x=0. The relative width of thespike is given by

xn

N

1, 18

N

2

2

D = = ( )

5

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

and in this sense it becomes narrower as the number of particlesN becomes larger. Consequently forN?1 thisis the reasonwhyW n xN

2=( ) and P n xN2=( ) arewell approximated even for largerfluctuations compared to

1nN

2 by the functions given by (15) and (16).A configuration ismore (in case of the systemof crystal withmolecules in nonoverlapping electronmagnetic

states) or less (in case of classicalmolecules in the container) counterpart to themicroscopic state of the physicalsystem.We exploit the Boltzmann definition of entropy, which for a systemofN independent anddistinguishable particles with two single particle states, apart from the Boltzmann constant k, is

S W nln , 19B = ( ) ( )

whereW(n) is themultiplicity (12) of thismacroscopic state.Mean value of the Boltzmann entropy (19) is thenobtained by averaging W nln ( ) over all possiblemacroscopic states whose probabilities are distributed by P(n):

S P n W nln . 20Bn N

N

2

2

å==-

( ) ( ) ( )

Using (14) and the normalization property of the probability distribution P(n), from (20) one obtains

S N NN

n1 ln 21

2ln

1

2ln 2

2. 21B 2p» + - - -( ) ( ) ( )

Introducing (17) in (21)we finally obtain

S N N1 ln 21

2ln

1

2ln 2

1

2ln 2 . 22B Np= + - - - »( ) ( ) ( ) ( )

This is themean value of the Boltzmann entropy for a systemofN independent and distinguishable particleswith two equally probable single particle states; for a large number of particlesN?1we can take that it is equalto N ln 2. The value of the Boltzmann entropy for amacroscopic state characterized by occupation numbersN nN1 2= - and N n

N2 2= + is obtained from (14) and (19):

S N Nn

NS

n

N1 ln 2

1

2ln

1

2ln 2

2 2. 23B B

2

max

2

p= + - - - = -( ) ( ) ( ) ( )

Equation (23) is the expansion of SB around themaximumvalue

S N N1 ln 21

2ln

1

2ln 2 ln 2 , 24B Nmax p= + - - »( ) ( ) ( ) ( ) ( )

valid for large number of particlesN?1 and smallfluctuations 1nN

2 .Knowing the exact configuration for a systemof distinguishable independent particles, specifies on the other

hand, not only the single particle state occupation numbers, but also the state inwhich each of thedistinguishable particles is found. Accordingly, for a systemofN distinguishable particles with two equallyprobable single particle states, the probabilities of individual configurations of a system are given by

1

2, 25i N = ( )

where configurations are denoted by i=1,K, 2N. Then, according to theGibbs definition of entropy, we havethat its value for the configuration probability (25) is equal to

S ln ln 2 . 26Gi

i iN

1

2N

å= - ==

( ) ( )

The equations (15), (16) and (23) tell us that, for largeN, the probability offluctuations of the Boltzmannentropy SB is exponentially (Gaussian) distributed around itsmaximumvalue S ln 2B Nmax =( ) ( ), which is also itsmost probable value. Furthermore, this value is also, according to (26), equal to theGibbs entropy SG:

S Sln 2 . 27B N Gmax = =( ) ( ) ( )

Themean value of the Boltzmann entropy SB, according to (22) and (27), becomes equal to SB max( ) and SG in thelimit of large number of particlesN?1.

Using (22) and (24), one obtains themean relative fluctuation of the Boltzmann entropy SB around itsmaximumandmost probable value,

S S

S N Nln 2

1, 28B B

B

max

max

1

2- =-

µ( )

( )( )

valid in the limit of large number of particlesN?1.

6

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

4. Boltzmann andGibbs entropy for nonuniform single particle state probabilitiesp1,K, pm

Now, consider a case ofN distinguishable and independent particles with nonuniformprobabilities p1,K, pm ofm single particle states. In analogy to derivation of (13), using (A.1), (B.2) and (B.3) in the general case ofnonuniformprobabilities p1,K, pm, we canwrite for the probability P(n1, ..., nm) of themacroscopic statecharacterized by the deviations n N Ni i i= - of occupation numbers from theirmean values N Npi i= ,i=1,K,m:

P n n W n n p n n, , , , , , . 29m m m1 1 1¼ = ¼ ¼( ) ( ) ( ) ( )

Here, probability P(n1, ..., nm) of themacroscopic state is given as a product of themultiplicityW(n1, ..., nm) ofthemacroscopic state, and equal probability p (n1, ..., nm) of configurations corresponding to it. The probabilityof corresponding configurations is given by

p n n p Np n p, , exp ln . 30mi

m

iNp n

i

m

i i i11 1

i i å¼ = = +=

+

=

⎡⎣⎢

⎤⎦⎥( ) ( ) ( )

For large number of particlesN?1 and small deviations 1nNp

i

i

, using (B.9)we easily obtain

W n nN

Np n

N

NpNp n p

n

Np

, ,

2exp ln

1

2. 31

m

i

mi i

mi

mi i

m

i i ii

mi

i

1

1

11 1 1

2

å å

p

¼ =+

» - + -

=

-= = =

⎡⎣⎢

⎤⎦⎥

( ) !( )!

( )( ) ( )

Then, using (29), (30) and (31)we get themacrostate probability in theGaussian approximation,

P n nN

Np

n

Np, ,

2exp

1

2. 32m m

i

mi i

mi

i

1 11 1

2

å

p¼ = -

-= =

⎛⎝⎜

⎞⎠⎟( ) ( ) ( )

Equations (32) and (A.3) tell us that themacroscopic state characterized by the occupation numbers equal totheirmean values N Npi i= , i=1,K,m, is themost probablemacroscopic state. On the other hand, themacroscopic state with occupation numbersNi=N/m for all i=1,K,m has the greatestmultiplicity (B.1). Inthe casewhen single particle state probabilities are equal p p m, , 1m1 ¼ = , this is also themost probable state, asis obvious from (A.1) and (B.1). However, for nonuniform single state probabilities p1,K, pm, themacroscopicstate with the greatestmultiplicity and themost probablemacroscopic state do not coincide, in contrast to thecase of uniform single state probabilities.

As physically very interesting example, we chooseGibbs canonical distribution. In case of a systemofNdistinguishable and independent particles withm single particle states, probabilities of configurations thatcorrespond to themacroscopic state with occupation numbersN1,K,Nm are equal

p N Ne

Z, , . 33m

i

m

N11

NiEikT

¼ = =-

( ) ( )

Here,Z is a single particle partition function

Z e , 34i

m

1

EikTå=

=

- ( )

andEi is the energy of the i-th single particle state. The partition function of the systemofN particles is then theN-th power of the single particle partition functionZ.

We note here that the configuration probability (33) comes from (30), too. It follows simply by rewriting (30)in terms of occupation numbers introduced in (B.2) and using theGibbs single particle state probabilities

pe

Z, 35i

EikT

=-

( )

for i=1,K,m.Macrostate probability (29), written in the formwhich uses the occupation numbersN1,K,Nmas the variables, reads

P N N W N N p N N, , , , , , . 36m m m1 1 1¼ = ¼ ¼( ) ( ) ( ) ( )

Then using (36) and (B.1) it is easy to obtain, for the example ofGibbs configuration probability (33), theprobability of themacroscopic state characterized by single particle state occupation numbers N N, , m1 ¼ .

It is convenient, for nonuniformprobabilities p1,K, pm of single particle states towrite the occupationnumbers in the formused in (B.2). Themean value of Boltzmann entropy is then obtained by averagingS W n nln , ,B m1= ¼( ) over themacrostate probability P(n1, ..., nm) given by (29).We use the smallfluctuation

7

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

approximation of W n nln , , m1 ¼( ) given by the expansion (B.9) around themost probable state, characterizedby occupation numbers equal to theirmean values N Npi i= , i=1,K,m. Using (A.4), we then obtain

S N p p N Np

m n

Np

ln1

2ln

1

2ln

1

2ln 2

1

2. 37

Bi

m

i ii

m

i

i

mi

i

1 1

1

2

å å

åp

=- + -

--

-

= =

=

( )

( ) ( )

For the squares of standard deviations (variances) ni2 we use (A.6) andfinally, using the normalization property

of single particle state probabilities, equation (37) becomes

S N p p N Npm

N p p

ln1

2ln

1

2ln

1

2ln 2 1

ln . 38

Bi

m

i ii

m

i

i

m

i i

1 1

1

å å

å

p=- + - --

+

»-

= =

=

( ) [ ( ) ]

( )

Here, for largeN?1, themean value of Boltzmann entropy of the systemofN distinguishable independentparticles withm single particle states with probabilities p1,K, pm is equal to the last expression in (38).

Now,we showbriefly that the last expression in (38) is equal to theGibbs entropy of this system. Probabilityof an individual configuration for the systemofN distinguishable independent particles with single particle stateprobabilities p1,K, pm is given by the corresponding product ofN of these probabilities. The product isdetermined by a state of each individual particle in this configuration. This is written as

p p , 39i j

N indices

i j

N times

, ,... = ( )

where each ofN indices denotes the state of an individual particle, and all indices go from i j m, ,... 1, ,= ¼ ,thus denotingmN different configurations of a system. According to theGibbs definition of entropy, forconfiguration probability (39) it is equal to

S N p pln ln . 40Gi j

m

N indices

i j i ji

m

i i, ,... 1

, ,... , ,...1

å å= - = -= =

( )

By comparing (38) and (40)we see that,

S S N Npm1

2ln

1

2ln

1

2ln 2 1 , 41G B

i

m

i1

å p= - + + - +=

( ) [ ( ) ] ( )

and furthermore, that for large number of particlesN?1, themean value of the Boltzmann entropy becomesequal to theGibbs entropy,

S S . 42B G» ( )Equations (41) and (42) are general equations valid for systems ofN distinguishable independent particles withm single particle states, and therefore they are also valid for the special case discussed in the previous section.Clearly, they represent a consequence of the negligibility of the entropy offluctuations in equation (11) in thethermodynamic limit.

However, as it has been pointed out in this section, for nonuniform single particle state probabilities, themacroscopic state withmaximummultiplicity (and therefore with themaximumBoltzmann entropy), and themost probablemacroscopic state do not coincide. From (32) and (B.9), we see that in the smallfluctuationexpansion of S W n nln , ,B m1= ¼( ) around themost probablemacroscopic state characterized byn1=L=nm=0, the Boltzmann entropy is equal to

S N p p N Npm

n pn

Np

ln1

2ln

1

2ln

1

2ln 2

ln1

2. 43

Bi

m

i ii

m

i

i

m

i ii

mi

i

1 1

1 1

2

å å

å å

p=- + - --

- -

= =

= =

( ) ( )

( )

The value of the Boltzmann entropy SBfluctuates, with probability approximated by (32), around itsmostprobable value for n1=L=nm=0, which is equal to thefirst line in (43). Themost probable value of SB isassociated to themost probablemacroscopic state, where the systemdescribed by the distribution (29) isexpected to spendmuchmore time than in any othermacroscopic state. Of course, this expectation is in someextent analogous to the ergodic hypothesis, that the time averages of physical quantities for a systemdescribed bymicrocanonical ensemble coincide with their statistical averages, and therefore that they are very close to theirstatisticallymost probable values. However, concepts of ergodic theory are a topic for themselves and too

8

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

complex to be discussed in this paper. This is particularly true for systems not described by amicrocanonicaldistribution, like the ones described by equilibriumGibbs canonical distribution.More importantly, we haveshown that for a systemofN distinguishable independent particles with general single particle state probabilities,themost probable value of the Boltzmann entropy SB becomes equal to theGibbs entropy SG,

S S N p pln . 44B Gi

m

i imost prob.1

å= = -=

( ) ( )

in the limit of large number of particlesN?1. In the same limit, according to (38) and (44), themean value SBof Boltzmann entropy becomes equal to them. Themean relative fluctuation of the Boltzmann entropy arounditsmost probable value is

S S

S N p p Nln

1, 45

B B

B

m

i

mi i

most prob.

most prob.

1

2

1å-

=-

-µ -

-

=

( )( )

( )

and it vanishes forN?1.

5. Conclusion

It is widely accepted to extrapolate theClausius definition of an equilibrium state of an isolated, ormoreprecisely thermally isolated, system as the state withmaximumentropy. Strictly speaking, from the statisticalpoint of view, a system in equilibriumdoes notfind itself in the state ofmaximumbut rathermean entropy.Averaging the Boltzmann entropywith respect to probabilities ofmacroscopic states, we come to theequation (11)which is the central result of this work. It tells us thatmeanBoltzmann entropy is equal to theGibbs entropyminus a quantity that includes fluctuations ofmacroscopic physical quantities. This quantity hasthe formofGibbs entropy andwe have named it entropy offluctuations. The equality betweenmeanBoltzmannandGibbs entropy depends on the ratio betweenfluctuation entropy andmeanBoltzmann entropy. If this ratioconverges to zero in thermodynamic limit themeanBoltzmann entropy becomes equal to theGibbs entropy.

In this workwe have considered a systemofN distinguishable and independent particles withm singleparticle states. It is shown that the ratio between the fluctuation entropy and themost probable value ofBoltzmann entropy vanishes as 1/N in the thermodynamic limit. In this way there is no practical differencebetweenmean Boltzmann andGibbs entropy.

From the pedagogical reasons, as a special case, the systemof particles with two equally probable singleparticle states was considered first. This case has a common characteristic with themicrocanonical ensemble anduniform ensemble that all configurations available to the system are equally probable. It is found thatmeanBoltzmann entropy converges tomaximumentropy in the thermodynamic limit, in accordance withClausisusinequality.

In the case of nonuniform single particle state probabilities,meanBoltzmann entropy is no longer close to itsmaximumvalue, as it is expected, but it converges toGibbs entropy.

It comes out that the ratio betweenfluctuation entropy andmeanBoltzmann entropy is a criterion of validityof the standard statistical physics approach. If this ratio does not vanish in the thermodynamic limit, standardstatistical approach should bemodified, or extendedwith othermethods like renormalization group.

It is pointed out that the ferromagnetic Isingmodel in the zero externalmagnetic field is an example ofsystemwith large fluctuation entropy for temperatures close and below the critical one. It is self-evident thatstandard statistical approach can give nonphysical values ofmeanmagnetization in this case.

Additional cases arematerials close to critical point in the (p,V ) diagram. The critical opalescence is theresult of largefluctuation entropy. One expects that critical phenomena in general are characterized by states oflargefluctuation entropy.

AppendixA.Multinomial distribution

Herewe use definitions from reference [25].Multinomial distribution [25] is a joint distribution of random variables that is defined for all nonnegative

integersN1,K,Nm satisfying the condition N Nim

i1å == ,Ni=0,K,N, i=1,K,m, by the formula

P N NN

Np, , . A.1m

i

mi i

m

iN

1

1 1

i

¼ =

= =

( ) !!

( )

Parameters of the distribution areN and p1,K, pm, and satisfy the condition p 1im

i1å == , pi0, i=1,K,m.In probability theory, (A.1) is the probability for the number of occurrencesN1,K,Nm ofmmutually exclusiveevents, obtained in N Ni

mi1å == repeated independent trials, where the probabilities of the events in each trial

9

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

are p1,K, pm.Multinomial distribution (A.1) is a natural generalization of the binomial distribution, which is itsspecial case form=2.

Themathematical expectations ofNi are, for i m1, ,= ¼ , given by

N N P N N Np, , . A.2iN N

N N

N

i m i, , 0,

1m

i

mi

1

1

å= ¼ =å

¼ ===

( ) ( )

Wedefine also, for i=1,K,m, the deviations (orfluctuations) ofNi from theirmean values

n N N . A.3i i i= - ( )Obviously, themathematical expectations of ni are, for i=1,K,m,

n 0. A.4i = ( )

Wealso define themathematical expectations of ni2, or equivalently, the variances ofNi. For i=1,K,m, they

are equal to

n N N P N N N N, , . A.5iN N

N N

N

i i m i i2

, , 0,

21

2 2

m

i

mi

1

1

å= - ¼ = -å

¼ ===

( ) ( ) ( )

Specifically, when they are takenwith respect to themultinomial distribution (A.1), one obtains

n Np p1 . A.6i i i2 = -( ) ( )

Appendix B.Multinomial coefficient

In this appendixwe use definitions from references [25] and [26].In combinatorics themultinomial coefficient expresses [26] the number of ways of puttingN different

elements inm different boxes inwhich box i containsNi elements, i=1,K,m, without taking the order inwhichwe put the elements into boxes into account. It is given by

W N NN

N, , B.1m

i

mi

1

1¼ =

=

( ) !!

( )

Nowwe assume that putting individual elements into boxes are independent events, and that for each elementthe probabilities for putting it intom different boxes are p1,K, pm, p 1i

mi1å == . Then the probabilities for the

number of elementsN1,K,Nm, N Nim

i1å == , inm boxes are distributed according to themultinomialdistribution (A.1).

Introducing the deviations (A.3) of N N, , m1 ¼ from themean values (A.2) takenwith respect to themultinomial distribution (A.1), we canwriteNi, i=1,K,m, as

N Np n . B.2i i i= + ( )

Introducing (B.2) in themultinomial coefficient (B.1)we obtain

W n nN

Np n, , . B.3m

i

mi i

1

1¼ =

+=

( ) !( )!

( )

The Stirling formula for x! is valid for large x,

x x x x xln ln1

2ln

1

2ln 2 . B.4p» - + +( !) ( ) ( )

For largeN, such that 1nNp

i

i

also, themultinomial coefficient (B.3) can be calculated using (B.4). One thenobtains

W n n N Np n

N N Np n Np n

N Np nm

ln , , ln ln

ln ln

1

2ln

1

2ln

1

2ln 2 . B.5

mi

m

i i

i

m

i i i i

i

m

i i

11

1

1

å

å

å p

¼ = - +

» - + +

+ - + --

=

=

=

( ) ( !) [( )!]

( ) ( )

( ) ( ) ( )

10

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

Wecanwrite (B.5) in the form

W n n N Nm

Npn

NpNp

n

Np

ln , ,1

2ln

1

2ln 2

1 ln ln 1 . B.6

m

i

m

i

i

ii

i

i

1

1

1

2å

p¼ = + --

- ++

+ +=

⎜ ⎟⎛⎝⎞⎠

⎛⎝⎜⎜

⎞⎠⎟⎟

⎡⎣⎢⎢

⎛⎝⎜

⎞⎠⎟

⎤⎦⎥⎥

( ) ( )

( ) ( )

The expansion of xln 1 +( ) around x=0 up to the order x2 is valid approximately for x 1∣ ∣ and given by

x x xln 11

2.... B.72+ = - +( ) ( )

Using (B.7)we obtain the expansion of (B.6) up to the order ni2,

W n n N p p N Np

mp

Npn

Np Npn

ln , , ln1

2ln

1

2ln

1

2ln 2 ln

1

2

1

2

1 1

2. B.8

mi

m

i ii

m

i

i

m

ii

ii

m

i ii

11 1

1 12

2

å å

å åp

¼ » - + -

--

- + - -

= =

= =

⎛⎝⎜

⎞⎠⎟

⎡⎣⎢

⎤⎦⎥

( ) ( )

( )( )

( )

SinceN?1 and 1nNp

i

i

, i=1,K,m , from (B.8)we obtain

W n n N p p N Np

mn p

n

Np

ln , , ln1

2ln

1

2ln

1

2ln 2 ln

1

2. B.9

mi

m

i ii

m

i

i

m

i ii

mi

i

11 1

1 1

2

å å

å åp

¼ »- + -

--

- -

= =

= =

( ) ( )

( ) ( )

ORCID iDs

Domagoj Kuić https://orcid.org/0000-0001-6399-7600

References

[1] Clausius R 1867TheMechanical Theory ofHeat,With Its Applications to the Steam-engine and to the Physical Properties of Bodies(London: John vanVoorst)

[2] Landau LD and Lifshitz EM2011 Statistical Physics (Oxford, GB: Elsevier)[3] Kittel C 1958Elementary Statistical Physics (NY.USA: JohnWilley& Sons, Inc)[4] Reif F 1967 Statistical Physics, Berkeley Physics Course, Volume 5 (NY.USA:McGraw-Hill BookCompany)[5] FeynmanRP 1972 StatisticalMechanics, a Set of Lectures (Massachusetts:W.A. Benjamin Inc)[6] Jaynes ET 1965Gibbs vs Boltzmann Entropies, AJP 33 391–8[7] Miranda EN2015Boltzmann or gibbs entropy? thermostatistics of twomodels with few particles Journal ofModern Physics 6 1051–7[8] Dunkel J andHilbert S 2014Consistent thermostatistics forbids negative absolute temperaturesNat. Phys. 10 67–72[9] FrenkelD andWarren P B 2015Gibbs, Boltzmann, and negative temperaturesAJP 83 163[10] SwendsenRHandWang J-S 2016Negative temperatures and the definition of entropyPhysicaA 453 24–34[11] Hilbert S,Hänggi P andDunkel J 2014Thermodynamics laws in isolated systems Phys. Rev.E 90 062116[12] CampisiM2015Construction ofmicrocanonical entropy on thermodynamic pillars Phys. Rev.E 91 052147[13] Attard P 2012 Is the Information Entropy the Same as the StatisticalMechanical Entropy? https://arxiv.org/abs/1209.5500[14] CrooksGE 1999 Entropy production fluctuation theorem and the nonequilibriumwork relation for free energy differences Phys. Rev.

E 60 2721[15] Seifert U 2005 Entropy Production along a Stochastic Trajectory and an Integral Fluctuation Theorem Phys. Rev. Lett. 95 040602[16] GarrodC 1995 StatisticalMechanics and Thermodynamics (NewYork:OxfordUniversity Press, Inc.)pp 56–61[17] WilsonKG1971Renormalization group and critical phenomena 1. Renormalization group andKadanoff scaling picture Phys. Rev.B 4

3174WilsonKG1971Renormalization group and critical phenomena 2. Phase space cell analysis of critical behavior Phys. Rev.B 4 3184

[18] Kadanoff L P 1966 Scaling laws for isingmodels near Tc Physics 2 263[19] ShannonCE 1948Amathematical theory of communicationBell Syst. Tech. J. 27 379–423 623–656

ShannonCE andWeaverW1949TheMathematical Theory of Communication (Urbana: University of Illinois Press)Reprinted In[20] Jaynes ET 2011Probability Theory; The Logic of Science (Cambridge, UK:CambridgeUniversity Press)[21] CatichaA 2008 Lectures on Probability, Entropy, and Statistical Physics https://arxiv.org/abs/0808.0012[22] Jaynes ET 1957 Information theory and statisticalmechanics Phys. Rev. 106 620[23] Gibbs JW1902Elementary Principles in StatisticalMechanics (NewHaven: YaleUniversity Press)[24] Lee J andPresse S 2012Microcanonical origin of themaximumentropy principle for open systems Phys. Rev.E 86 041126[25] JohnsonNL andKotz S 1969Distributions in statistics: Discrete distributions (Boston,Mass: HoughtonMifflin Co.)[26] HallM1986Combinatorial Theory (NewYork:Wiley)

11

J. Phys. Commun. 2 (2018) 045002 PŽupanović andDKuić

https://orcid.org/0000-0001-6399-7600https://orcid.org/0000-0001-6399-7600https://orcid.org/0000-0001-6399-7600https://orcid.org/0000-0001-6399-7600https://doi.org/10.1119/1.1971557https://doi.org/10.1119/1.1971557https://doi.org/10.1119/1.1971557https://doi.org/10.4236/jmp.2015.68109https://doi.org/10.4236/jmp.2015.68109https://doi.org/10.4236/jmp.2015.68109https://doi.org/10.1038/nphys2815https://doi.org/10.1038/nphys2815https://doi.org/10.1038/nphys2815https://doi.org/10.1119/1.4895828https://doi.org/10.1016/j.physa.2016.01.068https://doi.org/10.1016/j.physa.2016.01.068https://doi.org/10.1016/j.physa.2016.01.068https://doi.org/10.1103/PhysRevE.90.062116https://doi.org/10.1103/PhysRevE.91.052147https://arxiv.org/abs/1209.5500https://doi.org/10.1103/PhysRevE.60.2721https://doi.org/10.1103/PhysRevLett.95.040602https://doi.org/10.1103/PhysRevB.4.3174https://doi.org/10.1103/PhysRevB.4.3174https://doi.org/10.1103/PhysRevB.4.3184https://doi.org/10.1103/PhysicsPhysiqueFizika.2.263https://doi.org/10.1002/j.1538-7305.1948.tb01338.xhttps://doi.org/10.1002/j.1538-7305.1948.tb01338.xhttps://doi.org/10.1002/j.1538-7305.1948.tb01338.xhttps://arxiv.org/abs/0808.0012https://doi.org/10.1103/PhysRev.106.620https://doi.org/10.1103/PhysRevE.86.041126

1. Introduction2. General relationship between boltzmann and gibbs entropy3. A system of N distinguishable particles with two equally probable states4. Boltzmann and Gibbs entropy for nonuniform single particle state probabilities p1, …, pm5. ConclusionAppendix A.Appendix B.References