Error estimates for solid-state density-functional theory predictions: anoverview by means of the ground-state elemental crystals

K. Lejaeghere,1 V. Van Speybroeck,1 G. Van Oost,2 and S. Cottenier1, 3, ∗

1Center for Molecular Modeling, Ghent University,Technologiepark 903, BE-9052 Zwijnaarde, Belgium2Department of Applied Physics, Ghent University,

Sint-Pietersnieuwstraat 41, BE-9000 Ghent, Belgium3Department of Materials Science and Engineering, Ghent University,

Technologiepark 903, BE-9052 Zwijnaarde, Belgium

This is an Author’s Accepted Manuscript of an article published in Critical Reviews in

Solid State and Materials Sciences 39, p 1 (2014), available online at

http://www.tandfonline.com/bsms and with the DOI: 10.1080/10408436.2013.772503.

Predictions of observable properties by density-functional theory calculations (DFT) are used in-creasingly often by experimental condensed-matter physicists and materials engineers as data. Thesepredictions are used to analyze recent measurements, or to plan future experiments in a rationalway. Increasingly more experimental scientists in these fields therefore face the natural question:what is the expected error for such a first-principles prediction? Information and experience aboutthis question is implicitly available in the computational community, scattered over two decadesof literature. The present review aims to summarize and quantify this implicit knowledge. Thiseventually leads to a practical protocol that allows any scientist – experimental or theoretical – todetermine justifiable error estimates for many basic property predictions, without having to performadditional DFT calculations.

A central role is played by a large and diverse test set of crystalline solids, containing all ground-state elemental crystals (except most lanthanides). For several properties of each crystal, the differ-ence between DFT results and experimental values is assessed. We discuss trends in these deviationsand review explanations suggested in the literature.

A prerequisite for such an error analysis is that different implementations of the same first-principles formalism provide the same predictions. Therefore, the reproducibility of predictionsacross several mainstream methods and codes is discussed too. A quality factor ∆ expresses thespread in predictions from two distinct DFT implementations by a single number. To compare thePAW method to the highly accurate APW+lo approach, a code assessment of VASP and GPAW(PAW) with respect to WIEN2k (APW+lo) yields ∆-values of 1.9 and 3.3 meV/atom, respectively.In both cases the PAW potentials recommended by the respective codes have been used. Thesedifferences are an order of magnitude smaller than the typical difference with experiment, andtherefore predictions by APW+lo and PAW are for practical purposes identical.

CONTENTS

I. Introduction 2

II. Predicting experimental properties bymeans of DFT 3A. Computational recipes 3B. Comparing theory and experiment 5

III. Intrinsic errors 6A. Test set preparation 6B. Statistical analysis 7

1. Linear regression 72. Eliminating outliers 83. Predicting experiment 11

∗ Stefaan.Cottenier@UGent.be

C. Agreement with experiment 14

1. Errors per materials type 14

2. Errors per property 16

IV. Numerical errors 17

A. Test set preparation 18

B. Agreement betweenimplementations 19

V. Conclusions 20

Acknowledgments 23

Appendix: Calculating the ∆-factor 23

References 24

arX

iv:1

204.

2733

v5 [

cond

-mat

.mtr

l-sc

i] 5

Aug

201

3

2

I. INTRODUCTION

Density-functional theory [1, 2] (DFT) remainsone of the most popular methods to treat bothmodel systems and realistic materials in a quan-tum mechanical way [3–8]. In condensed-matterphysics, DFT is not only used to understandthe observed behavior of solids, but increasinglymore to predict characteristics of compoundsthat have not yet been determined experimen-tally (for example in Refs. 9–11). In both casesthe first-principles results provide a point of ref-erence, either to analyze data from measure-ments or to plan future experiments. It is there-fore essential to have a quantitative idea of theexpected deviation between a DFT predictionof a certain property and the corresponding ex-perimental value. Error estimates are routinelyprovided in experimental physics, but in DFTapplications this is much less common practice.When confronted with a disagreement betweentheory and experiment, one usually resorts tohigher-order levels of theory instead [12–14].

DFT as such is an exact reformulation of quan-tum physics, and does not involve any approxi-mation. From a purely theoretical point of view,it should lead to exact predictions, with no needfor an error estimate. In practice, however,one requires an educated guess for an essen-tial ingredient of DFT: the exchange-correlationfunctional (hereafter referred to as ‘functional’).Apart from this main approximation, there aresome other features that go beyond DFT in theway it is usually applied, such as the failure ofthe Born-Oppenheimer approximation [15, 16]or high-Z radiative and other corrections fromquantum electrodynamics [17–19]. They affectthe results to some extent as well, but theyare generally much less important than the par-ticular choice of the exchange-correlation func-tional. For any of these choices, DFT predic-tions will not agree perfectly with experimentalobservations. Deviations of this kind will be re-ferred to as the intrinsic error for a particularfunctional.

Regardless of the difference with experiment,however, all predictions should be independentof how the DFT (Kohn-Sham) equations aresolved numerically: each of the many availableDFT implementations should give identical re-sults for the same functional. In reality, therewill be some scatter in the predictions of differ-ent codes [20, 21], as each of them introducesa distinct amount of numerical noise. This sec-ond source of fluctuations leads to a numerical

error. It is therefore also important to assessto what extent predicted properties vary amongDFT approaches.The goal of this work is to quantify the knowl-edge about these two kinds of DFT errors, in-trinsic and numerical ones, and — where rele-vant — to review the physics behind them.A way to obtain insight into computational er-rors is by means of benchmark studies, exam-ining the performance of different implementa-tions and functionals for a large set of materialsand properties. For molecular benchmark setsthe intrinsic errors have already been assessed ingreat detail, often with the aim of selecting thebest functional for a particular property (for ex-ample in Refs. 20, 22–26). Similar studies existfor solid-state DFT as well [27–42], but they aremostly limited to a small number of propertiesand/or compounds. In addition, their focus isoften on understanding the differences betweenfunctionals, so they do not lead to quantita-tive and universally applicable error estimateswith respect to experiment. A benchmark thatis really comprehensive, should meet two cri-teria: the number of elements that is includedin the test set should be sufficiently large, andthe crystal structures in the set should be suf-ficiently diverse. This guarantees the transfer-ability of the benchmark conclusions with re-spect to both the intrinsic and the numericalerrors.A natural choice to construct such an exten-sive test set emerges from the periodic table ofelements. By taking the ground-state crystalstructures of all elements [43], the two crite-ria for a comprehensive solid-state benchmarkset are simultaneously fulfilled. All elementsare included, thus trivially fulfilling the first re-quirement. In addition, the corresponding crys-tal structures range from simple hexagonal andcubic configurations to low-symmetry geome-tries, like orthorhombic and monoclinic cells.Of course, extrapolating the obtained insightsto more complex materials, such as multicom-ponent compounds, requires some care. It isnot impossible, however, as will be shown forexample in Sec. III B.An additional advantage of using all elementalcrystals follows from the periodic table’s inher-ent ability to display trends and correlations.The systematic behavior of observable quanti-ties along periods or groups is well known, butthe deviations between DFT predictions and ex-perimental values also appear to follow suchtrends (Sec. III). In particular, the largest er-rors are restricted to distinct regions. So apart

3

from providing a complete test set, the classifi-cation of elements in the periodic table allowsfor an easy visualization and interpretation ofthe data. Furthermore, the elemental materi-als are among the best known and most studiedmaterials on Earth. Experimental data collec-tions are hence rather easy to find, and one canassume with sufficient confidence that the re-ported data are accurate.By means of the ground-state elemental crys-tals, the present review offers an overview of thepower and limitations of solid-state DFT calcu-lations. Although this knowledge is often im-plicitly available in the computational commu-nity, most of it is scattered over two decades ofliterature. The current work therefore summa-rizes and quantifies this implicit knowledge intoa practical protocol, that will allow any scientist– experimental or theoretical – to provide jus-tifiable error estimates for many basic propertypredictions. Intrinsic errors follow from a sta-tistical analysis of the deviations between DFTpredictions and experimental values for a givenfunctional. Subsets of materials for which thedeviations are particularly large are identified,and the reasons for this behavior are discussed.Numerical errors, on the other hand, express towhat extent two independent DFT approachesproduce identical predictions. We will focuson the correspondence between two particularmethods, APW+lo and PAW, by means of rep-resentative mainstream codes. For the PAWmethod, the use of different atomic potentialscan have large effects, but by only consider-ing the sets of potentials recommended by eachcode, we hope to establish a general idea of thePAW error.Instead of performing an extensive level-of-theory study for various functionals, we will fo-cus on one typical example within the gener-alized gradient approximation of DFT (GGA).For this, the PBE functional is chosen, becauseit is known to yield good results for solids ofa wide range of elements and properties [39].Moreover, its popularity [44] guarantees thecomparibility and applicability of our results.Other GGA functionals are expected to dis-play approximately the same behavior, exceptmaybe for very specific material classes. Kurthet al. [27] for instance have shown how fourGGA functionals provide similar trends for boththe equilibrium volume and the bulk modulus.Of course, the determination of the error esti-mates is not limited to GGAs. The presentedmethodology is also applicable to other func-tionals (such as LDA or hybrid functionals)

or first-principles approches (such as Hartree-Fock, GW or RPA [45]).This review is organized as follows. Sec. IIdescribes the computational procedure for allproperties under consideration and discussesthe prerequisites for a sound comparison be-tween theory and experiment. Within Sec. IIIthe differences between DFT-GGA predictionsand experimental values are assessed (intrinsicerrors), whereas Sec. IV focuses on the methodand code dependence of the theoretically deter-mined properties (numerical errors).

II. PREDICTING EXPERIMENTALPROPERTIES BY MEANS OF DFT

A. Computational recipes

DFT computations for five distinct sets of ma-terials properties will be discussed. They canbe divided into energetic (∆Ecoh) and elasticquantities (V0, B0, B1, Cij) [46]. Of coursemany more properties may be determined bymeans of DFT, but the quantities introducedhere are directly available from straightforwardtotal energy calculations.The cohesive energy or atomization energy∆Ecoh is a popular benchmark quantity [20, 22,27–29, 32, 33, 39]. Expressed as an energy dif-ference per atom, it is defined as

∆Ecoh = − (E0 − Eat) (1)

Here E0 represents the energy per atom of thecompound under investigation in its groundstate, i.e. at 0 K and without external stress.One can determine it by means of a standardpressure optimization procedure, or by fitting afew E(V ) data points to an empirical equationof state (EOS) and extracting the equilibriumenergy analytically. In this work the latter op-tion was chosen, using a common third-orderBirch-Murnaghan relation [47] :

E(V ) = E0 +9V0B0

16

[(

V0V

)2/3

− 1

]3B1

+

[(V0V

)2/3

− 1

]2 [6− 4

(V0V

)2/3] (2)

V0 represents the equilibrium volume, B0 thebulk modulus and B1 its pressure derivative.Other equations of state exist as well, but no

4

significant difference with respect to ground-state properties is to be expected.

Eat on the other hand is the energy of one iso-lated atom in its electronic ground state. Sincemany solid-state DFT codes only allow for theuse of periodic boundary conditions, the iso-lated atom needs to be calculated in a periodicunit cell as well. In the present computations afree atom is placed in a big orthorhombic unitcell, such that every atom is surrounded by atleast 15 A of vacuum. In this way one can suf-ficiently suppress spurious interactions betweenperiodic images (< 1 meV). The orthorhombicsymmetry is chosen over e.g. a simpler, cu-bic one, to avoid physically incorrect sphericalstates. After all, the use of a unit cell forces theelectron density to assume the same symmetryas the lattice [48]. This is in most cases onlypossible by means of partial occupation of thedifferent electron orbitals, which is not physical.Lowering the crystal symmetry counteracts thisphenomenon and should lead to strictly integeroccupation numbers. However, some atoms endup with partially filled states, even when thisapproach is applied. In such cases, the occupa-tion numbers have to be fixed manually beforelooking for the usual, self-consistent solution. Inthis work only experimental ground-state elec-tron configurations are used: even when DFTpredicts a different configuration to be more sta-ble [49] (e.g. for W), the experimental occupa-tion numbers are taken in order to guarantee ameaningful comparison to measurements. Onlyfor spin-orbit coupled calculations for the Pbatom it is not possible to impose the experi-mental electronic state. The PBE ground state1S0 is therefore used, and not the experimental3P0 state.

The negative sign in Eq. (1) causes positive co-hesive energies to correspond to stable phases(with respect to atomic decohesion). The othersign convention, however, is commonly used aswell.

One of the key physical properties of a givencompound is its volume. In first-principles cal-culations the equilibrium volume per atom V0can be obtained easily. One either employs anoptimization routine or fits some E(V ) pointsto an empirical equation of state. This is simi-lar to the procedure used to determine E0, andagain the latter option is chosen in this work.

The bulk modulus is closely related to theE(V ) behavior as well. It is proportional tothe curvature of the equation of state at the

equilibrium volume:

B0 = − V∂P

∂V

∣∣∣∣V=V0

= V∂2E

∂V 2

∣∣∣∣V=V0

(3)

It represents the resistance of the unloaded ma-terial to volume change, and hence to uniformpressure. Because it is linked to the curvature ofthe E(V ) relation, B0 is a numerically sensitivequantity. A small deviation at a few data pointsis already able to change its value noticeably, es-pecially when the bulk modulus is small (shal-low EOS). This is increasingly so when only anarrow volume range is inspected.B1 stands for the derivative of the bulkmodulus with respect to pressure, evaluatedat the equilibrium volume:

B1 =∂B

∂P

∣∣∣∣V=V0

=∂

∂P

(V∂2E

∂V 2

)∣∣∣∣V=V0

(4)

It is a third-order derivative of the energy andhence describes effects that are one order highereven than the bulk modulus. It is related tothe volume-dependence of the E(V ) curvature.B1 is therefore the most sensitive elastic quan-tity discussed in this study. Again, both thebulk modulus and its pressure derivative are ob-tained from fitting an EOS to calculated E(V )data points.The mechanical behavior of a crystal cannotbe described solely by means of the bulk mod-ulus. When anisotropic deformations are ap-plied, other elastic constants come into playas well. The full set of these constants makes upthe stiffness matrix C. It represents a tensor ofrank 2 and relates (small) cell strains to the cor-responding stresses via Hooke’s law σ = C · ε.C is a symmetric 6 × 6 matrix, containing 21independent constants at the most. In the caseof hexagonal crystals five distinct values remain(C11, C12, C33, C13, and C44), while for cubiccompounds there are only three (C11, C12, andC44). The Cij parameters can also be trans-lated into more general elastic moduli, such asYoung’s modulus E, the shear modulus G andPoisson’s ratio ν. Even the bulk modulus can beobtained from a simple combination of the Cij .In addition, the elastic constants are known torelate to structural stability and various otherimportant physical properties [50, 51].Several methods are available to obtain the elas-tic constants from first principles, either by re-lating energy and strain [52] or stress and strain[53, 54]. In most cases a stress-based proce-dure is preferred, because it is inherently faster.

5

However, it requires an ab initio code that candetermine the stress tensor. In a first step thecell pressure components are then extracted fora minimal set of deformed geometries. Togetherwith the corresponding strains, this results in asystem of linear equations. Solving that systemyields the required elastic constants. When itis important to obtain an accurate value of Cij ,one should construct an overdetermined system,by applying the same strain sets at differentmagnitudes. The elastic constants can then beretrieved by using a least-squares method.

B. Comparing theory and experiment

When a DFT prediction is compared to a num-ber from experiment, the corresponding ambi-ent conditions should be as identical as possi-ble. This means in the first place that the ex-perimental result should refer to 0 K. Moreover,the measurement should be corrected for zero-point vibrational effects, which are not presentin standard DFT calculations. The followingparagraphs discuss how to extrapolate the ex-perimental values to absolute zero and correctthem for zero-point vibrations.For the cohesive energy it takes little effort tomatch up theory and experiment consistently.Experimental data at low temperatures are inmost cases available. Only the zero-point en-ergy ζ hinders a direct comparison between 0 Kand experiment. From quantum mechanics thisquantity is known to be 3

2~〈ω〉, with 〈ω〉 the av-erage phonon frequency. The latter can be es-timated from Debye theory, where it is propor-tional to the maximum vibrational frequency,and hence to the Debye temperature ΘD. Thezero-point energy correction becomes [55]

ζ =9

8kBΘD (5)

Theoretical cohesive energies can only be com-pared to experiment if this contribution isadded to the experimental values (added, dueto the chosen sign convention in Eq. (1)).When no experimental value is available, ΘD

can be estimated. Here the Debye-Gruneisenapproximation [56]

ΘD = 0.617~kB

(6π2)1/3

V1/60

(B0

M

)1/2

(6)

will be used. Both V0 and the mass M are ex-pressed per particle, corresponding to a single

atom for most materials. For dimeric crystals,however, the diatomic molecule is chosen as aunit of repetition. The regular, room tempera-ture experimental values for B0 and V0 are filledin, except when the difference with low temper-ature results (see further) is significant. This isthe case for Cl, Br, and I.Thermal volume corrections consist of twoparts. Assuming to have a room temperaturemeasurement at one’s disposal, the first stepconsists in accounting for thermal expansionfrom absolute zero to ambient temperature:

∆V (1)

V=

∫ Trt

0

αV (T ) dT (7)

αV (T ) represents the temperature-dependentvolume expansion coefficient. It is zero at 0 Kand αV,rt at room temperature (Trt). Since

∆V (1) constitutes only a small correction withrespect to the total volume V , Eq. (7) will beapproximated here as

∆V (1)

V≈∫ Trt

0

αV,rtT

TrtdT =

αV,rtTrt2

(8)

In a limited number of cases the experimentalexpansion coefficient is not known. It can thenbe estimated from an empirical correlation tothe ‘moleculization’ energy [57]:

αV,rt = 3× 48.14 · 10−6 eV/K/atom

∆Emol(9)

∆Emol is defined as the energy difference peratom between the crystalline material and itsgaslike molecules. For elements with an atomicgas phase, it reduces to the atomization energy(cohesive energy). In the absence of experimen-tal data on both αV,rt and ∆Emol, Eq. (9) iscompleted with DFT values.A second modification is again due to zero-pointeffects. Because of the volume-dependence ofthe zero-point energy ζ, the equilibrium volumeis shifted slightly. According to Alchagirov et al.[37, 55], this small difference per atom amountsto

∆V (2) =(B1 − 1)ζ

2B0=

9

16(B1 − 1)

kBΘD

B0(10)

Dacorogna and Cohen [58] propose an alterna-tive definition of the zero-point volume shift.They obtain a similar formula, but with B1

instead of B1 − 1 in Eq. (10). However, themathematical expression is preceded by somesignificant simplifications. When calculating

6

zero-point effects it is therefore advisable to useEq. (10) instead, especially when B1 is small.For the bulk modulus thermal effects shouldbe taken into account as well. A first contribu-tion originates in the thermal expansion of thematerial. Similar to ∆V (1), a correction ∆B(1)

can be determined too. Roughly approximatingthe relevant behavior, one can write [59]

∆B(1) = B1 · P(

∆V (1))

= −B0B1∆V (1)

V0(11)

On the other hand, the effect of zero-point vi-brations on the bulk modulus boils down to [55]

∆B(2)

B0= −∆V (2)

V0

[1

2(B1 − 1)

+2

B1 − 1

(2

9− 1

3B1 −

1

2B0B2

)](12)

B2 stands for the second-order derivative of thebulk modulus with respect to pressure. It is ahighly sensitive parameter and very difficult toextract from a few E(V ) data points. In addi-tion, B2 is not included in Eq. (2) and a higher-order Birch-Murnaghan fit should be applied.Instead, the present work will use the intrinsicBirch-Murnaghan value:

(B0B2)BM

= B0∂2

∂P 2

(V∂2EBM

∂V 2

)∣∣∣∣V=V0

= −143

9+ 7B1 −B2

1 (13)

There are other possibilities as well [55], rang-ing from a different equation of state to an accu-rate numerical determination of B2. In order toestablish the small correction ∆B(2), however,this more consistent approach suffices.Since it is already hard to accurately measure ahigh-order parameter like B1 or to determine itfrom first principles, zero temperature modifica-tions will often be negligible compared to exper-imental or computational errors. B1 is thereforenot adjusted to incorporate thermal expansionor zero-point effects.No thermal corrections are applied to the elasticconstants Cij as well. One can however imaginea modification similar to that of Dacorogna andCohen [58] for the bulk modulus:

∆C(m)ij =

∂Cij∂P

· P(

∆V (m))

= −B0∂Cij∂P

∆V (m)

V0(14)

with m = 1 to account for thermal expansionand m = 2 for zero-point effects. Unfortunatelyexperimental data about the pressure derivativeof the elastic constants are scarce.

III. INTRINSIC ERRORS

A. Test set preparation

In order to establish statistically justified intrin-sic error estimates, the ground-state elementalcrystals at 0 K will be used as a benchmark set.Pettifor [60] lists these crystal structures, basedon an overview by Villars and Daams [43]. How-ever, in some cases literature suggests anotherphase to be even more stable at low temper-atures. In order to ensure the use of 0 K cellgeometries as much as possible, such an alter-nate structure is taken for boron [61], nitrogen[62], oxygen [63], and sulfur [64]. Tab. I presentsan overview of all structures used in the currenttest set.Using a 0 K benchmark set entails two distinctadvantages. On the one hand some elementsonly crystallize just above absolute zero. Col-lecting both 300 K and 0 K compounds in oneset might then seem a bit inconsistent. On theother hand this approach facilitates the extrap-olation from the experimental temperature to0 K, as there are no phase transformations alongthe way.All structures are considered in their stress-freeground state. This means that, when the spacegroup allows some freedom in the internal posi-tions, an optimization with respect to the totalenergy is necessary. This optimization proce-dure calls for a fast and well-accepted DFT algo-rithm. The projector augmented wave method[65, 66] (PAW) as implemented in VASP [8, 67](version 5.2.2) fulfills both criteria. The ele-mental crystal structures have therefore beenrelaxed by means of this code, using the rec-ommended PAW atomic potentials listed in themanual [68]. A force convergence criterionof 0.01 eV/A was set. All calculations havebeen performed using the tetrahedron methodwith Blochl corrections [69], while the reciprocalspace was sampled by means of a Monkhorst-Pack grid [70]. Further computational detailsfor the calculations are given in the Supplemen-tary Material [71].The equilibrium structure has been obtainedin two stages. For the determination of theequilibrium volume a uniformly spaced 13-point

7

Table I. Ground-state crystal structures for all elements up to radon. Both the space group number and thePearson notation are given (with hRx standing for x atoms in the hexagonal setting of the rhombohedralunit cell)

EOS (up to V0 ± 6 %) has been calculatedand fitted to a least-squares third order Birch-Murnaghan relation (see Sec. II A). Only for anumber of shallow E(V ) curves — in particularfor H, N, S, the halogens, and the noble gases —an increased volume range turned out to be nec-essary. For each of the 13 crystal volumes, theatomic positions and the cell shape have beenindividually optimized. In a second step, thecrystal has been reinitialized at the fitted V0and has then been optimized again.

These optimized crystal structures form thedefinitive test set (submitted to the COD [72]and ICSD [73] crystallographic databases). Foreach of them, most of the properties discussedin Sec. II A have been determined in order toquantify the difference between PBE and ex-perimental values (Sec. III B 3). The DFT partof the comparison has been performed by meansof VASP, using the settings mentioned earlier.They allow to converge all energy differences upto a few meV per atom at the most. For O andCr (antiferromagnetic), Mn (ferrimagnetic), Fe,Co, and Ni (ferromagnetic), spin polarizationhas been taken into account, while for the heav-iest elements (as from Lu) spin-orbit contribu-tions have been incorporated. At that pointrelativistic effects beyond the scalar-relativisticapproach become important, as will be shownlater (Tab. VII).

The analysis in Sec. III B 3 will not show the rawcalculated data, but will rather elaborate on thedeviation between theory and experiment. Thefirst-principles results and the thermally cor-

rected experimental numbers [40, 43, 46, 62, 74–106] have been included in the SupplementaryMaterial [71]. A tabulation of calculated andexperimental values for the elastic constantsCij was published before by Shang et al. [40]for most of the present benchmark set. InSec. III B 3 their data are used. Only for theexperimental numbers of Ba we found more re-alistic results elsewhere [99, 100]. Since the au-thors only considered bcc, fcc and hcp struc-tures, this implies that for Li and Na a differentgeometry was applied than in the rest of thiswork (bcc instead of hR9). Moreover their re-sults are based on a PW91 functional [107, 108],rather than the PBE approximation employedin the rest of this work. Although these GGAapproaches yield different results in a few situ-ations [109], they are in most cases very similarand for the elastic constants no significant de-viation is expected.

B. Statistical analysis

1. Linear regression

Benchmark studies usually analyze the differ-ence between DFT and experiment statistically.The most common characteristics investigatedare the mean error (signed) and the mean abso-lute error (unsigned). However, this approachimplicitly assumes that the offset between DFTpredictions and experimental results is the samefor large and small values. For strictly positive

8

Figure 1. (Color online) The intrinsic error betweenexperiment and DFT can be decomposed in a sys-tematic deviation and a residual error bar. Sys-tematic differences are quantified by comparing thelinear regression line X = βT (red, dashed) to thebisector X = T (black, full). The residual error bar(green arrows) is defined as the standard deviationof the regression errors (SER)

quantities a relative shift seems more reason-able. The present analysis therefore explicitlytreats relative deviations, in addition to the re-maining scatter on that trend. This is done bymeans of a linear regression between DFT dataand experimental results.The linear regression is performed by means ofa least-squares fit, from which we obtain theslope as well as the scatter with respect to theregression line [110]. The model hence presumesthat a perfect correlation between experimentX and theory T exists, distorted by a randomerror ε centered around a zero mean: X = βT +ε. If the exact exchange-correlation functionalwere known, it would approximately lead to ε =0 and β = 1. In practice, comparing the least-squares estimate of β to 1 offers a good measureof any systematic deviations, while the standarddeviation of ε (also denoted as standard errorof the regression or SER) expresses the residualerror bar (see Fig. 1).Although the experimental community com-monly employs the nomenclature ‘systematicerrors’ and ‘non-systematic errors’, we willavoid using the second term. Non-systematicerrors suggest a certain degree of randomness,which is not present in DFT. When one per-forms the same experiment several times, the re-sults are spread around a mean value. In DFT,repeating the same calculation always yieldsidentical results. In contrast to experimen-tal error analysis, the spread of deviations in

DFT results becomes apparent only when pre-dictions for many different compounds are com-pared to experiment, i.e. when a benchmarkset is used. The DFT scatter is then causedby some (subsets of) crystals being describedbetter by the functional than others. Intrin-sic error bars are hence fundamentally differentfrom experimental error bars as well. The sys-tematicness of DFT also appears when study-ing results within a certain family of materials:similar systems behave almost identically, prov-ing the DFT spread not to behave randomly atall. An excellent example is found in literature,where the correspondence with experiment dis-plays much less scatter when chemically similarcompounds are involved [42].

As an additional side note, one should be awarethat experimental error bars have not beentaken into account in any way. In fact, thestatistical model should be X + η = βT + ε,where η represents an additional (but uncorre-lated) zero-mean perturbation. This does notonly affect the comparison between individualDFT and experimental results, but also influ-ences the values of the intrinsic systematic devi-ations and residual error bars that are presentedin this work. By considering a test set that issufficiently large, such as in the present study,one may hope that these effects level out. In ad-dition, the elemental crystals belong to the mostintensively studied materials, such that the ex-perimental errors are much smaller than for reg-ular compounds. Without full knowledge of theexperimental errors, however, we can only statethat the SER provides an upper limit for thereal PBE spread σε. A possible solution consistsin comparing DFT values to results from highlyaccurate many-body techniques instead. Suchhigh-precision data are not available for manyof the materials considered here, however, andto calculate them ourselves exceeds the scope ofthe current review.

2. Eliminating outliers

A full statistical analysis of all elemental datacannot be performed straightforwardly. Somesubsets of elements strongly distort the agree-ment between DFT and experiment. Moremeaningful error estimates are obtained whenthe most striking outliers are removed from thedata set. Since the deviating behavior is oftencaused by a bad description of some underly-ing physical mechanism, most of them will be

9

Figure 2. (Color online) Decomposition of the pe-riodic table into smaller subsets of elements, basedon common physical properties of the correspondingground-state crystals: (1) alkali and alkaline earthmetals, (2) nonmagnetic transition metals, (3) mag-netic materials, (4) correlation-dominated materi-als, (5) high-coordination p block compounds, (6)low-coordination p block compounds, (7) molecularcrystals, and (8) noble gases. Subsets 7 and 8 cor-respond to materials where dispersion interactionsare essential

grouped in subsets of similar compounds. In-stead of removing one outlier at a time from thedata set, we choose only to remove entire struc-ture types at once. This way, individual mate-rials that behave well for the wrong reasons, areexcluded as well, avoiding bias towards smallererrors.A decomposition of the test set into eight sub-sets is proposed, based on some common phys-ical properties of the corresponding elementalcrystals (Fig. 2). They are: (1) alkali and al-kaline earth metals, (2) nonmagnetic transitionmetals, (3) magnetic materials, (4) correlation-dominated materials, (5) high-coordination pblock compounds, (6) low-coordination p blockcompounds, (7) molecular crystals, and (8) no-ble gases. Obviously for some boundary ele-ments, the most appropriate subset can be mat-ter for discussion, but the classification in Fig. 2explains most trends for the intrinsic errors ina satisfactory manner (see further).These eight subsets of elemental materials arerepresentative for more complex (multicompo-nent) crystals as well. They provide prototypesystems for particular bond types and physicalphenomena, such as London dispersion (subsets7 and 8), magnetism (subset 3) and electroniccorrelation (subset 4). Observations of DFTperformance for these eight subsets will there-fore carry over to multicomponent compounds.In order to eliminate deviating subsets in anobjective way, the following procedure has been

used. All subsets from Fig. 2 that have halfor more of their elements differing significantlyfrom the dominating trend, have been excluded.A two-sided p-value of 10 % is maintained. Inother words, a data point is considered to de-viate substantially when the (signed) relativeresidual error (expressing the difference betweenthe DFT regression value and the experimen-tal one) belongs to the outer 10 % of a nor-mal distribution. This approach has been re-peated in an iterative way: after the elimina-tion of each subset the significance criterion hasbeen reestablished, until no deviating subsetsremained. For solids belonging to an excludedcategory, PBE is not expected to provide reli-able property predictions.This selection criterion has been visualized inTabs. II to III. For each elemental crystalthe relative residual error |xexp − xreg|/xexp isshown. Large numbers suggest a significant de-viation from the regression line and hence allowto recognize outliers. Because these differencesare displayed in the shape of the periodic ta-ble, they allow for easy identification of deviat-ing subsets1. A color code has been added toimprove intuition, with the darkest shades cor-responding to the largest deviations. The devi-ations with respect to the elastic constants rep-resent the mean absolute errors over C11, C12,C33, C13, and C44.Another way to visualize the assessment pro-cedure is presented in Fig. 3. In order to geta better view on systematic deviations, boththe linear regression line (dashed red) and thefirst quadrant bisector (full black, represent-ing a perfect match between theory and exper-iment) have been added for all accepted ele-ments (see Tab. IV). As an additional qualityindicator, the Pearson product-moment corre-lation coefficient r has been included for eachproperty [114], with r = 1 indicating a perfectpositive correlation. Data points correspond-ing to omitted subsets, on the other hand, havebeen represented by an open symbol.

1 The graphical representations in this work employ theconventional, medium-form periodic table. Hydrogenis kept in group IA (contrary to the vivid discussionin e.g. Ref. 111) and lutetium in group IIIB [112].This increases the intuitive character of the results.Insights should be conveyed at a glance and manyresearchers are most familiar with the standard formatof the periodic table. For the same reason the two-dimensional representation is preferred over some 1Dalternatives [113].

10

Table II. (Color online) Relative residual errors (of the VASP-PBE regression results with respect to thethermally corrected experimental values) for ∆Ecoh [46, 74, 75] (green), V0 [43, 76–83] (red) and B0 [46,62, 84–91] (blue) of the elemental crystals. The darkest shades correspond to the largest errors

11

Table III. (Color online) Relative residual errors (of the VASP-PBE / -PW91 regression results with respectto the thermally corrected experimental values) for B1 [92–98] (purple) and Cij [40, 99, 100] (PW91, cyan)of the elemental crystals. The darkest shades correspond to the largest errors

3. Predicting experiment

Using all crystals that survived this selectionprocedure (filled symbols in Fig. 3), a least-squares linear regression can now be computedfor all properties from Sec. II A. Tab. IV sum-marizes the resulting intrinsic errors in termsof relative systematic deviations (1 − β) andresidual error bars (SER). Systematic errors arementioned for PBE with respect to experiment,so a positive number implies PBE to overes-timate that property. Each percentage is ex-pressed relative to the PBE result, allowing fora straightforward calculation of the regressionvalue (xreg = xth− (1−β)xth). Between brack-ets the significance level of β 6= 1 is mentioned.It represents the two-sided p-value when a null

hypothesis of β = 1 is assumed: if there reallywere no deviation at all, a small p-value wouldindicate that finding an even more extreme re-sult would be highly unlikely. For the residualerror bars a 95 % confidence interval is given.The last column lists the subsets of elementsthat were excluded from the regression analy-sis by the selection procedure described before.For this the naming convention from Fig. 2 hasbeen used.

This statistical treatment makes several im-plicit assumptions. One of the most importantpremises is the use of a relative error over aconstant offset. After all, for strictly positivequantities, such as V0 or Tm, an invariable shiftseems counterintuitive. The impact of such anerror is much larger when the investigated prop-

12

(a) (b)

(c) (d)

(e)

Figure 3. (Color online) Linear regression (dashed red) between the (thermally corrected) experimentaland theoretical results (VASP-PBE / -PW91, see text) for the cohesive energy [46, 75], equilibrium volume[43, 76–83], bulk modulus [46, 62, 84–91], pressure derivative of the bulk modulus [92–98], and elasticconstants [40, 99, 100]. The full, black line stands for xexp = xth. r represents the Pearson product-moment correlation coefficient for all elements included in the regression (filled symbols). The criterion forexcluding certain elements from the fit (open symbols) is discussed in the text

13

Table IV. Systematic deviations 1− β and intrinsicerror bars (SER) for the VASP-PBE/-PW91 (seetext) properties presented in Tabs. II-III, comparedwith experiment. The significance of the system-atic deviation from xexp = xth is indicated betweenbrackets, by means of the two-sided p-value (low p-values for high significance). For the standard errorof the regression a 95 % confidence interval is givenin terms of the upper and lower limit (superscriptand subscript). Subsets containing a lot of outliershave been excluded from the data set by means ofthe procedure mentioned in the text (notation fromFig. 2)

1 − β SER excl.

∆Ecoh [kJ/mol] −0.0 % (0.99) 30 +7−4 4, 8

V0 [A3/atom] +3.6 % (10−10) 1.1 +0.2−0.2 4, 7, 8

B0 [GPa] −4.9 % (10−3) 15 +4−2 7, 8

B1 [–] +4.8 % (0.03) 0.7 +0.2−0.1 6, 7

Cij [GPa] [40] −2.0 % (0.01) 23 +3−2

erty is small. In addition, when using rela-tive systematic deviations, the difference fromβ = 1 indeed matters. For the equilibriumvolume (p = 6 · 10−11) and the bulk modulus(p = 5 · 10−4) the deviation from the bisectorxexp = xth is clearly significant. For other prop-erties this is not always that obvious, but dueto physical connections with V0 and B0, it isrelevant to consider systematic deviations thereas well.

A second remark concerns the nature of theintrinsic residual error bars. The numbers inTab. IV were computed assuming a normal dis-tribution for the random error ε. This results inan absolute residual error bar. Indeed, a Pear-son’s χ2-test does not contradict the applica-bility of a normal distribution to the intrinsicrandom errors. A null hypothesis assuming aGaussian distribution around zero yields a 1-sided p-value of 0.21 for the volume. Withinthe DFT community, however, relative errorbars are often implicity assumed. As a conse-quence, small volumes are expected to be pre-dicted much more accurately, for example. Ac-cording to our analysis, the matter appears tobe not that simple, as is shown in Fig. 4. InFig. 4(a), the relative residual errors are plot-ted against the volume obtained by the least-squares fit. The overall decreasing trend sug-gests that the DFT errors are best described interms of absolute error bars. Fig. 4(b) displaysthe absolute residual errors instead. A roughlinear correlation emerges, that implies that on

(a)

(b)

Figure 4. Relative (a) and absolute (b) differencesof the DFT regression data (DFT*) with respect tothermally corrected experimental values [43, 76–83]for the equilibrium volume

the contrary a relative error bar is more appro-priate. Both conclusions are compatible, by as-suming a relative residual error bar of about 3 %for small to median volumes (< 50 A3/atom)and an absolute residual error bar of approx-imately 1.5 A3/atom for larger volumes (withthe exception of a few outliers2). The numberof data points is not sufficiently large for thisfinding to be really convincing, however. Wetherefore prefer to adopt the overall absolute er-

2 The fitting error is primarily correlated to the bondtype. Although badly performing subsets of mate-rials have already been excluded from the test set,some crystal types are described significantly betterby DFT than others, especially those with strongbonds (covalent p-bonds, half-filled d-shell elements).Their strong bonds also lead to more compact struc-tures, which explains the smaller errors for smallervolumes (Fig. 4(b)). Other subsets of crystals withsimilar volumes perform worse, however, and give riseto the larger relative errors in Fig. 4(a).

14

Table V. Demonstration of the use of intrinsic errors (Tab. IV) to improve DFT-PBE predictions and assesstheir reliability

V0(W) [A3/atom] B0(diamond) [GPa] V0(GaAs) [A3/atom]

PBE (bare) 16.28 434.8 23.73 [37]

systematic deviation 15.69 (−3.6 %) 456.0 (+4.9 %) 22.87 (−3.6 %)

zero-point correction 15.71 (+0.02) 438.4 (−17.6) 23.00 (+0.13) [37]

residual error bar 15.7 ± 1.1 438 ± 15 23.0 ± 1.1

experiment (0 K) 15.8 [37] 443 [46] 22.5 [37]

ror bar of 1.1 A3/atom (Tab. IV) as being validfor all volumes. Admittedly, for most small vol-umes the intrinsic residual error bars are thenoverestimated.

For any given material, the information inTab. IV can now be used to determine a mean-ingful estimate of a certain property. As anexample we consider the bulk modulus of dia-mond. This material is not included in the testset (the ground-state crystal structure of car-bon is graphite), but it can be assigned to sub-set 5, the high-coordination p-block compounds(similar to Si, Ge, and Sn). Subset 5 does notbelong to one of the excluded subsets for B0,and the intrinsic error for the bulk modulusprediction of diamond should therefore be rep-resentative. A bare VASP-PBE computationyields 434.8 GPa. When taking into accountthat PBE bulk moduli are systematically toosmall by 4.9 % (Tab. IV), this value increases to456.0 GPa. By means of Eqs. (12), (10), and (6)zero-point corrections (−17.6 GPa) are addedback in, yielding 438.4 GPa. Using the ap-propriate intrinsic residual error bar of 15 GPa(Tab. IV), the final result becomes 438±15 GPa.This is the most accurate DFT-PBE predictionof the experimental bulk modulus at 0 K, in-cluding an error bar on the computed value.In comparison, the experimental value extrap-olated to absolute zero (Eqs. (11) and (8))amounts to 443 GPa [46]. This number remainsneatly within the error bar and is indeed closerto the regression-corrected bulk modulus thanto the bare DFT value. A similar procedure canbe used for all properties in Tab. IV. Tab. V of-fers a few more examples.

Tab. IV is based on elemental solids only. Onehas to verify that the results from this statisti-cal analysis are transferable to multicomponentmaterials. A good test case in that respect is thecollection of thirty-one binary compounds forwhich Haas et al. [37, 38] calculated lattice pa-rameters by means of PBE. When we take theirDFT results, the experimental volume falls out-

side the confidence interval for seven crystals.In all of these cases, the PBE volume is toolarge. By taking into account the systematicoverestimation of 3.6 %, however, only in twocases the experimental value exceeds the pre-dicted range. Both of these compounds are ion-ically bound (NaF and NaCl), a bond type thathas not been considered in any of the proposedsubsets (Fig. 2). Although the multicomponenttest set may therefore not contain all materialstypes and although it only relates to the atomicvolumes, these results already strongly supportthe transferability of our error estimates.For one example from Haas et al., GaAs, the in-trinsic error contributions are listed in Tab. V.It illustrates that the systematic deviation re-ally matters if it is the goal to get as close aspossible to the experimental value. This alsoappears from the mean absolute difference be-tween experiment and the 31 theoretical pre-dictions by Haas et al.. When one does notapply the relative deviation from Tab. IV, thisnumber amounts to 0.72σV (0.78 A3/atom),while for the regression values it is only 0.37σV(0.40 A3/atom).

C. Agreement with experiment

1. Errors per materials type

Because Tabs. II-III are shaped like the periodictable, the color code immediately allows to sin-gle out the areas where PBE breaks down. Itleads to a number of subsets (Fig. 2) which canbe eliminated from the test set. They are listedin Tab. IV. For these elements, PBE performssignificantly worse than usual, mostly becausesome key physical phenomenon is not described(well) by the functional. In this subsection theselocalized error zones are discussed in more de-tail, as well as the mechanisms on which theyare based.A first, well-known example of the failure

15

of PBE is the class of dispersion-governedcompounds. Although some more advancedDFT approaches address this issue specifically[115, 116], regular GGA functionals do not de-scribe London forces. This translates into adecreased cohesion, and hence an inflated vol-ume and underestimated bulk modulus (see alsoSec. III C 2).

Although London forces have been demon-strated to play a role in other structures as well[117–119], the most important crystals that suf-fer from this shortcoming, belong to the non-metals (subsets 7 and 8). They include the no-ble gases, the dimeric crystals, graphite, andsulfur. In these materials the London dispersioninteraction governs the bonding between atoms,diatomic molecules, graphene sheets, and 8-membered rings, respectively. Nevertheless it isessential to realize that both the element typeand the crystal structure contribute to the im-portance of dispersion. It is perfectly plausiblethat a certain element behaves badly in struc-ture A, while there are no problems when it as-sumes structure B. This can be illustrated nicelyby means of carbon. The dispersion forces be-tween the graphene layers in graphite give riseto a large discrepancy between DFT and exper-iment. Diamond on the other hand follows thesame behavior as neighboring (semi)metallic el-ements or even outperforms them (Tab. VI).

For the molecular crystals (subset 7) the PBEcohesive energy is larger than the experimentalvalue [71], contrary to the expectation. This isdue to the overestimation of the intramolecularbond strength (see e.g. Lany [36] and Tab. II ofPaier et al. [20]), which covers up any influenceof the lack of dispersion. Elastic properties onthe other hand are in most cases not affectedby intramolecular effects and show a similar be-havior as for the remaining nonmetals.

The magnetic materials (subset 3) stand outas well, predominantly with respect to ∆Ecoh.Although the use of the generalized gradientapproximation and a correct atomic reference

Table VI. Residual errors (of the VASP-PBE regres-sion results with respect to the zero-kelvin extrap-olated experimental values [43, 46, 74, 75, 84]) fortwo allotropes of carbon

∆Ecoh [kJ/mol] V0 [A3/at] B0 [GPa]

graphite 39 (5 %) 3.2 (39 %) 54 (97 %)

diamond 13 (2 %) 0.1 (2 %) 5 (1 %)

state have already reduced the gap between the-ory and experiment substantially [48], the re-maining difference cannot be neglected. Cur-rent GGA functionals are not able to describemagnetic compounds very well. Manganeseillustrates this nicely. Its intricate magneticstate [120] has been approximated by assum-ing only collinear magnetism, but this does notexplain the observed differences. The cohesiveenergy, for example, would be higher in its cor-rect ground state, leading to an even more pro-nounced deviation from experiment. An expla-nation is found with Singh and Ashkenazi [121],who noticed that GGAs overestimate the mag-netic energy. This is caused by the increasednumber of degrees of freedom in spin-polarizedsystems (two spins), while the number of phys-ical relations the GGA must fulfill stays thesame.The discrepancies between theory and exper-iment are not caused by the DFT functionalalone, however. For some magnetic elementsthe applied thermal extrapolations are no longervalid, because of phase transition effects in thevicinity of the Curie or Neel temperature. Ex-perimental chromium is a good example, dis-playing large magnetic distortions of the ther-mal expansion coefficient near 311 K [122]. Arelation as simple as that of Eq. (8) cannot cap-ture these complex underlying phenomena.The transition metals with (nearly) full d shellssometimes deviate from experiment as well(subset 4). The effects are smaller than for theprevious two classes of materials, but they areunmistakably present, especially in terms of thecohesive energy and the elastic constants. Onecan attribute this phenomenon to electroniccorrelation. For Zn, Cd, and Hg a full-fledgedmany-body treatment has indeed convincinglyshown the influence of d electron correlationon ∆Ecoh and the potential energy landscape[13, 123, 124]. Data in Tabs. II-III even im-ply that similar (but smaller) effects show up inother elements at the end of the d block, suchas in Pd, Ag, Pt, and Au. In noble metals, dis-persion phenomena play an important role too[118], however, and it is not immediately clearhow much of the remaining discrepancy can beattributed to electronic correlation. Since theinfluence of correlation in these elements ap-pears limited, only Cd an Hg have been assignedto subset 4.It seems that at the end of the d block, the highnumber of localized d electron pairs in combi-nation with a small interatomic distance and aclose-packed environment enhances correlation

16

effects. Moreover, Philipsen and Baerends [48]suggest that at the very beginning and the veryend of the 3d transition metals the GGA ex-change energy drops, causing the electron corre-lation to gain importance. Any fortuitous can-cellation of errors between exchange and corre-lation has therefore disappeared for the transi-tion metals at the border of the d block.

These correlation effects appear to be of amainly anisotropic nature, since all elastic con-stants Cij are affected, but the deviations of V0and B0 are less pronounced. This is also sug-gested by Wedig et al. [124] for Zn and Cd,where a different interlayer and intralayer be-havior is observed.

Relativistic effects are expected to stronglyinfluence heavy elements. VASP thereforemakes use of the scalar-relativistic Kohn-Shamequations by default [125]. The major remain-ing contribution is due to spin-orbit coupling.However, it is shown by Philipsen and Baerends[126] that this does not change physical prop-erties substantially. Only for gold and bismutha distinct change is reported, but without clos-ing the gap between theory and experiment en-tirely. The remaining difference for Au is pri-marily due to correlation and dispersion effects,as was already suggested above. For the 6p ele-ments on the other hand spin-orbit coupling re-ally plays an important role. Some key proper-ties for the 5d and 6p compounds have been cal-culated, both with and without spin-orbit cou-pling (Tab. VII). It is immediately clear that,starting from the end of the 5d block, a spin-orbit treatment becomes indispensable. Hence,for all 5d and 6p elements this contribution hasbeen included, except for Cij [40].

2. Errors per property

The previous section shows that PBE is not‘complete’: some features just cannot be de-scribed by a simple GGA functional. One canexclude the affected materials (outliers) before-hand, however, and limit the analysis to thosecases where PBE should perform well. The in-trinsic errors from Tab. IV are applicable tothese crystals. Tab. IV then shows that thePBE error estimates largely depend on whatproperty is considered. The behavior of theresidual error bar and the systematic deviationfrom experiment can be traced back to boththe functional and the numerical determinationof that particular property. Nevertheless, it

Table VII. Relative residual errors (of the VASP-PBE regression results with respect to the zero-kelvin extrapolated experimental values [43, 46, 83])for Ag and the 5d and 6p materials, both with (SO)and without spin-orbit coupling (n-SO)

∆Ecoh V0 B0

n-SO SO n-SO SO n-SO SO

Ag 16 % 16 % 3 % 3 % 11 % 11 %

Lu 7 % 8 % 3 % 3 % 18 % 18 %

Hf 1 % 3 % 3 % 3 % 2 % 2 %

Ta 1 % 2 % 2 % 2 % 1 % 0 %

W 0 % 1 % 1 % 1 % 3 % 5 %

Re 3 % 2 % 2 % 1 % 3 % 1 %

Os 1 % 3 % 0 % 1 % 1 % 4 %

Ir 4 % 0 % 0 % 0 % 0 % 2 %

Pt 7 % 9 % 1 % 1 % 9 % 11 %

Au 22 % 19 % 4 % 3 % 21 % 19 %

Hg 75 % 69 % 29 % 22 %

Tl 6 % 20 % 9 % 7 % 27 % 25 %

Pb 44 % 4 % 4 % 4 % 10 % 18 %

Bi 15 % 5 % 2 % 5 % 17 % 26 %

Po 59 % 8 % 2 % 3 % 73 % 30 %

Rn 83 % 81 %

is important to note that, although the over-all error estimate can be linked to theoreticalaspects, the correspondence to experiment fora single compound depends on the experimen-tal accuracy as well. This is especially truefor higher-order properties (such as the elas-tic constants and their derivatives), which aregenerally measured at a lower precision thanthose from (quasi)direct measurements (the lat-tice constants or the cohesive energy, for exam-ple). This is illustrated by the sometimes largespread on the data in Knittle’s overview of B1

values [92].

From a computational viewpoint, however, theequilibrium volumes offer the best resultsamong all considered quantities. After elimi-nating the outliers (listed in Tab. IV and rep-resented in Fig. 3(b) by open symbols), an al-most perfect correlation is obtained. Even so,the regression line does not coincide with thefirst quadrant bisector. Tab. IV shows that thecell volumes are consistently too large by ap-proximately 4 %. This deviation is a well-knownproperty of any GGA [127], including PBE.It originates in a systematic underestimationof the bond strength (underbinding), resultingin slightly larger volumes. More particularly,

17

GGAs favor inhomogeneous systems, with large(reduced) density gradients. Small unit cells,which have a more evenly distributed electrondensity, are therefore energetically less prefer-able. This phenomenon especially affects openstructures, where the high-gradient tails of thevalence electron orbitals become non-negligible[128].Immediately linked to this observation is theunderestimation of the bulk modulus. Moreweakly bound structures will be more easilycompressible, leading to smaller B0 values. Justlike the too large predicted volumes, it is com-mon behavior for GGA functionals [127]. Onthe other hand, PBE bulk moduli are predictedwith a larger uncertainty than the volumes. Theintrinsic residual error remains in most cases be-low 10 to 15 %, however (Tab. II). The magni-tude of this difference is mostly due to the sen-sitivity of the E(V ) curvature.Since B0 and the other elastic constants areclosely related, the intrinsic errors with respectto the Cij parameters are of a comparable scale.The bulk moduli are larger on average, whichleads to slightly smaller relative errors (Tabs. IIand III). However, a good correlation is foundin both cases, with a similar value of r for theelastic constants (Fig. 3(e)) and the bulk moduli(Fig. 3(c)).For the cohesive energy PBE yields verygood results as well. The intrinsic error bar of30 kJ/mol is of the same order of magnitude asthe rms error found by Lany [36] for PBE heatsof formation of semiconductors and insulators(0.24 eV/atom). It is therefore representativefor PBE energy differences between chemicallydifferent compounds. As mentioned before, forsimilar systems the intrinsic residual error baris much smaller [42]. This also explains the suc-cess of evolutionary algorithms. They are basedon energy differences of the order of a few meVper atom [129–132], but some results have al-ready been confirmed experimentally neverthe-less [9].Contrary to V0 and B0 there is now no system-atic under- or overestimation compared to ex-periment. The typical underbinding of GGAdoes not show as conclusively in ∆Ecoh. This isdue to the magnetic materials and the molecularcrystals. As mentioned before, GGA function-als bias solutions towards magnetism for the for-mer and overestimate the intramolecular contri-bution in the latter. In both cases this causesthe cohesive energy to oppose the dominatingtrend. Without the crystals from subsets 3 and7 in the test set, the cohesive energies would

have been underestimated by 2 % instead (p-value of 0.008). This behavior is in accordancewith the expected underbinding of GGA.

Since the bulk modulus derivative B1 is ahigher-order parameter than B0, the errors areexpected to be one order worse as well. Al-though this is certainly the case, eliminatingthe outliers substantially improves the results(Fig. 3(d)). However, even when they are re-moved, the resulting correlation coefficient (r =0.849) remains significantly lower than for anyother property already discussed.

B1 appears to be overestimated with respect toexperiment. This systematic deviation is sig-nificant, although the p-value may not show itconclusively (Tab. IV). It is again caused byGGA underbinding. As mentioned before, largevolumes are favored due to their substantialdensity gradients. GGA hence lowers the en-ergies of bigger cells most and straightens outthe equation of state. This causes the E(V )line to alter its decreasing curvature even morerapidly, increasing the rate of change of the bulkmodulus with pressure (and volume), B1. Italso explains the deviating behavior of crystalswith a low coordination, such as the molecularcrystals. In these compounds the tails of theelectron wave functions dominate the intersti-tial space, leading to considerable density gra-dients. The increase of the sensitive parameterB1 is then enhanced even further.

IV. NUMERICAL ERRORS

The previous section describes the intrinsicPBE errors for five different properties, basedon a statistical treatment. They are used in aprotocol which allows experimentalists and the-oreticians to correct the bare DFT-PBE valuesfor the observed systematic deviation from ex-periment and which quantifies the uncertaintyon the obtained predictions (Tab. V). A pre-requisite for such a protocol is that differentDFT implementations provide the same pre-dictions: using different algorithms to solvethe same (Kohn-Sham) equations should ideallylead to identical solutions. In practice, differ-ent amounts of noise are inevitably introducedin the predictions, even when numerical con-vergence has been achieved for each individualcode. This scatter is due to several aspects ofthe solution algorithm. It can be due its nature(e.g. the kind of basis set or the frozen-core ap-proximation), its specific ingredients (e.g. the

18

chosen pseudopotential) or its use of particularroutines for standard tasks (e.g. Fourier trans-form routines). In order to guarantee the re-producibility of the intrinsic errors in Tab. IV,one needs to examine to what extent these is-sues affect the DFT computations. Once again,a reliable benchmark can be established usingthe ground-state elemental crystals.

The current section describes a procedure to ex-press the difference between predictions fromindependent solid-state DFT approaches in aquantitative way, yielding a numerical error es-timate. It will be used to examine differencesbetween the PAW and APW+lo method, rep-resenting both methods by suitable mainstreamcodes (see further). However, the difference be-tween codes can be attributed to other aspectsas well, as was mentioned earlier. The influenceof standard task routines is most likely small,but we will also use two PAW codes with dif-ferent PAW atomic potentials, which can havequite drastic effects on the DFT results. Allcodes are therefore considered here with theirrecommended potentials. These can be thoughtof as representative for the quality of the investi-gated code. Although it is not our primary goalhere, the procedure that will be described canalso be used to select better performing PAWatomic potentials. The same holds for pseu-dopotentials in the case of plane-wave codes.

A. Test set preparation

The present implementation assessment startsfrom one reference code, the all-electron pro-gram WIEN2k [133] (version 11.1). It uses theAPW+lo basis set [134, 135], which is consid-ered to be a standard for the numerical accu-racy of solid-state DFT. WIEN2k predictionscan therefore be considered to yield the exactresults for a given functional, as long as numer-ical accuracy is achieved [37] (large basis set anddense k-mesh, see Supplementary Material formore details [71]). Two codes are compared tothis reference code: VASP [8, 67] (version 5.2.2)and GPAW [136–138] (version 0.8.0), both us-ing the PAW method [65]. GPAW calculatesall wave functions, densities and potentials asgrid-based quantities, while VASP uses a plane-wave basis set. All calculations with these pro-grams are performed by means of the poten-tials recommended by the respective developers:the 2010 recommended PAW potentials [71] forVASP and the 0.6 atomic set-ups for GPAW.

Detailed computational parameters are summa-rized in the Supplementary Material [71].

For reasons of uniformity and comparability thesame PBE functional has been selected for allthree codes. It is used in a protocol that seeks toevaluate a particular DFT approach in an eas-ily reproducible manner. The VASP-optimizedground-state crystal (Sec. III A) serves as astarting point for each computation and froma 7-point equation of state (0.94V0 to 1.06V0)the properties of interest (E0, V0, B0, and B1)are extracted. All geometries are kept frozen(the cell shape and relative atomic positions arekept fixed at their initial values), instead of al-lowing for relaxation changes. This not onlylowers the computational load, it also restrictsthe code evaluation to the implementation ofDFT-PBE itself. Indeed, the task of optimiz-ing the cell shape or internal positions belongsto another computational layer, on top of thetask of solving the DFT equations for a givenrigid geometry. This section aims to examinehow different implementations compare with re-spect to the DFT-PBE procedure only. It doesnot intend to study how close every individualapproach comes to experiment.

In the same spirit some other modifications ofthe hitherto employed test set have been made.All calculations have been limited to the scalar-relativistic part (using the Koelling-Harmon ap-proach [125]). By neglecting the spin-orbit con-tribution, an additional secondary algorithmimplementation is avoided. The computationalprocedure also becomes more uniform this way,since all elements are now treated on equalterms. Because no spin-orbit coupling is addedto the system’s Hamiltonian, it suffices to usenon-spin-orbit geometries as a starting point forthe 7-point equation of state.

A simplified unit cell has been selected for Mnand S as an additional means of lowering thecomputational effort. Manganese is treated inan antiferromagnetic fcc phase (space group225, cF4), while for sulfur the β Po phase isimposed (space group 166 or hR3). These ge-ometries are physically relevant, as they can befound in the Mn and S phase diagram respec-tively [139, 140].

All other elements have been kept at the struc-ture previously optimized by VASP (Sec. III A),in order to conserve the large diversity of theinput set. The CIF files for all crystals in thiscode benchmark set are available in the Supple-mentary Material [71].

19

B. Agreement between implementations

The procedure mentioned above results in alarge collection of numbers for each code (71elements × 4 properties). It is not conve-nient to compare them directly, however. Be-cause of the different units involved, a coher-ent approach would require the use of rela-tive deviations. Tabs. II-III on the other handshow that each property corresponds to a dif-ferent magnitude of relative error. This scale ismainly determined by the computational proce-dure and therefore does not alter substantiallywhen shifting from a code-experiment compari-son to an intercode assessment. A single numer-ical error value, expressing the difference be-tween two particular DFT methods by meansof one number, can be obtained by applyinga weighed average. As all properties of inter-est depend on the equation of state, it is moststraightforward to compare the E(V ) curvesproduced by different approaches directly. Thedispersion-governed compounds illustrate thisstrategy well. Since their E(V ) curves are veryshallow, small deviations in the bulk moduluswill inflate the relative error considerably. How-ever, the equations of state as such can be verysimilar, the two curves at no point differing bymore than a few meV per atom (Fig. 5). Forthat reason a numerical error estimate ∆ is de-fined as follows:

∆ =

⟨√∫∆E2(V ) dV

∆V

⟩(15)

In other words, the rms energy difference be-tween the E(V ) curves of these particular pro-

Figure 5. (Color online) The EOS parameters candiffer significantly, while the E(V ) curves them-selves are very similar. In that case the area be-tween the two functions is a better indicator of theoverall deviation

grams is averaged over all elemental crystals. ∆hence provides an intuitive measure of the en-ergy distance between equations of state.

Because different codes sometimes employ dif-ferent reference energies E0, depending on theconcept, all equations of state are set to zero attheir equilibrium volume. An alternative solu-tion would entail the calculation of cohesive en-ergies, in order to provide a common referencefor the equilibrium energy. However, not allprograms allow for an easy manipulation of theelectronic configuration of atoms. Moreover,the computational load would increase consid-erably.

The computation of ∆ can be automated quiteeasily. The fitted Birch-Murnaghan equationallows Eq. (15) to be written in an analyticalform. Only V0, B0, and B1 are then needed forboth codes under investigation. The resultingexpressions have been added in the appendixfor convenience. The WIEN2k data necessaryfor a code comparison have been provided inthe Supplementary Material [71].

The interval of integration is linked to the refer-ence data. In view of how the E(V ) parametersare determined, the intercode difference is to beintegrated between V0,WIEN2k±6 %. ∆V hencecorresponds to 0.12V0,WIEN2k. By definition∆(APW+lo)(WIEN2k) becomes zero.

The rms energy differences between the equa-tions of state predicted by APW+lo(WIEN2k)

and PAW(VASP), or APW+lo(WIEN2k) andPAW(GPAW), are represented in Tab. IX. Theyshow that most critical elements are char-acterized by approximately half-filled d lev-els. Such numerical errors can amount toup to 8.3 meV/atom for PAW(VASP) (Tc) and20.9 meV/atom for PAW(GPAW) (Ru). Thisagrees with physical intuition, because thesecrystals are among the least compressible.Their equations of state are very steep, and rel-atively small modifications of the parameterscan strongly change the energy. The least sen-sitive elements are for the same reason locatednear the alkali metals and the noble gases (0 -0.7 meV/atom numerical error) (see Tab. VIII).Only in comparison to experiment the lattergroup of materials stands out, but this is be-cause PBE grossly overestimates the rare gasvolumes.

When averaging the numbers in Tab. IX overall elements, the numerical error of each DFTapproach can be determined for the givenset of recommended PAW potentials. ∆is 1.9 meV/atom for PAW(VASP), while for

20

Table VIII. Comparison between codes for two ex-treme situations: large (Tc an Ru) and small (Ar)numerical errors ∆i. V0 is given in A3/atom, B0 inGPa, and ∆i in meV/atom. B1 is dimensionless

V0 B0 B1 ∆i

Tc APW+lo(WIEN2k) 14.47 301.4 4.56 0

PAW(VASP) 14.60 298.5 4.55 8.3

Ru APW+lo(WIEN2k) 13.81 315.4 4.96 0

PAW(GPAW) 14.09 310.9 4.87 20.9

Ar APW+lo(WIEN2k) 52.21 0.7 7.84 0

PAW(VASP) 52.65 0.8 7.35 0.1

PAW(GPAW) 52.66 0.8 3.27 0.1

PAW(GPAW) it is 3.3 meV/atom. This agree-ment between implementations is an order ofmagnitude better than the difference with ex-perimental results. To show this, a similar en-ergy difference between DFT-PBE and experi-ment is computed. It uses experimental valuesas the reference situation, while the method un-der test is the full-fledged version of PAW(VASP).This means that the E(V ) parameters havebeen taken from Tabs. II and III. The deviationsper element are presented in Tab. IX, leadingup to a ∆-factor of 23.5 meV/atom. This differ-ence in magnitude can also be observed with theE(V ) characteristics themselves. Fig. 6 showsthe distribution of volume errors between twocodes and with respect to experiment. Again,the spread is much larger in the latter case.

∆(PAW)(VASP) does not change noticeablywhen the number of elements is reduced to thatof PAW(GPAW). This shows that GPAW andVASP, while both using the same PAW method,do not produce entirely identical results. Thisvariation most likely originates in the differentquality of the atomic potentials and the differ-ent type of basis functions used. However, incomparison to experiment, the differences arenegligible. The intrinsic residual error bars andregression slopes provided in Tab. IV can there-fore be applied to DFT-PBE results, irrespec-tive of which approach was used to calculatethem.

This comparison of three DFT implementationscan easily be extended. Ideally every solid-stateDFT approach should be tested in the sameway, and have its ∆-value computed. As suchtests are preferably performed by specialists inthe individual codes, all input CIF files havebeen made available in the Supplementary Ma-terial [71], as well as some ready-made post-

Figure 6. (Color online) Intrinsic (PBE re-gression versus experiment) and numerical errors(PAW(VASP) versus APW+lo(WIEN2k)) for the equi-librium volume of the ground-state elemental crys-tals, using the subsets of elements that have beenshown to perform well for PBE (see Tab. IV). A nor-mal distribution has been fitted to both data sets(dotted line)

processing scripts. In addition, the ASE devel-opers have implemented a framework for per-forming the necessary calculations [136, 141].On the CMM website [142] an updated overviewwill be maintained of all ∆-factors reported tous. Such information not only provides insightinto the reproducibility of the intrinsic errors ofTab. IV, but can also guide users to select amethod for a specific task, at least as far as ac-curacy of energy-versus-volume relations is con-cerned.

V. CONCLUSIONS

Using the ground-state elemental crystals as atest set, DFT-PBE computational errors havebeen reviewed. Errors intrinsic to the functionalwere quantified for five materials properties, de-scribing energetic (∆Ecoh) and elastic (V0, B0,B1, Cij) quantities. They explain the deviationof DFT predictions from experiment. Numer-ical errors, due to the implementation of theDFT scheme into a computer code, were stud-ied for the PAW method (VASP and GPAW),and were expressed with respect to the referenceAPW+lo method (WIEN2k). Both types of er-rors have been discussed for PBE, one of themost widely applied functionals in solid-stateDFT. The results are expected to be represen-tative of GGA in general.Each of the five properties has been assessed

21

Table IX. (Color online) Rms energy differences ∆i between the equations of state predicted byAPW+lo(WIEN2k) and PAW(VASP) (green), APW+lo(WIEN2k) and PAW(GPAW) (red), and experiment andPAW(VASP) (blue) for the ground-state elemental crystals. All values are expressed in meV per atom. Thedarkest shades correspond to the largest errors. The average numerical error ∆ is shown for each code atthe header of the table

22

with respect to the ground-state elemental crys-tals. The correspondence to experiment hasbeen analyzed statistically, leading to a decom-position of the intrinsic error into systematicdeviations and residual error bars. These in-trinsic errors have been shown to agree withsome generally known GGA traits. The typi-cal underbinding of GGA has been reproducedand quantified, for example. Tab. X presentsa summary of our results, as well as a similaranalysis for other properties that are availablefrom data sets in the literature. Contrary toTab. IV, however, Tab. X presents systematicdeviations in terms of the experimental value((xth − xexp)/xexp = 1/β − 1). This expressesmore intuitively to what extent DFT varies fromexperiment: a 1/β − 1 of +1 % for examplemeans that PBE overestimates the experimen-tal result by 1 %.Based on the quantification of intrinsic errors, acomputational recipe has been presented whichallows to correct bare DFT-PBE results forthe systematic deviation from experiment, andwhich attaches meaningful error estimates tothe obtained predictions. (Tab. V). An exam-ination of 31 binary compounds not includedin the benchmark set [37, 38] indicate that ouranalysis carries over to multicomponent crys-tals. Errors can hence be estimated straight-forwardly for PBE predictions already availablefrom literature.The overall agreement between VASP-PBEand experiment is quite good, but some sub-sets of elements perform better than others.DFT predictions for magnetic materials andcorrelation-dominated compounds deviate sig-nificantly from experimental values, for exam-ple, especially with respect to the cohesive en-ergy. Long-range interaction is another is-sue. Although some solutions exist to incorpo-rate London dispersion into DFT, such as theDFT-D [115] or vdW-DF2 method [116], regu-lar GGAs do not describe dispersion-governedcrystal types well. Bulk moduli are found to beconsistently underestimated, while predictionsfor both the volume and the pressure deriva-tive of the bulk modulus are systematically toolarge. Results for heavy elements are acceptableas long as spin-orbit coupling is added, startingfrom the 6p block. Based on these observations,some general guidelines have been summarizedin Tab. X as to what categories of materials willnot be described well. Some classes were notor only marginally represented in the elementalbenchmark set, such as ionic or strongly cor-related compounds. For these, a similar study

using an extended benchmark set, by includ-ing some binary ionic compounds and transitionmetal oxides, would be useful.

All conclusions with respect to the intrin-sic PBE errors can only be universally ap-plicable when it does not matter how theDFT formalism is implemented. Such nu-merical errors should be much smaller thanintrinsic ones. By means of a quality fac-tor ∆, which conveys exactly this informa-tion, APW+lo(WIEN2k) has been comparedto PAW(VASP) (∆ = 1.9 meV/atom) andPAW(GPAW) (∆ = 3.3 meV/atom), both fortheir recommended sets of atomic potentials.The rms energy distance between equations ofstate from different methods indeed appearsto be an order of magnitude smaller than thegap between theory and experiment (see alsoTab. X). The intrinsic systematic deviationsand residual error bars presented in Tab. X canhence be applied to PBE predictions regardlessof the computational approach. This is usefulwhen discussing the implications of DFT resultsin an experimental context.

This accuracy review is to be considered as astarting point only. The presented statisticalprocedure is applicable to other functionals ormethods as well. It would be useful to de-termine the intrinsic systematic deviations andresidual error bars for e.g. LDA or hybrid func-tionals. A comparison to results from high-level many-body techniques would even allowto eliminate the influence of experimental er-rors. Another extension would be to take intoaccount experimental error bars in the statisti-cal analysis. For the relevant properties of theelemental materials, this requires an extensiveliterature search for the most accurately knownvalues and their error bars. These error barsare not commonly available in tabulations in theliterature and are therefore beyond the scope ofthis work. With respect to the assessment ofnumerical errors, we invite both code develop-ers and users to determine the quality factor ∆for their code as well. It not only guaranteesthe transferability of the intrinsic errors to allcodes, but also provides a criterion to evaluatethe accuracy of a particular DFT approach. Acomprehensive list of ∆-factors [142] can thenserve as a guideline through the maze of avail-able DFT methods.

23

Table X. Systematic deviations 1/β − 1 and residual error bars for DFT-GGA predictions compared withexperiment (intrinsic errors) and between codes (numerical errors). ∆ represents the average rms energydifference between the equations of states of two codes. All data were (re)analyzed in the present study,except for the error bar for ∆Eevol, which is based on a proof of principle (see Sec. III C 2). Subsets ofmaterials to which the error estimates do not apply are mentioned explicitly

Intrinsic errors (this work, VASP-PBE)

systematic deviation residual error bar not applicable to:

∆Ecoh [kJ/mol] −0.0 % 30 mostly correlation; pure dispersion

V0 [A3/atom] +3.8 % 1.1 mostly correlation; dispersion

B0 [GPa] −4.7 % 15 dispersion

B1 [–] +5.0 % 0.7 low coordination number

Intrinsic errors (other works, VASP-PW91/-PBE/-PBE+U)

Cij [GPa] [40] −2.0 % 23

∆Eform [kJ/mol] [36] −13.1 % 15

∆Ereact [kJ/mol] [42] −4.8 % 3.2 chemically dissimilar

∆Eevol [kJ/mol] [9, 129–132] . 1 chemically / structurally dissimilar

Numerical errors (this work, PBE)

∆(PAW)(VASP) [kJ/mol] 0.19 —

∆(PAW)(GPAW) [kJ/mol] 0.32 —...

...

ACKNOWLEDGMENTS

This work is supported by the Fund for Scien-tific Research – Flanders (FWO) and by theResearch Board of Ghent University. StefaanCottenier acknowledges financial support fromOCAS NV by an OCAS-endowed chair at GhentUniversity. Calculations were carried out us-ing the Stevin Supercomputer Infrastructureat Ghent University, funded by Ghent Univer-sity, the Hercules Foundation, and the FlemishGovernment (EWI Department). The authorsthank Martijn Marsman for his useful insightson the spin-orbit coupled Pb atom, Fabien Tranfor a helpful discussion on zero-point correc-tions, and Manuel Richter for some interest-ing comments with respect to scalar-relativistictechniques. They acknowledge the implemen-tation by Marcin Du lak of the code compari-son database into ASE [136], and thank him forvaluable remarks with respect to the ∆-factorcomputation script. This work has benefited alot from remarks and input given by many col-leagues at conference presentations, as well asfrom the extensive and constructive commentsprovided by three anonymous referees.

APPENDIX: CALCULATING THE∆-FACTOR

For a particular compound, the energy differ-ence between the equations of state of WIEN2k(w) and a code under investigation (c) canbe evaluated analytically, using the Birch-Murnaghan relation of Eq. (2). Some mathe-matical manipulations yield∫ Vf

Vi

(Ec(V )− Ew(V ))2

dV = F (Vf )− F (Vi)

(A.16)where the primitive function F (V ) can be writ-ten as a power series in V −1/3:

F (V ) =

4∑n=−2

xnV−(2n+1)/3 (A.17)

The coefficients xn are given by

xn = − 3

2n+ 1

∑i+j=n+2

(aci − awi )(acj − awj )

(A.18)with i, j ∈ {0, 1, 2, 3}. Then x−1 for examplebecomes

x−1 = 6(ac1 − aw1 )(ac0 − aw0 ) (A.19)

The constants ai are the coefficients of theBirch-Murnaghan equation in its polynomial

24

form:

a3 =9V 3

0 B0

16(B1 − 4) (A.20)

a2 =9V

7/30 B0

16(14− 3B1) (A.21)

a1 =9V

5/30 B0

16(3B1 − 16) (A.22)

a0 =9V0B0

16(6−B1) (A.23)

When evaluating Eq. (A.18), aci stands for thecoefficient of the code under test, while awimeans it corresponds to the WIEN2k equationof state.

[1] P. Hohenberg and W. Kohn, Phys. Rev. 136,B864 (1964)

[2] W. Kohn and L. Sham, Phys. Rev. 140,A1133 (1965)

[3] M. Segall, P. Lindan, M. Probert, C. Pickard,P. Hasnip, S. Clark, and M. Payne, J. Phys.:Condens. Matter 14, 2717 (2002)

[4] W. Liu, E. Karpov, S. Zhang, and H. Park,Comput. Methods Appl. Mech. Eng. 193,1529 (2004)

[5] R. Martin, Electronic Structure: Basic Theoryand Practical Methods (Cambridge UniversityPress, 2004)

[6] K. Capelle, Braz. J. Phys. 36, 1318 (2006)[7] J. Hafner, C. Wolverton, and G. Ceder (Guest

Editors), MRS Bull. 31 (2006)[8] J. Hafner, J. Comput. Chem. 29, 2044 (2008)[9] Y. Ma, M. Eremets, A. Oganov, Y. Xie, I. Tro-

jan, S. Medvedev, A. Lyakhov, M. Valle, andV. Prakapenka, Nature 458, 182 (2009)

[10] G. Hautier, A. Jain, S. Ong, B. Kang,C. Moore, R. Doe, and G. Ceder, Chem.Mater. 23, 3495 (2011)

[11] A. Jain, G. Hautier, C. Moore, S. Ong, C. Fis-cher, T. Mueller, K. Persson, and G. Ceder,Comp. Mater. Sci. 50, 2295 (2011)

[12] L. Schimka, J. Harl, A. Stroppa, A. Gruneis,M. Marsman, F. Mittendorfer, and G. Kresse,Nature Mater. 9, 741 (2010)

[13] N. Gaston, D. Andrae, B. Paulus, U. Wedig,and M. Jansen, Phys. Chem. Chem. Phys. 12,681 (2010)

[14] A. Bil, B. Kolb, R. Atkinson, D. Pettifor,T. Thonhauser, and A. Kolmogorov, Phys.Rev. B 83, 224103 (2011)

[15] D. Beratan and J. Hopfield, J. Chem. Phys.81, 5753 (1984)

[16] I. Bersuker, J. Comput. Chem. 18, 260 (1997)[17] P. Pyykko, Chem. Rev. 112, 371 (2012)

[18] E. Engel, A. Facco Bonetti, S. Keller, I. An-drejkovics, and R. Dreizler, Phys. Rev. A 58,964 (1998)

[19] E. Engel and U. Lechner, Chem. Phys. 311,209 (2005)

[20] J. Paier, R. Hirschl, M. Marsman, andG. Kresse, J. Chem. Phys. 122, 234102 (2005)

[21] A. Kiejna, G. Kresse, J. Rogal, A. De Sarkar,K. Reuter, and M. Scheffler, Phys. Rev. B 73,035404 (2006)

[22] J. Perdew, K. Burke, and M. Ernzerhof, Phys.Rev. Lett. 77, 3865 (1996)

[23] S. Kotochigova, Z. Levine, E. Shirley,M. Stiles, and C. Clark, Phys. Rev. A 55, 191(1997)

[24] S. Kotochigova, Z. Levine, E. Shirley,M. Stiles, and C. Clark, Phys. Rev. A 56, 5191(1997)

[25] Y. Zhao and D. Truhlar, Acc. Chem. Res. 41,157 (2008)

[26] L. Goerigk and S. Grimme, Phys. Chem.Chem. Phys. 13, 6670 (2011)

[27] S. Kurth, J. Perdew, and P. Blaha, Int. J.Quantum Chem. 75, 889 (1999)

[28] L. Vitos, B. Johansson, J. Kollar, andH. Skriver, Phys. Rev. B 62, 10046 (2000)

[29] V. Staroverov, G. Scuseria, J. Tao, andJ. Perdew, Phys. Rev. B 69, 075102 (2004)

[30] V. Staroverov, G. Scuseria, J. Tao, andJ. Perdew, Phys. Rev. B 78, 239907(E) (2008)

[31] J. Heyd, J. Peralta, G. Scuseria, and R. Mar-tin, J. Chem. Phys. 123, 174101 (2005)

[32] J. Paier, M. Marsman, K. Hummer,

G. Kresse, I. Gerber, and J. Angyan, J.Chem. Phys. 124, 154709 (2006)

[33] F. Tran, R. Laskowski, P. Blaha, andK. Schwarz, Phys. Rev. B 75, 115131 (2007)

[34] B. Grabowski, T. Hickel, and J. Neugebauer,Phys. Rev. B 76, 024309 (2007)

25

[35] M. Ropo, K. Kokko, and L. Vitos, Phys. Rev.B 77, 195445 (2008)

[36] S. Lany, Phys. Rev. B 78, 245207 (2008)[37] P. Haas, F. Tran, and P. Blaha, Phys. Rev. B

79, 085104 (2009)[38] P. Haas, F. Tran, and P. Blaha, Phys. Rev. B

79, 209902(E) (2009)[39] G. Csonka, J. Perdew, A. Ruzsinszky,

P. Philipsen, S. Lebegue, J. Paier, O. Vy-drov, and J. Angyan, Phys. Rev. B 79, 155107(2009)

[40] S. Shang, A. Saengdeejing, Z. Mei, D. Kim,H. Zhang, S. Ganeshan, Y. Wang, and Z. Liu,Comput. Mater. Sci. 48, 813 (2010)

[41] R. Nazarov, T. Hickel, and J. Neugebauer,Phys. Rev. B 85, 144118 (2012)

[42] G. Hautier, S. Ong, A. Jain, C. Moore, andG. Ceder, Phys. Rev. B 85, 155208 (2012)

[43] P. Villars and J. Daams, J. Alloys Compd.197, 177 (1993)

[44] http://www.marcelswart.eu/dft-poll

[45] W. Dickhoff and D. Van Neck, Many-BodyTheory Exposed!, 2nd ed. (World Scientific,2008)

[46] C. Kittel, Introduction to Solid State Physics,8th ed. (John Wiley & Sons, Inc, 2005)

[47] F. Birch, Phys. Rev. 71, 809 (1947)[48] P. Philipsen and E. Baerends, Phys. Rev. B

54, 5326 (1996)[49] E. Baerends, V. Branchadell, and M. Sodupe,

Chem. Phys. Lett. 265, 481 (1997)[50] D. Kim, S. Shang, and Z. Liu, Comput. Mater.

Sci. 47, 254 (2009)[51] N. Miao, B. Sa, J. Zhou, and Z. Sun, Comput.

Mater. Sci. 50, 1559 (2011)[52] Y. Le Page and P. Saxe, Phys. Rev. B 63,

174103 (2001)[53] Y. Le Page and P. Saxe, Phys. Rev. B 65,

104104 (2002)[54] S. Shang, Y. Wang, and Z. Liu, Appl. Phys.

Lett. 90, 101909 (2007)[55] A. Alchagirov, J. Perdew, J. Boettger, R. Al-

bers, and C. Fiolhais, Phys. Rev. B 63, 224115(2001)

[56] V. Moruzzi, J. Janak, and K. Schwarz, Phys.Rev. B 37, 790 (1988)

[57] Y. Tsuru, Y. Shinzato, Y. Saito, M. Shimazu,M. Shiono, and M. Morinaga, J. Ceram. Soc.Jpn. 118, 241 (2010)

[58] M. Dacorogna and M. Cohen, Phys. Rev. B34, 4996 (1986)

[59] R. Gaudoin and W. Foulkes, Phys. Rev. B 66,052104 (2002)

[60] D. Pettifor, Bonding and Structure ofMolecules and Solids (Oxford UniversityPress, 1995)

[61] A. Masago, K. Shirai, and H. Katayama-Yoshida, Phys. Rev. B 73, 104102 (2006)

[62] T. Scott, Phys. Rep. 27C, 89 (1976)[63] Y. Freiman and H. Jodl, Phys. Rep. 401, 1

(2004)

[64] W. David, R. Ibberson, S. Cox, and P. Wood,Acta Crystallogr. Sect. B 62, 953 (2006)

[65] P. E. Blochl, Phys. Rev. B 50, 17953 (1994)[66] G. Kresse and D. Joubert, Phys. Rev. B 59,

1758 (1999)[67] G. Kresse and J. Furthmuller, Comput.

Mater. Sci. 6, 15 (1996)[68] http://cms.mpi.univie.ac.at/vasp/vasp/

vasp.html

[69] P. Blochl, O. Jepsen, and O. Andersen, Phys.Rev. B 49, 16223 (1994)

[70] H. Monkhorst and J. Pack, Phys. Rev. B 13,5188 (1976)

[71] See Supplementary Material atarXiv:1204.2733 [cond-mat.mtrl-sci] forcomputational details, the full calculated andexperimental data and the files necessary fora code comparison

[72] S. Grazulis, D. Chateigner, R. Downs,A. Yokochi, M. Quiros, L. Lutterotti, E. Man-akova, J. Butkus, P. Moeck, and A. Le Bail,J. Appl. Crystallogr. 42, 726 (2009)

[73] A. Belsky, M. Hellenbrandt, V. Karen, andP. Luksch, Acta Crystallogr. Sect. B 58, 364(2002)

[74] N. Greenwood and A. Earnshaw, Chem-istry of the Elements, 2nd ed. (Butterworth-Heinemann, 1997)

[75] T. Tohei, A. Kuwabara, F. Oba, andI. Tanaka, Phys. Rev. B 73, 064304 (2006)

[76] L. Bolz, M. Boyd, F. Mauer, and H. Peiser,Acta Crystallogr. 12, 247 (1959)

[77] L. Meyer, C. Barrett, and S. Greer, J. Chem.Phys. 49, 1902 (1968)

[78] D. Batchelder, D. Losee, and R. Simmons,Phys. Rev. 162, 767 (1967)

[79] B. Powell, K. Heal, and B. Torrie, Mol. Phys.53, 929 (1984)

[80] O. Peterson, D. Batchelder, and R. Simmons,Phys. Rev. 150, 703 (1966)

[81] D. Losee and R. Simmons, Phys. Rev. 172,944 (1968)

[82] D. Sears and H. Klug, J. Chem. Phys. 37,3002 (1962)

[83] C. Swenson, Phys. Rev. 111, 82 (1958)[84] E. Kim and C. Chen, Phys. Lett. A 326, 442

(2004)[85] Y. Endoh, G. Shirane, and J. Skalyo, Phys.

Rev. B 11, 1681 (1975)[86] E.-F. Dusing, W. Grosshans, and

W. Holzapfel, J. Phys. Colloques (Paris)45, 203 (1984)

[87] G. Keeler and D. Batchelder, J. Phys. C 3,510 (1970)

[88] H. Fujihisa and K. Takemura, Phys. Rev. B52, 13257 (1995)

[89] J. Skalyo, Y. Endoh, and G. Shirane, Phys.Rev. B 9, 1797 (1974)

[90] D. Price, J. Rowe, and R. Nicklow, Phys. Rev.B 3, 1268 (1971)

26

[91] N. Lurie, G. Shirane, and J. Skalyo, Phys.Rev. B 9, 2661 (1974)

[92] E. Knittle, in Mineral Physics and Crystal-lography: A Handbook of Physical Constants,edited by T. Ahrens (American GeophysicalUnion Press, 1995) pp. 98–142

[93] S. Batsanov, in Effects of Explosions on Mate-rials: Modification and Synthesis under HighPressure Shock Compression (Springer, NewYork, 1994)

[94] M. Guinan and D. Steinberg, J. Phys. Chem.Solids 35, 1501 (1974)

[95] C. Menoni, J. Hu, and I. Spain, Phys. Rev. B34, 362 (1986)

[96] H. Beister, K. Strossner, and K. Syassen,Phys. Rev. B 41, 5535 (1990)

[97] T. Kenichi, Phys. Rev. B 70, 012101 (2004)[98] J. Li, S. Liang, H. Guo, and B. Liu, J. Alloys

Compd. 431, 23 (2007)[99] L. Hector, J. Herbst, W. Wolf, P. Saxe, and

G. Kresse, Phys. Rev. B 76, 014121 (2007)[100] U. Buchenau, M. Heiroth, H. Schober, J. Ev-

ers, and G. Oehlinger, Phys. Rev. B 30, 3502(1984)

[101] D. Lide (ed.), CRC Handbook of Chemistryand Physics, 89th ed. (CRC Press, Boca Ra-ton, Florida, 2008-2009)

[102] J. Nelson and D. Riley, Proc. Phys. Soc. Lon-don 57, 477 (1945)

[103] R. Keyes, Phys. Rev. 92, 580 (1953)[104] J. Wallis, I. Sigalas, and S. Hart, J. Appl.

Crystallogr. 19, 273 (1986)[105] R. Pawar and V. Deshpande, J. Mater. Sci. 5,

1061 (1970)[106] M. Anderson and C. Swenson, Phys. Rev. B

28, 5395 (1983)[107] J. Perdew, J. Chevary, S. Vosko, K. Jackson,

M. Pederson, D. Singh, and C. Fiolhais, Phys.Rev. B 46, 6671 (1992)

[108] J. Perdew, J. Chevary, S. Vosko, K. Jackson,M. Pederson, D. Singh, and C. Fiolhais, Phys.Rev. B 48, 4978 (1993)

[109] A. Mattsson, R. Armiento, P. Schultz, andT. Mattsson, Phys. Rev. B 73, 195123 (2006)

[110] M. DeGroot and M. Schervish, Probability andStatistics, 3rd ed. (Addison-Wesley, 2002)

[111] E. Scerri, Found. Chem. 12, 69 (2010)[112] E. Scerri, J. Chem. Educ. 86, 1188 (2009)[113] P. Villars, K. Brandenburg, M. Berndt,

S. LeClair, A. Jackson, Y.-H. Pao, B. Igelnik,M. Oxley, B. Bakshi, P. Chen, and S. Iwata,J. Alloys Compd. 317-318, 26 (2001)

[114] J. Rodgers and W. Nicewander, Am. Stat. 42,59 (1988)

[115] S. Grimme, J. Comput. Chem. 27, 1787(2006)

[116] K. Lee, E. Murray, L. Kong, B. Lundqvist,and D. Langreth, Phys. Rev. B 82, 081101(R)(2010)

[117] N. Pace, G. Saunders, and Z. Sumengen, J.Phys. Chem. Solids 31, 1467 (1970)

[118] D. Richardson and J. Mahanty, J. Phys. C:Solid State Phys. 10, 3971 (1977)

[119] S. Appalakondaiah, G. Vaitheeswaran,S. Lebegue, N. Christensen, and A. Svane,Phys. Rev. B 86, 35105 (2012)

[120] D. Hobbs, J. Hafner, and D. Spisak, Phys.Rev. B 68, 014407 (2003)

[121] D. Singh and J. Ashkenazi, Phys. Rev. B 46,11570 (1992)

[122] R. Roberts, G. White, and E. Fawcett, Phys-ica B (Utrecht) 119, 63 (1983)

[123] N. Gaston, B. Paulus, K. Rosciszewski,P. Schwerdtfeger, and H. Stoll, Phys. Rev. B74, 094102 (2006)

[124] U. Wedig, M. Jansen, B. Paulus,K. Rosciszewski, and P. Sony, Phys. Rev. B75, 205123 (2007)

[125] D. Koelling and B. Harmon, J. Phys. C: SolidState 10, 3107 (1977)

[126] P. Philipsen and E. Baerends, Phys. Rev. B61, 1773 (2000)

[127] V. Ozolins and M. Korling, Phys. Rev. B 48,18304 (1993)

[128] P. Haas, F. Tran, P. Blaha, K. Schwarz, andR. Laskowski, Phys. Rev. B 80, 195109 (2009)

[129] A. Oganov and C. Glass, J. Chem. Phys. 124,244704 (2006)

[130] N. Abraham and M. Probert, Phys. Rev. B73, 224104 (2006)

[131] N. Abraham and M. Probert, Phys. Rev. B77, 134117 (2008)

[132] A. Kolmogorov, A. Shah, E. Margine,A. Bialon, T. Hammerschmidt, andR. Drautz, Phys. Rev. Lett. 105, 217003(2010)

[133] P. Blaha, K. Schwarz, G. Madsen, D. Kvas-nicka, and J. Luitz, “WIEN2k, An AugmentedPlane Wave + Local Orbitals Program forCalculating Crystal Properties,” (KarlheinzSchwarz, Techn. Universitat Wien, Austria,1999)

[134] E. Sjostedt, L. Nordstrom, and D. Singh, SolidState Commun. 114, 15 (2000)

[135] G. Madsen, P. Blaha, K. Schwarz, E. Sjostedt,and L. Nordstrom, Phys. Rev. B 64, 195134(2001)

[136] S. Bahn and K. Jacobsen, Comput. Sci. Eng.4, 56 (2002)

[137] J. Mortensen, L. Hansen, and K. Jacobsen,Phys. Rev. B 71, 035109 (2005)

[138] J. Enkovaara, C. Rostgaard, J. Mortensen,J. Chen, M. Du lak, L. Ferrighi, J. Gavn-holt, C. Glinsvad, V. Haikola, H. Hansen,H. Kristoffersen, M. Kuisma, A. Larsen,L. Lehtovaara, M. Ljungberg, O. Lopez-Acevedo, P. Moses, J. Ojanen, T. Olsen,V. Petzold, N. Romero, J. Stausholm-Møller,M. Strange, G. Tritsaris, M. Vanin, M. Wal-ter, B. Hammer, H. Hakkinen, G. Mad-sen, R. Nieminen, J. Nørskov, M. Puska,T. Rantala, J. Schiøtz, K. Thygesen, and

27

K. Jacobsen, J. Phys.: Condens. Matter 22,253202 (2010)

[139] E. Rapoport and G. Kennedy, J. Phys. Chem.Solids 27, 93 (1966)

[140] H. Luo, R. Greene, and A. Ruoff, Phys. Rev.Lett. 71, 2943 (1993)

[141] M. Du lak, private communication[142] http://molmod.ugent.be/DeltaCodesDFT