Atomistic and molecular simulations on massively parallel ...bigdft.org/images/3/32/2013-07_TD_BigDFT_Introduction.pdfBigDFT Introduction Running BigDFT Atom positions Basis set Pseudopotential

BigDFThttp://bigdft.org

Introduction

RunningBigDFTAtom positions

Basis set

Pseudopotential

XC

SCF Loop

PerformancesPoisson Solver

Relaxation

HPC

Conclusion

Atomistic and molecular simulations on massivelyparallel architectures

CECAM PARIS, FRANCE

BigDFT — An introduction

Thierry Deutsch, Damien Caliste, Luigi Genovese

L_Sim - CEA Grenoble

17 July 2013

Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Thierry Deutsch


Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Objectives of the lesson

To know what are the main parameters in a DFT calculation

To run a calculation (i.e. the physicist problem)

1 Introduction

2 Running BigDFTAtom positionsBasis setPseudopotentialExchange-correlationSCF Loop

3 PerformancesPoisson SolverRelaxationHigh-Performance Computing (parallel and GPU)

4 Conclusion



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

A basis for nanosciences: the BigDFT project

STREP European project: BigDFT(2005-2008)Four partners, 15 contributors:CEA-INAC Grenoble, U. Basel, U. Louvain-la-Neuve, U. Kiel

Aim: To develop an ab-initio DFT codebased on Daubechies Wavelets, to beintegrated in ABINIT, distributed freely(GNU-GPL license)

L. Genovese, A. Neelov, S. Goedecker, T. Deutsch, et al.,“Daubechies wavelets as a basis set for density functional pseudopotential calculations”,

J. Chem. Phys. 129, 014109 (2008)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

BigDFT version 1.7: capabilities

http://bigdft.orgFree, surface and periodic boundary conditions

Geometry optimizations (with constraints)

Born-Oppenheimer Molecular Dynamics

Saddle point searches (Nudged-Elastic Band Method)

Vibrations

External electric fields

Unoccupied KS orbitals

Collinear and Non-collinear magnetism

All XC functionals of the ABINIT package

Hybrid functionals

Empirical van der Waals interactions (many flavors)

Also available within the ABINIT package



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Kohn-Sham formalism

H-K theorem: E is an unknown functional of the densityE = E[ρ]→ Density Functional Theory

Kohn-Sham approachMapping of an interacting many-electron system into asystem with independent particles moving into an effectivepotential.

Find a set of orthonormal orbitals Ψi(r) that minimizes:

E =−12

N/2

∑i=1

∫Ψ∗i (r)∇

2Ψi(r)dr +12

∫ρ(r)VH(r)dr

+ Exc[ρ(r)] +∫

Vext(r)ρ(r)dr

ρ(r) =N/2

∑i=1

Ψ∗i (r)Ψi(r) ∇2VH(r) =−4πρ(r)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Spirit of BigDFT

Few input files (posinp.xyz, input.dft)

Few input parametersex. not parameters for parallel calculations

Use input files with mandatory parameters

Optional input filesas input.kpt (for periodic systems)or input.perf (parameters for performance)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

How to run BigDFT

Some pre-processingbigdft-tool nprocs [name]Help to find the optimal number of processors for the run thatfit into the computer memory. Also useful to check that allinput files are without errors.

The main runmpirun -np 8 bigdft [name] | tee [name].logNumber of MPI tasks : 8OpenMP parallelization : YesMaximal OpenMP threads per MPI task : 1

Material acceleration : No #iproc=0

The outputslog is output on stdout, so redirect it.

generated data are stored in a subdirectory calleddata[-name].



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

The BigDFT input files (or not)

posinp.xyz: the atomic coordinates and box definition.psppar.Element: pseudo-potentials for each element;input.dft: DFT related parameters;input.geopt: geometrie relaxations and MD;input.kpt: the k -point mesh and band structure path;input.mix: diagonalisation mechanism;input.sic, tddft, occ, perf: other parameters;

Only the coordinates are mandatory.All lines of different input files are mandatory and output onBigDFT log. When BigDFT is run, all default values areoutput in files called default.xxx.

Naming schemeAll input files may be renamed by providing a name argumentto the command line i.e. bigdft name.



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Atomic positions: posinp.xyz or posinp.ascii

units reduced, angstroem, bohr or atomic

Boundary conditions periodic, surface

8 reducedperiodic 10.26085 10.2608510.26085Si 0. 0. 0.Si 0.5 0.5 0.Si 0.5 0. 0.5Si 0. 0.5 0.5Si 0.25 0.25 0.25Si 0.75 0.75 0.25Si 0.75 0.25 0.75Si 0.25 0.75 0.75



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion




3 angstroemsurface 3.0 0.0 3.0O 1.5 0.0 1.5H .7285 0.620919 1.5H 2.2715 0.620919 1.5



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion




5 atomic

Si 1.62 1.62 1.62H 3.24 3.24 3.24H 0 0 3.24H 3.24 0 0H 0 3.24 0



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Atomic coordinate format

BigDFT uses XYZ format plus additional modifications:physical unit, after the number of atoms on the first lineput either bohr or angstroem.boundary conditions, on second line, start with one offreeBC, surface X lat 0 Zlat or periodic X lat Y lat Zlat.frozen positions, add a f, fy or fxz at the end of anatom line.input polarisation, add values at the end of an atom line.

On output, forces are added for each element at the end ofthe XYZ file:2 angstroem

freeBCNa -1 0 0Cl +1 0 0forcesNa 2.654367E-2 0 0Cl -2.654367E-2 0 0Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Thierry Deutsch


Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Performing a DFT calculation

A self-consistent equation

ρ(r) = ∑i

Ψ∗i (r)Ψi(r), where |ψi〉 satisfies(−1

2∇

2 + VH [ρ] + Vxc[ρ] + Vext + Vpseudo

)|ψi〉= Ei |ψi〉 ,

(Kohn-Sham) DFT “Ingredients”

A basis set for expressing the |ψi〉An XC potential, functional of the densityseveral approximations exists (LDA,GGA,. . . )

A choice of the pseudopotential (if not all-electrons)(norm conserving, ultrasoft, PAW,. . . )

An (iterative) algorithm for finding the wavefunctions |ψi〉A (good) computer. . .



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Basis sets in real space: Wavelet basis sets

Properties of wavelet basis sets:

localized both in realand in Fourier space

allow for adaptivity(for core electrons)

are a systematic basis set

-1.5

-1

-0.5

0

0.5

1

1.5

-6 -4 -2 0 2 4 6 8

x

φ(x)

ψ(x)

A basis set both adaptive and systematic, real space based



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

A brief description of wavelet theory

Two kind of basis functions

A Multi-Resolution real space basisThe functions can be classified following the resolution levelthey span.

Scaling FunctionsThe functions of low resolution level are a linear combinationof high-resolution functions

= +

φ(x) =m

∑j=−m

hjφ(2x− j)

Centered on a resolution-dependent grid: φj = φ0(x− j).



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

A brief description of wavelet theory

WaveletsThey contain the DoF needed to complete the informationwhich is lacking due to the coarseness of the resolution.

= 12 + 1

2

φ(2x) =m

∑j=−m

h̃jφ(x− j) +m

∑j=−m

g̃jψ(x− j)

Increase the resolution without modifying grid spaceSF + W = same DoF of SF of higher resolution

ψ(x) =m

∑j=−m

gjφ(2x− j)

All functions have compact support, centered on grid points.



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Adaptivity of the mesh

AdaptivityResolution can be refined following the grid point.

The grid is divided in:

Low resolution pts(SF, 1 DoF)

High resolution pts(SF + W, 8 DoF)

Points of different resolutionbelong to the same grid.

Data can thus be stored in compressed form.

Localization propertyEmpty regions must not be “filled” with basis functions.Optimal for big inhomogeneous systems, O(N) approach.



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


Atomic positions (H2O), hgrid



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


Fine grid (high resolution, frmult)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


Coarse grid (low resolution, crmult)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Defining the basis set in input.dft

Main specific variables to control the basis set

hgrid the grid size

frmult the expansionaround atoms for fine grid

crmult the expansionaround atoms for coarsegrid

Convergence propertiesDecreasing hgrid→ more degrees of freedom.

Increasing crmult→ less box constraint.

It is important to improve both parameters together to avoiderror on one hiding improvement made on the other one.



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

input.dft: hgrid, crmult, frmult

hgrid Step grid: twice as bigger than the lowestgaussian coefficient of pseudopotentialRoughly 0.45 (only orthorhombic mesh)

crmult Extension of the coarse resolution (factor 6.):depends on the extension of the HOMO for theatom

frmult Extension of the fine resolution (factor 8.):depends on the pseudopotential coefficient

In practice, we use only hgrid from 0.3 to 0.55.



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Wavelets: No integration error

Orthogonality, scaling relation∫dx φk (x)φj(x) = δkj φ(x) =

1√2

m

∑j=−m

hjφ(2x− j)

The hamiltonian-related quantities can be calculated up tomachine precision in the given basis.

The accuracy is only limited by the basis set (O(h14

grid

))

Exact evaluation of kinetic energyObtained by convolution with filters:

f (x) = ∑`

c`φ`(x) , ∇2f (x) = ∑

`

c̃` φ`(x) ,

c̃` = ∑j

cj a`−j , a` ≡∫

φ0(x)∂2xφ`(x) ,



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Wavelets: Separability in 3D

The 3-dim scaling basis is a tensor product decomposition of1-dim Scaling Functions/ Wavelets.

φex ,ey ,ezjx ,jy ,jz (x ,y ,z) = φ

exjx (x)φ

eyjy (y)φ

ezjz (z)

With (jx , jy , jz) the node coordinates,

φ(0)j and φ

(1)j the SF and the W respectively.

Gaussians and waveletsThe separability of the basis allows us to save computationaltime when performing scalar products with separablefunctions (e.g. gaussians):

Initial wavefunctions (input guess)

Poisson solver

Non-local pseudopotentials



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



ρ(r) = ∑i


2∇


)|ψi〉= Ei |ψi〉 ,







Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Gaussian type separable Pseudopotentials (HGH)

Local part

Vloc(r) =−Zion

rerf

[r√2rloc

]+ exp

[−1

2

(r

rloc

)2]

{C1 + C2

(r

rloc

)2

+ C3

(r

rloc

)4

+ C4

(r

rloc

)6}

Nonlocal (separable) part H(~r ,~r ′)

Hsep(~r ,~r ′) =2

∑i=1

∑m

Ys,m(r̂) psi (r) hs

i psi (r ′) Y ∗s,m(r̂ ′)

+ ∑m

Yp,m(r̂) pp1(r) hp

1 pp1(r ′) Y ∗p,m(r̂ ′)

pl1(r) =

√2

r le−12 (

rrl)2

rl+ 3

2l

√Γ(l + 3

2 )pl

2(r) =√

2r l+2e−

12 (

rrl)2

rl+ 7

2l

√Γ(l + 7

2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Pseudopotential file (as psppar.N)

No need if you use LDA or PBE functionals

Default pseudopotentials inside BigDFT!

Put in the same directory as the calculation.

Large library:http://www.abinit.org/downloads/psp-links

Accurate and transferable



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



ρ(r) = ∑i


2∇


)|ψi〉= Ei |ψi〉 ,







Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Exchange-Correlation Energy (input.dft)

Exc [ρ] = Kexact [ρ]−KKS [ρ] + Ve−e exact [ρ]−VHartree [ρ]

KKS [ρ]: Kinetic energy for a non-interaction electron gas

Kohn-Sham formalism is exact (mapping of aninteracting electron system into a non-interactingsystem)

Many types from

ABINIT: Use same codes(ixc: LDA=1, PBE=11, see manual)

libXC: − ’code for exchange’ ’code for correlation’(−101130 see manual)

Hybrid functionals from libXC



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



ρ(r) = ∑i


2∇


)|ψi〉= Ei |ψi〉 ,







Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Kohn-Sham Equations: Operators

Apply different operators

Having a one-electron hamiltonian in a mean field.[−1

2∇

2 + Veff (r)(r)

]ψi = εi ψi

Veff (r) = Vext(r) +∫

V

ρ(r ′)|r − r ′|

dr ′+ µxc(r)

E[ρ] can be expressed by the orthonormalized states of oneparticule: ψi(r) with the fractional occupancy number fi(0≤ fi ≤ 1) :

ρ(r) = ∑i

fi |ψi(r)|2



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Kohn-Sham Equations: Computing Energies

Calculate different integrals

E[ρ] = K [ρ] + U[ρ]

K [ρ] =−12~2

me∑

i

∫V

dr fiψ∗i ∇

2ψi

U[ρ] =∫

Vdr Vext(r)ρ(r)+

12

∫V

dr dr ′ρ(r)ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+ Exc[ρ]︸︷︷︸exchange−correlation

We minimise with the variables ψi(r) and fi

with the constraint∫

Vdr ρ(r) = Nel .



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

KS Equations: Self-Consistent Field

Set of self-consistent equations:{−1

2~2

me∇

2 + Veff

}ψi = εiψi

with an effective potential:

Veff (r) = Vext (r)+∫

Vdr ′

ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+δExc

δρ(r)︸︷︷︸exchange−correlation

and: ρ(r) = 2∑i fi |ψi (r)|2

Poisson Equation: ∆VHartree = ρ (Laplacian: ∆ = ∂2

∂x2 + ∂2

∂y2 + ∂2

∂z2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



2~2

me∇

2 + Veff

}ψi = εiψi



Vdr ′

ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+δExc


and: ρ(r) = 2∑i fi |ψi (r)|2


∂x2 + ∂2

∂y2 + ∂2

∂z2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



2~2

me∇

2 + Veff

}ψi = εiψi



Vdr ′

ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+δExc


and: ρ(r) = 2∑i fi |ψi (r)|2


∂x2 + ∂2

∂y2 + ∂2

∂z2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



2~2

me∇

2 + Veff

}ψi = εiψi



Vdr ′

ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+δExc


and: ρ(r) = 2∑i fi |ψi (r)|2


∂x2 + ∂2

∂y2 + ∂2

∂z2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



2~2

me∇

2 + Veff

}ψi = εiψi



Vdr ′

ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+δExc


and: ρ(r) = 2∑i fi |ψi (r)|2


∂x2 + ∂2

∂y2 + ∂2

∂z2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion



2~2

me∇

2 + Veff

}ψi = εiψi



Vdr ′

ρ(r ′)|r − r ′|︸︷︷︸

Hartree

+δExc


and: ρ(r) = 2∑i fi |ψi (r)|2


∂x2 + ∂2

∂y2 + ∂2

∂z2 )



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Direct Minimisation: Flowchart

{ψi = ∑

ac i

aφa

}Basis: adaptive mesh (0, . . . ,Nφ)

Orthonormalized

ρ(r) =occ

∑i|ψi (r)|2 Fine non-adaptive mesh: < 8Nφ

inv FWT + Magic Filter

VH(r) =∫

G(.)ρ VeffectiveVxc [ρ(r)] VNL({ψi})

Poisson solver

− 12 ∇2

Kinetic Termδc i

a =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja

Λij =< ψi |H|ψj >

FWT

FWT

cnew ,ia = c i

a + hstepδc ia

Steepest Descent,DIIS

preconditioning

Stop when δc ia small



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


{ψi = ∑

ac i

aφa


Orthonormalized

ρ(r) =occ



VH(r) =∫


Poisson solver

− 12 ∇2

Kinetic Termδc i

a =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja


FWT

FWT

cnew ,ia = c i

a + hstepδc ia


preconditioning




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


{ψi = ∑

ac i

aφa


Orthonormalized

ρ(r) =occ



VH(r) =∫


Poisson solver

− 12 ∇2

Kinetic Termδc i

a =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja


FWT

FWT

cnew ,ia = c i

a + hstepδc ia


preconditioning




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


{ψi = ∑

ac i

aφa


Orthonormalized

ρ(r) =occ



VH(r) =∫


Poisson solver

− 12 ∇2

Kinetic Term

δc ia =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja


FWT

FWT

cnew ,ia = c i

a + hstepδc ia


preconditioning




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


{ψi = ∑

ac i

aφa


Orthonormalized

ρ(r) =occ



VH(r) =∫


Poisson solver

− 12 ∇2

Kinetic Termδc i

a =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja


FWT

FWT

cnew ,ia = c i

a + hstepδc ia


preconditioning




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


{ψi = ∑

ac i

aφa


Orthonormalized

ρ(r) =occ



VH(r) =∫


Poisson solver

− 12 ∇2

Kinetic Termδc i

a =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja


FWT

FWT

cnew ,ia = c i

a + hstepδc ia


preconditioning




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion


{ψi = ∑

ac i

aφa


Orthonormalized

ρ(r) =occ



VH(r) =∫


Poisson solver

− 12 ∇2

Kinetic Termδc i

a =− ∂Etotal

∂c∗i (a)+∑

jΛij c

ja


FWT

FWT

cnew ,ia = c i

a + hstepδc ia


preconditioning




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

input.dft: idsx DIIS history

Use DIIS algorithmDirect Inversion in the Iterative Subspace

Use idsx=6 previous iterations Memory-consuming(decrease if too big)

Steepest descent if idsx=0

itermax=50 Specify maximum number of iterationsTypically 15 iterations to converge

gnrm_cv=1.e-4 convergence criterionResidue |δψ|2 < gnrm_cv



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Description of the file input.dft

0.450 0.450 0.450 hx, hy, hz5.0 8.0 crmult, frmult, coarse and fine radius1 ixc: XC parameter (LDA=1,PBE=11)0 0.0 ncharge: charge of the system, Electric field1 0 nspin=1 non-spin polarization, total magnetic moment1.E-04 gnrm_cv: convergence criterion gradient50 10 itermax, nrepmax7 6 # CG (preconditioning), length of diis history0 dispersion correction functional (0, 1, 2, 3)0 0 0 InputPsiId, output_wf, output_grid5.0 30 length of the tail, # tail CG iterations0 0 0 davidson treatment, # virtual orbitals, # plotted orbitalsT disable the symmetry detection



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion





Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Basis sets: wavelets

[− 1

2∇2

non-relativistic

relativistic

N3 scaling

O(N) methods

+V (r)

all electrons

pseudopotential norm-conserving

PAWself-consistent (SCF)

Harris-Foulkes functional

periodic

non-periodic

+µxc(r)]LDA,GGA

beyond LDA GW

Hybrid functionals

LDA+U

non-spin polarized

spin polarizednon-collinear

collinear

ψki

atomic orbitals Gaussians

Slater

numerical

plane waves

augmented

real space finite difference

Wavelet

= εki ψk

i



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Optimal for isolated systems

Test case: cinchonidine molecule (44 atoms)

10-4

10-3

10-2

10-1

100

101

105 106

Abs

olut

e en

ergy

pre

cisi

on (

Ha)

Number of degrees of freedom

Ec = 125 Ha

Ec = 90 Ha

Ec = 40 Ha

h = 0.3bohr

h = 0.4bohr

Plane waves

Wavelets

Allows a systematic approach for moleculesConsiderably faster than Plane Waves codes.10 (5) times faster than ABINIT (CPMD)Charged systems can be treated explicitly with the same time



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Systematic basis set

Two parameters for tuning the basisThe grid spacing hgrid

The extension of the low resolution points crmult



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Comparison with a code based on gaussian basis set

The Daubechies expansion of a gaussian is known→Results from gaussian codes can be imported in BigDFT

Gaussian code (CP2K) test, H2O and DZVP basis

crmult (basis extension)

E [Ha]

3 4 5 6

−17.033

−17.123

−17.132

6-31G Conv energy

Wavelet Conv energy

BigDFT Input Guess Energy

CP2K Imported energy

BigDFT converged energy

BigDFT with finite-size corrections



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Poisson Solver using interpolating scaling function

The Hartree potential is calculated in the interpolating scalingfunction basis.

Poisson solver with interpolating scaling functions

From the density ρ(~j) on an uniform grid it calculates:

VH(r) =∫

ρ(r′)|r− r′|

dr′

4 Very fast and accurate, optimal parallelization

4 Can be used independently from the DFT code

4 Integrated quantities (energies) are easy to extract

8 Non-adaptive, needs data uncompression

Explicitly free boundary conditionsNo need to subtract supercell interactions (charged systems).



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Poisson Solver for surface boundary conditions

A Poisson solver for surface problemsBased on a mixed reciprocal-direct space representation

Vpx ,py (z) =−4π

∫dz ′G(|~p|;z− z ′)ρpx ,py (z) ,

4 Can be applied both in real or reciprocal space codes

4 No supercell or screening functions

4 More precise than other existing approaches

Example of the plane capacitor:

Periodic

x

y

V

x

Hockney

x

y

V

x

Our approach

x

y

V

x



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

input.geopt: Relaxation or molecular dynamics

DIIS500 ncount_cluster: max steps during geometry relaxation1.d0 1.d-3 frac_fluct: geometry optimization stops

if force norm less than frac_fluct of noisemaxval: maximum value of atomic forces

0.0d0 randdis: random amplitude for atoms4.d0 5 betax: steepest descent step size, diis history

Different algorithms:

SDCG: Steepest Descent Conjugate Gradient

DIIS: Direct Inversion in the iterative subspace

VSSD: Variable Size Steepest Descent

LBFGS: Limited BFGD

AB6MD: Use molecular dynamics routines fromABINIT 6



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion









LBFGS: Limited BFGD




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion









LBFGS: Limited BFGD




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion









LBFGS: Limited BFGD




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion









LBFGS: Limited BFGD




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion









LBFGS: Limited BFGD




Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

High-Performance Computing: Massively parallel

Two kinds of parallelisationBy orbitals (Hamiltonian application, preconditioning)

By components (overlap matrices, orthogonalisation)

A few (but big) packets of dataMore demanding in bandwidth than in latency

No need of fast network

Optimal speedup (eff. ∼ 85%), also for big systems

Cubic scaling codeFor systems bigger than 500 atomes (1500 orbitals) :orthonormalisation operation is predominant (N3)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Orbital distribution scheme

Used for the application of the hamiltonianThe hamiltonian (convolutions) is applied separately ontoeach wavefunction

ψ5

ψ4

ψ3

ψ2

ψ1

MPI 0

MPI 1

MPI 2



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Coefficient distribution scheme

Used for scalar product & orthonormalisationBLAS routines (level 3) are called, then result is reduced

ψ5

ψ4

ψ3

ψ2

ψ1

MPI 0 MPI 1 MPI 2

Communications are performed via MPI_ALLTOALLV



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Performance in parallel (BigDFT)

Distribution per orbitals (wavelets)Nothing to specify in input files



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

GPU-ported operations in BigDFT (double precision)

Convolutions(OpenCL rewritten)GPU speedups between10 and 20 can beobtained for differentsections 2

4

6

8

10

12

14

16

18

20

0 10 20 30 40 50 60 70

GP

U s

peed

up (

Dou

ble

prec

.)

Wavefunction size (MB)

locdenlocham

precond

0

10

20

30

40

50

60

70

0 500 1000 1500 2000 2500

GP

U s

peed

up (

Dou

ble

prec

.)

Number of orbitals

DGEMMDSYRK

Linear algebra(CUBLAS library)The interfacing withCUBLAS is immediate,with considerablespeedups



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

BigDFT code on Hybrid architectures

BigDFT code can run on hybrid CPU/GPU supercomputersIn multi-GPU environments, double precision calculations

No Hot-spot operationsDifferent code sections can be ported on GPUup to 20x speedup for some operations,7x for the full parallel code

0

20

40

60

80

100

1 2 4 6 8 10 12 14 16 1 2 4 6 8 10 12 14 16 1

10

100

1000

Per

cent

Sec

onds

(lo

g. s

cale

)

Number of cores

LinAlglocdenlochamprecondPureCPUOtherCommTime (sec)

Hybrid code (rel.)CPU code



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

input.perf file

OpenCL acceleration:

Add this line in in input.perf

accel OCLGPU

See the web pagehttp://bigdft.org/Wiki/index.php?title=Input.perf

for more information about all keywords (verbosity, inputguess, ...)associated to the performance of the BigDFT code.

The log file gives the list of possible keywords.



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

BigDFT: Future developments

[− 1

2∇2

non-relativistic

relativistic

N3 scaling

O(N) methods

+V (r)

all electrons

pseudopotential norm-conserving

PAWself-consistent (SCF)

Harris-Foulkes functional

periodic

non-periodic

+µxc(r)]LDA,GGA

beyond LDA GW

Hybrid functionals

LDA+U

non-spin polarized

spin polarizednon-collinear

collinear

ψki

atomic orbitals Gaussians

Slater

numerical

plane waves

augmented

real space finite difference

Wavelet

= εki ψk

i



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Linear scaling version

16 min: 1000 atoms with the cubic scaling,8000 atoms with the linear scaling one



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Conclusion

Functionality: Mixed between physics and chemistryapproach

Order N (real space basis set as wavelets)

Multi-scale approach (more than 1000 atoms feasible fora better parametrization)

Numerical experience:One-day simulation

Better exploration of atomic configurations

Molecular Dynamics(4s per step for 32 water molecules)



Introduction


Basis set

Pseudopotential

XC

SCF Loop


Relaxation

HPC

Conclusion

Hands on

Seehttp://bigdft.org/Wiki/index.php?title=Category:Tutorials

First runs with BigDFT

Basis-set convergence

Acceleration example on different platforms:Kohn-Sham DFT Operation with GPU acceleration


Documents

Atomistic and molecular simulations on massively parallel ...bigdft.org/images/3/32/2013-07_TD_BigDFT_Introduction.pdfBigDFT Introduction Running BigDFT Atom positions Basis set Pseudopotential