GTC – Gyrokinetic Toroidal Code Training course: Part Iphoenix.ps.uci.edu/gtc_group/gtc/presentations/tutorial/GTC_tutorial.pdf · March 5-7, 2011 GTC Training Course - PKU 4 Gyrokinetic

GTC – Gyrokinetic Toroidal CodeTraining course: Part I

Ihor HolodUniversity of California, Irvine

March 5-7, 2011 GTC Training Course - PKU 2

Outline

• Introduction• Gyrokinetic approach• GK equations of motion. GK Poisson’s equation. Delta-f

method• Fluid-kinetic hybrid electron model: continuity equation,

Ampere’s law, drift-kinetic equation• Tokamak geometry. Magnetic coordinates• Particle-in-cell approach. General structure of GTC. Grid.

Parallel computing.• Setup and loading


Introduction

• Gryrokinetic simulations – important tool to study turbulence in magnetically confined plasma• Main target – low frequency microinstabilities• Source of free energy is needed to drive instabilities in plasma

• Pressure gradient of ions, electrons or fast (energetic) particles

• Ion temperature gradient mode (ITG)• Trapped electron mode (TEM)• Electron-temperature gradient mode (ETG)• Energetic particle mode (EPM)


Gyrokinetic approach

1~~~~ < <Ω

εδδφωF

f

B

B

T

e

• Main advantage: gyroaveraging reduces dimensionality of particle dynamics from 6 to 5 [Hahm et al. PoF’88]• Gyrokinetic ordering:

• Frequency smaller than gyrofrequency• Small fluctuation amplitude

1~sk ρ⊥ 1~|| < <⊥

εk

k• Additionally

• Magnetic field inhomogeneity is small: 1~ln < <∇ BB ερ


Charged particle motion in constant uniform magnetic filed


Guiding center motion in the inhomogeneous magnetic filed

dvdt

dvb

X += 0||ˆ 0=

dt

dµ

gcd vvv +=

0

2|| b×∇

Ω=v

vc

• Magnetic drift velocity

• Magnetic curvature drift

• Magnetic gradient drift

00ˆ B

mg ∇×Ω

= bvµ

00|| ˆ B

mdt

dv∇⋅−= b

µ

( ) ( ) ( )|||| ,,,,, vv, GYingGyroaverag

GC µθµ XXvx →→


• Gyrocenter equations of motion [Brizard and Hahm, RMP‘07]:

Gyrocenter equations of motion

Edvdt

dvvb

X ++= 0||ˆ

( )t

A

mc

ZZB

Bmdt

dv

∂∂

−∇+∇⋅−= ||0

0

*|| 1 φµB

BbBBBB δδ +×∇Ω

+=+= 0||0

0*0

* ˆvB

0

0ˆ

B

cE

φ∇×= bv

• Modified magnetic field:

• ExB drift velocity:

• Magnetic field perturbation:

0||bBB A×∇== ⊥δδ


Gyrokinetic Vlasov-Maxwell’s system of equations

• Gyrokinetic Poisson’s equation

( ) ( )eiii

ii ennZT

nZ −=− πφφπ4

~4 2

• Gyrokinetic Ampere’s law

( )iiiee unZuenc

A ||||||2 4 −=∇⊥

π

0||

|| =∂∂+∇⋅+

∂∂

v

fvf

t

fX

• Gyrokinetic Vlasov equation

• Charge density and current

∫

∫=

=

fvddvm

BZZnu

fddvm

BZZn

||||0

||

||0

µπ

µπ


Particle-in-cell method

Monte-Carlo sampling of phase space volume

• Phase space is randomly sampled by Lagrangian “markers”• Number of markers is much smaller than # of physical particles• Dynamics of markers is governed by the same equations of motion as for physical particles• Fluid moments, like density and current, are calculated on Eulerian grid

Fluid moments and potentialsare calculated on stationary grid


Delta-f method• Discrete particle noise – statistical error from MC sampling• Noise can be reduced by increasing # of markers, hence increasing CPU time• Delta-f method to reduce noise:

Perturbed part of the distribution function is considered as additional dynamical quantity associated with particle, and is governed by gyrokinetic equation Fluid moments are calculated integrating contributions from each individual particle

Edvdt

dvvb

X ++= 0||ˆ

( )t

A

mc

ZZB

Bmdt

dv

∂∂

−∇+∇⋅−= ||0

0

*|| 1 φµB

( )

∂∂

∂∂

+∇⋅+∇⋅+∇⋅

+−−=

||

0||

0

*

00

00

||

ln11ln1

v

f

mt

A

cBZB

Bf

Bvw

dt

dwE φδµδ BBv

B


Fluid-kinetic hybrid electron model - I

• Perturbed diamagnetic drift velocity

( )⊥+∇×Ω

= PPmn ee

δδ ||00

*ˆ1bv

( ) 0lnˆ0*0

0

00

0

||000 =∇+−∇⋅+∇⋅+

∂∂

BnB

nB

B

unB

t

nEE

ee vvvbδδ

• Electron continuity equation

• Perturbed pressure

∫

∫=

=⊥

fmvddvm

BP

fBddvm

BP

δµπδ

δµµπδ

2||||

0||

0||0

• Simulations of various Alfven modes


Fluid-kinetic hybrid electron model - II

• Inverse gyrokinetic Ampere’s law

iiiee unZc

Auenc ||||

2||

44 ππ +∇= ⊥

δψψ

δφ∂

∂−= 0

0

)0(ln n

n

n

T

ee

e

eff

• Electron adiabatic response

• Faraday’s law (linear tearing mode is removed)

( )δφφδφ −∇⋅=∇⋅−−=∂∂

effEt

A

c 00|||| ˆˆ1

bb

δψψ

φδ∂

∂+= 0

)0(

0

)0( ln f

T

e

f

f

e

effe

• Lowest order solution of electron DKE based on ω/k||v|| expansion


Fluid-kinetic hybrid electron model - III

∂∂

+∇⋅−

−∇⋅

−

∂∂−∇⋅−

−−=

t

A

cT

ev

f

f

T

e

f

f

f

f

tfw

f

f

dt

dw

e

eE

e

ed

eeEe

ee

||||

0

)0(

0

)0(

0

)0(

00

)0(

ln1

δφδ

δδ

vv

v

• High-order electron kinetic response

0

)1()1(

n

n

T

ee

e

eff δφ−=

• Zonal flow – nonlinearly self-generated large scale ExB flow with electrostatic potential having k||=0


Tokamak geometry

0 g Iζ θ δ ψ= + + B

• Equilibrium magnetic field (covariant representation) in flux coordinates

( )1d

d q

θζ ψ

=

• Helicity of magnetic field is determined by safety factor q (number of toroidal turns of magnetic field line per one poloidal turn)

Magnetic flux surface


Magnetic coordinates - definition

• Covariant representation of the magnetic field ψδθζ ∇+∇+∇= Ig0B

• Contravariant representation of the magnetic field

ζψθψ ∇×∇−∇×∇= q0B

• Jacobian

)()()(

),(201

ψψψθψζθψIqg

BJ

+=∇×∇⋅∇=−

( )( )( )( )

( )( )εζζεθθ

εδεε

εθε

O

O

O

Og

OI

OB

+=+=

+=+=+=

+−=

0

0

2

2

20

0

1

0

cos1


Magnetic coordinates - example

• Parallel derivative

( )

∂∂+

∂∂=∇×∇⋅∇

∂∂+

∂∂

=

∇∂∂+∇

∂∂+∇

∂∂⋅∇×∇−∇×∇=∇⋅

θφ

ζφζθψ

θφ

ζφ

ζζφθ

θφψ

ψφζψθψφ

qJ

qq

q

1

0B

ζθψ ∇×∇⋅∇=−1J


GTC grids

• Two spatial grids

• Coarse grid for equilibrium and profiles

• Field-aligned simulation mesh for fields

• Spline interpolation is used to calculate equilibrium quantities on simulation grid and particle positions


Simulation grid

• Radial grid in ψ (i=0:mpsi)

• Poloidal angle (θ) grid with continuous indexing (ij=1:mgrid) over the whole poloidal plane

• Number of poloidal grid points mtheta(i)differs from one flux surface to another

mtheta(i)

2πθ =∆

∆

=θ

θintegerj jigrid(i)ij +=

ij igrid(0)=1

do i=1,mpsi

igrid(i)=igrid(i-1) + mtheta(i-1) + 1

enddo


Parallel computing

• Distribution of independent tasks between different processors

• Domain decomposition: each processor simulates it’s own part of simulation domain

• Particle decomposition: each processor is responsible for it’s own set of particles

Node 0

Core 0

Core 1

Core 2

Core 3

Node 1

Core 0

Core 1

Core 2

Core 3

Supercomputer structure


MPI-Parallelization

MPI_BCAST(send_buff,buff_size,data_type,send_pe,comm,ierror)

MPI_REDUCE(send_buff,recv_buff,buff_size,data_type,operation,recv_pe,comm,ierror)

MPI_ALLREDUCE(send_buff,recv_buff,buff_size,data_type,operation,comm,ierror)

• Parallelization between nodes

• Each node has independent memory

• Inter-node communication through special commands


OpenMP-Parallelization

!$omp parallel do private(m)

do m=1,mi

…

enddo

• Parallelization between cores of one node

• Each core has access to common memory

• Used for parallelization of cycles

• Useful with modern multi-core processors (GPU)


δne δA||

δue

δge1&δfi

δneδne1δuiδA|| δni δue1

δφindδφesδA|| ZF

Dynamics

Sources

Fields

GTC structure


main.F90 CALL INITIAL(cdate,timecpu)

CALL SETUP

CALL LOAD

CALL LOCATEI

if(fload>0)CALL LOCATEF

! main time loop

do istep=1,mstep

do irk=1,2

! idiag=0: do time history diagnosis at irk=1 & istep=ndiag*N

idiag=mod(irk+1,2)+mod(istep,ndiag)

if(idiag==0)then

if(ndata3d==1)CALL DATAOUT3D !write 3D fluid data

if(track_particles==1)CALL PARTOUT !write particle data

endif

CALL FIELD_GRADIENT

if(magnetic==1)CALL PUSHFIELD

CALL PUSHI

CALL SHIFTI(mimax,mi)

CALL LOCATEI

CALL CHARGEI

if(fload>0)then

CALL PUSHF

CALL SHIFTF(mfmax,mf)

CALL LOCATEF

CALL CHARGEF

endif

if(magnetic==0)then

CALL POISSON_SOLVER("adiabatic-electron")

else

CALL POISSON_SOLVER("electromagnetic")

CALL FIELD_SOLVER

endif


main.F90 (cont)

! push electron, sub-cycling do i=1,ncycle*irk! 1st RK step CALL PUSHE(i,1) CALL SHIFTE(memax,me)! 2nd RK step CALL PUSHE(i,2) CALL SHIFTE(memax,me)! electron-ion and electron-electron collision if(ecoll>0 .and. mod(i,ecoll)==0)then call lorentz call collisionee endif enddo CALL CHARGEE CALL POISSON_SOLVER("kinetic-electron") enddo if(idiag==0)CALL DIAGNOSIS enddo! i-i collisions if(icoll>0 .and. mod(istep,icoll)==0)call collisionii

if(mod(istep,mstep/msnap)==0 .or. istep*(1-irun)==ndiag)CALL SNAPSHOT if(mod(istep,mstep/msnap)==0)CALL RESTART_IO("write") enddo CALL FINAL(cdate,timecpu)


if(numereq==1)then ! Cyclone case psiw=0.0375 !poloidal flux at wall! q and zeff profile is parabolic: q=q1+q2*psi/psiw+q3*(psi/psiw)^2 q=(/0.82,1.1,1.0/) !cyclone case er=(/0.0,0.0,0.0/) itemp0=1.0 ! on-axis thermal ion temperature, unit=T_e0 ftemp0=2.0 ! on-axis fast ion temperature, unit=T_e0 fden0=1.0e-5 ! on-axis fast ion density, unit=n_e0! density and temperature profiles are hyperbolic: ! ne=1.0+ne1*(tanh(ne2-(psi/psiw)/ne3)-1.0) ne=(/0.205,0.30,0.4/) !Cyclone base case te=(/0.415,0.18,0.4/) ti=(/0.415,0.18,0.4/) ! ITG tf=ti ! fast ion temperature profile nf=ne ! fast ion density profile

analytic.F90

• Specifies profiles for different analytic equilibriums


! allocate memory for equilibrium spline array allocate (stpp(lsp),mipp(lsp),mapp(lsp),nepp(3,lsp),tepp(3,lsp),& nipp(3,lsp),tipp(3,lsp),zepp(3,lsp),ropp(3,lsp),erpp(3,lsp),& qpsi(3,lsp),gpsi(3,lsp),ppsi(3,lsp),rpsi(3,lsp),torpsi(3,lsp),& bsp(9,lsp,lst),xsp(9,lsp,lst),zsp(9,lsp,lst),gsp(9,lsp,lst),& rd(9,lsp,lst),nu(9,lsp,lst),dl(9,lsp,lst),spcos(3,lst),spsin(3,lst),& rgpsi(3,lsp),cpsi(3,lsp),psirg(3,lsp),psitor(3,lsp),nfpp(3,lsp),& tfpp(3,lsp))

if(mype==0)then if(numereq==0)then ! use EFIT & TRANSP data! EFIT spdata used in GTC: 2D spline b,x,z,rd; 1D spline qpsi,gpsi,torpsi call spdata(iformat,isp)! TRANSP prodata used in GTC: tipp,tepp,nepp,zeff,ropp,erpp call prodata else! Construct analytic equilibrium and profile assuming high aspect-ratio, concentric cross-section call analyticeq endif

eqdata.F90• Equilibrium and profiles are represented using 2D-spline on (psi, theta) or 1D-spline on (psi)


!$omp parallel do private(i,pdum,isp,dpx,dp2) do i=0,mpsi pdum=psimesh(i) isp=max(1,min(lsp-1,ceiling(pdum*spdpsi_inv))) dpx=pdum-spdpsi*real(isp-1) dp2=dpx*dpx

! 1D spline in psi meshni(i)=nipp(1,isp)+nipp(2,isp)*dpx+nipp(3,isp)*dp2 kapani(i)= 0.0-(nipp(2,isp)+2.0*nipp(3,isp)*dpx)/meshni(i) ... enddo

setup.F90

• Read input parameters

• Initialize quantities determined on equilibrium mesh

! allocate memory allocate(phi(0:1,mgrid), gradphi(3,0:1,mgrid),...,STAT=mtest)

• Allocate fields


!iload=0: ideal MHD with nonuniform pressure profile!iload=1: Marker uniform temperature & density: n0,te0,ti0 (two-scale expansion) !iload=2: Marker non-uniform temperature & uniform density: n0,te(r),ti(r), marker weight corrected!iload=3: Physical PDF for marker with non-uniform temperature & density: n(r),te(r),ti(r)!iload>99: non-perturbative (full-f) simulation

loadi.F90• Loads ion markers

zion(nparam,mimax)nparam=6 (8 if particle tracking is on)zion(1,m) – ψzion(2,m) – θzion(3,m) – ζzion(4,m) – ρ| | (parallel velocity)zion(5,m) – wi (particle weight)zion(6,m) – µ (magnetic moment)

• Marker data array:

Documents

GTC – Gyrokinetic Toroidal Code Training course: Part Iphoenix.ps.uci.edu/gtc_group/gtc/presentations/tutorial/GTC_tutorial.pdf · March 5-7, 2011 GTC Training Course - PKU 4 Gyrokinetic