1
Gyrokinetic simulations of turbulence in magnetic fusion
plasmas
L. Villard, P. Angelino (*) , A. Bottino(1), S. Brunner, S. Jolliet, B.F. McMillan(2), B. Teaca (*) , T.M. Tran, T. Vernay
Centre de Recherches en Physique des Plasmas, EPFL, Switzerland(1) Max Planck IPP, Garching, Germany
(2) Center for Fusion, Space and Astrophysics, U Warwick, UK(*) Supported by the HP2C project
Speedup Meeting, ETH, Zurich, September 7, 2012
2
3
2D sliceSnapshot
Contours of density perturbations
1G grid (3D)(*)
2G particles (5D)
HELIOS 1.5 PFlops
Turbulence in an ITER plasma
(*) more grid points on this 2D slice than pixels
4
Outline
1. IntroductionFusion, tokamaks and turbulence Scalability issues: size matters
2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates
3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows
4. Conclusion and outlook
5
Outline
1. IntroductionFusion, tokamaks and turbulenceScalability issues: size matters
2. The ways to performance improvementsProfiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates
3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows
4. Conclusion and outlook
6
Magnetic fusionIN: D, Li OUT: He + energy
Abundant fuel, He is neither radioactive nor greenhouse gasLow activation, short period structure materials
D-T mix in the core of the reactorT recycled on site0.3g of T in the core Total site T inventory: 1 – 2 kg
No nuclear chain reactionEven in the case of a maximum conceivable accident no population evacuation is requiredExtremely weak nuclear proliferation risk
No U, no Pu, etc. -- no material in Non Proliferation TreatyResearch was declassified in 1958 -- middle of cold war
7
ITER: the way to fusion
EU+CH, Japan, USA, China, India, South Korea, RussiaO(10G€) construction cost (0.30€ / person / year)
Source:ITER
The plasma will be there (see next p)
You are here
8
Magnetic confinement: tokamak
Larmorradius ρL
linefieldBr
surfacemagnetic drifts curvature and B∇
Trapped particle
Passing particleparticle
trajectory
particles confine toessentialis lines field of transform)
l(rotationatorsionhelicalBv
9
Collisions, turbulenceMagnetic fusion plasmas are very weakly collisional
Temperature ~300’000 x standard atmosphereDensity ~ 1/100’000 x standard atmospherePressure ~ 3 bar
Mean free path (>1km) >> system size (10m)Particles have the time to span a large fraction of the magnetic field configuration before the effect of collisions are felt
Geometry of the magnetic field affects not only single particle motion, but also collective effects like turbulenceKinetic effects are important (i.e. fluid models insufficient)
The main losses (heat flux) are due to turbulencei.e. much larger than collision-induced
Degradation of the quality of confinement large enough system sizes are required for achieving economical fusion
10
Timescales in the ITER plasma
]s[t
machine lifetime 1 shot energy
confinement
turbulence
ion cyclotron
electron cyclotron
810−510−010310810 1210−
Physics spans several orders of magnitudeDirect Numerical Simulation (DNS) of “everything” is unthinkable
Need to separate timescales using approximations
11
Net energy transfer from the wave to the particles if
Collisionless Landau damping
0/ <∂∂ vf
Kinetic effects: wave-particle interaction
Surfers with velocity just below the phase velocity of the wave will be accelerated
-> momentum and energy transfer
Surfers with velocity too different from the phase velocity of the wave will not ride the wave
General: distribution function in 6D phase spaceTo be solved with consistent electromagnetic fields
);,( tvxf rr
f
vk/ω
),(),,( txBtxE rrrr
12
Turbulence and transportFinite system size gradients of T, n, BWhen these gradients exceed threshold values, instabilities develop and saturate to a turbulent state heat, particle, momentum transport
Low frequencies (ω << ωci)Small scales perpendicular to B (~ρs)Large scales parallel to B (~ system size)
Strong anisotropy(~1000 for ITER)
Average out the fast motion of the particle around the guiding centerFast parallel motion, slow perpendicular motion (drifts) of the gyro-centre
phase space dimension reduction 6D ---> 5D
13
Gyrokinetic equations
),,( // μvRfs
rdistribution function of species s in 5D phase space
),( '//
sssss ffC
Rf
dtdv
Rf
dtRd
tf
=∂∂
+∂∂⋅+
∂∂
rr
r
),(fct...),,(fct... // Adt
dvAdtRd rrr
φφ ==
),( Ar
φ solution of Maxwell’s equations, with ρ, j obtained as moments of fs
PDE, 3D
advection-diffusionPDE, 5D
equations of motion(orbits)
ODE, 5D
14
Solving GK equations: ORB5 code
5D phase space: Particle-In-Cell (PIC)3D field solver: cubic B-spline finite elementsTime stepping: Runge-Kutta 4th orderSampling noise reduction measures
Control variates, δf schemeField-aligned Fourier filter: eliminate unphysical modes
Sampling noise control measuresMomentum- and zonal flow-conserving Krook-like operatorCoarse graining scheme
Parallelization schemeDomain decompositionDomain cloning
Developed at CRPPContributions from Max-Planck IPP and U Warwick
15
8/10ITERsize
4/10
Turbulence: system size matters
2/10
1/10ITERsize
16
Size matters
effective system size
heat
tran
spor
t
ORB5: Lagrange-PIC [TM Tran et al] – GENE: Euler [F Jenko et al]
now ITER
17
Scalability with system size
With a PIC code, toroidal coordinates:3D field grid ~ (size)3
Nparticles ~ (size)3
Ntimesteps ~(size)1
Computational cost ~(size)4
With field-aligned Fourier filter:3D field grid ~ (size)3
Nparticles ~ (size)2
Ntimesteps ~(size)0
Computational cost ~(size)2 for local (particle) operations~(size)3 for (grid) communications
Remove that bottleneck limiting // scalability for large nr of procs and large system size
Identify, remove or reduce non-scalable communications
18
Outline
1. IntroductionFusion, tokamaks and turbulenceScalability issues: size matters
2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates
3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows
4. Conclusion and outlook
19
Parallelization scheme – ORB5
Domain decomposition (toroidal direction)
Domain Cloning:field quantities (density, potential) are replicated
shift
globalsum
Npes = Nd*Nc
particles
field
s
Nd = 100 ~ 1000
Nc
= 1
~ 10
0
// Data transpose (Fourier)
20
ORB5 code profiling
In-depth performance analysis by A. Tineo (CSCS) on Cray XE6 (Palu). Scalasca, Tau.Main findings: data locality & communications bottlenecks
CSCS
CRPP
--localField solverFourier+grid
grid
grid
domains
clones
global
global
// data transposeradial reassembly
grid to/from Fourier
-grid
-clones
localglobal
assignsum
particles to/from grid
-particles
-domains
localglobal
pushshift
particles
21
Load imbalance for global sums over the clones
ORB5 code profiling
3d system topology of processors (Palu 2048 procs)
Training of CRPP staff to use profiling toolsCSCS
CRPP
4 clones of the same grid domain
22
Kernel extraction
Bottlenecks identified in some global communications Essentially non-scalable with number of clonesExtract subset of routines from ORB5 that deal with these operations: essentially 3D grid data communications
CSCS
CRPP
At CSCS: Adrian Tineo, Gilles Fourestey, Neil Stringfellow
23
Kernel performance improvement
Transposition of cartesian MPI communicator put clones on nodeReplace all_reduce with all_gather in // data transpose operations (radial domain decomposition)Kernel performance improved by a factor of 5 (for 24 clones)Kernel scalability with clones drastically improvred
Cray XE6 Palu160 domains3-24 clones(960x1980x160) grid
old
new
24
Improvements implemented in ORB5 (1)
Strong scalability with clones is improved from 72% to 90%(32768 procs) for a grid (512x1024x1024), #particles=6G
Cray XE6 (Monte Rosa)Clones 8-32
25
For a 4 x larger grid (1024x2048x1024) the improvement is a factor of 2 at 32768 procs. Case relevant for ITER-size simulation of ITG turbulence
Improvements implemented in ORB5 (2)
Cray XE6 (Monte Rosa)Clones 324x larger grid
26
Petascale Computing
Improvements brought in the frame of the HP2C project helped us to get access to a 1.5 PetaFlop platform (HELIOS, IFERC-CSC, Rokkasho, Japan) dedicated to magnetic fusion research (EU+CH & Japan). [#12 on June 2012 top500]ORB5 was one among 4 codes designated as high-level benchmark used for the acceptance tests of HELIOS (capability testing)Demonstrated strong scalability from 16k to 64k cores: speedup 3.36 (84% parallel efficiency)Nominated as one of 4 “LightHouse Projects” with several million cpu-hours allocation grantedAllowed us to perform the first global gyrokinetic turbulence simulation of ITER
27
Outline
1. IntroductionFusion, tokamaks and turbulence Scalability issues: size matters
2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates
3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows
4. Conclusion and outlook
28
ITG turbulence in ITER
29
30
31
Zonal Flows and turbulence
Turbulence zonal flows (ZF) shearing turbulent eddies turbulence suppression
Self-organizationRadial structure of alternating bands of ZFs, related to regions of suppressed turbulence
ITER plasma shapingmore effective turbulence suppression by ZFs than circular shaped plasmas(see next slide)
32
Geometry, Zonal Flows and Turbulence
ITERcircular
33
Zonal flow dynamics - ITER
Contours of ZF shearing rate vs radius (s) and time
edge
core
34
Zonal flow dynamics - circular
Contours of ZF shearing rate vs radius (s) and time
core
edge
35
Turbulent heat transport
Drastic reduction of turbulent ion heat transport with the ITER shape
36
Outline
1. IntroductionFusion, tokamaks and turbulence Scalability issues: size matters
2. The ways to performance improvements (HP2C)Profiling, Kernel extraction, Kernel optimization, Code implementation, Performance testsMatch the numerical scheme to the physics: geometry, anisotropy, choice of coordinates
3. First global gyrokinetic turbulence simulations of ITERMain finding: ITER plasma shaping favours turbulence suppression by increasing coupling to zonal flows
4. Conclusion and outlook
37
Conclusion and outlook
First principles based “direct” numerical simulations of turbulence in magnetized plasmas remain a challenge – and impact fusion researchOur codes are ready for PetaFlop range platforms. The HP2C project was instrumental in bringing us to this important stepPresent and future and works: I/O, data analysis & visualization (field and particle data), …First global gyrokinetic turbulence simulations of ITER have predict a drastic reduction of turbulent heat transport due to plasma shaping – to be confirmed with more complete physics