Very Large Scale Computing In Accelerator Physics

Robert D. RyneLos Alamos National Laboratory

Robert Ryne 2

…with contributions from members of

Grand Challenge in Computational Accelerator Physics

Advanced Computing for 21st Century Accelerator Science and Technology project

Robert Ryne 3

Outline

Importance of Accelerators Future of Accelerators Importance of Accelerator Simulation Past Accomplishments:

Grand Challenge in Computational Accelerator Physics– electromagnetics

– beam dynamics

– applications beyond accelerator physics

Future Plans Advanced Computing for 21st Century Accelerator S&T

Robert Ryne 4

Accelerators have enabled some of the greatest discoveries of the 20th century

“Extraordinary tools for extraordinary science” high energy physics nuclear physics materials science biological science

Robert Ryne 5

Accelerator Technology BenefitsScience, Technology, and Society

electron microscopy beam lithography ion implantation accelerator mass spectrometry medical isotope production medical irradiation therapy

Robert Ryne 6

Accelerators have been proposed to address issues of international importance

Accelerator transmutation of waste Accelerator production of tritium Accelerators for proton radiography Accelerator-driven energy production

Accelerators are key tools for solving problems related to energy, national security, and quality of the environment

Robert Ryne 7

Future of Accelerators: Two Questions

What will be the next major machine beyond LHC? linear collider -factory/ -collider rare isotope accelerator 4th generation light source

Can we develop a new path to the high-energy frontier? Plasma/Laser systems may hold the key

Example: Comparison of Stanford Linear

Collider and Next Linear Collider

Possible Layout of a Neutrino Factory

Robert Ryne 10

Importance of Accelerator Simulation

Next generation of accelerators will involve: higher intensity, higher energy greater complexity increased collective effects

Large-scale simulations essential for design decisions & feasibility studies:

– evaluate/reduce risk, reduce cost, optimize performance

accelerator science and technology advancement

Robert Ryne 11

Cost Impacts

Without large-scale simulation: cost escalation SSC: 1 cm increase in aperture due to lack of

confidence in design resulted in $1B cost increase

With large-scale simulation: cost savings NLC: Large-scale electromagnetic simulations

have led to $100M cost reduction

Robert Ryne 12

DOE Grand Challenge In Computational Accelerator Physics (1997-2000)

Goal - “to develop a new generation of accelerator modeling tools on High Performance Computing (HPC) platforms and to apply them to present and future accelerator applications of national importance.”

Beam Dynamics:LANL (S. Habib, J. Qiang, R. Ryne)UCLA (V. Decyk)

Electromagnetics:SLAC (N. Folwell, Z. Li, V. Ivanov, K. Ko, J. Malone, B. McCandless, C.-K. Ng, R. Richardson, G. Schussman, M. Wolf)Stanford/SCCM (T. Afzal, B. Chan, G. Golub, W. Mi, Y. Sun, R. Yu)

Computer Science & Computing Resources - NERSC & ACL

Robert Ryne 13

New parallel applications codes have been applied to several major accelerator projects

Main deliverables: 4 parallel applications codes Electromagnetics:

3D parallel eigenmode code Omega3P 3D parallel time-domain EM code Tau3P

Beam Dynamics: 3D parallel Poisson/Vlasov code, IMPACT 3D parallel Fokker/Planck code, LANGEVIN3D

Applied to SNS, NLC, PEP-II, APT, ALS, CERN/SPL

New capability has enabled simulations 3-4 orders of magnitude greater than previously possible

Robert Ryne 14

Parallel Electromagnetic Field Solvers: Features

C++ implementation w/ MPI Reuse of existing parallel libraries (ParMetis, AZTEC) Unstructured grids for conformal meshes New solvers for fast convergence and scalability Adaptive refinement to improve accuracy & performance Omega3P: 3D finite element w/ linear & quadratic basis

functions Tau3P: unstructured Yee grid

Robert Ryne 15

Why is Large-Scale Modeling Needed? Example: NLC Rounded Damped Detuned Structure (RDDS) Design

highly three-dimensional structure detuning+damping manifold for wakefield

suppression require 0.01% accuracy in accelerating

frequency to maintain efficiency simulation mesh size close to fabrication

tolerance (order of microns) available 3D codes on desktop computers

cannot deliver required accuracy, resolution

Robert Ryne 16

11.424

11.4242

11.4244

11.4246

11.4248

11.425

11.4252

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14

Fre

qu

ency

(G

Hz)

2.9 M D.O.F.

1.8 M D.O.F.

0.38 M D.O.F.

NLC - RDDS Cell Design (Omega3P)

Accelerating Mode

Frequency accuracy to 1 part in 10,000 is achieved

1 MHz

h4

Robert Ryne 17

+0.41 MHz

+13.39 MHz

+4.86 MHz+1.05 MHz

+0.42 MHz

+0.35 MHz

+0.23 MHz

+1.12 MHz

+2.60 MHz

-2.96 MHz

+0.42 MHz

+0.55 MHz

+0.52 MHz

+0.41 MHz

+0.14 MHz

NLC - RDDS 6 Cell Section (Omega3P)

Robert Ryne 18

NLC - RDDS Output End (Tau3P)

Robert Ryne 19

PEP II, SNS, and APT Cavity Design (Omega3P)

Robert Ryne 20

refined mesh size: 5 mm 2.5 mm 1.5mm # elements : 23390 43555 106699 degrees of freedom: 142914 262162 642759 peak power density: 1.2811 MW/m2 1.3909 MW/m2 1.3959 MW/m2

Peak Wall Loss in PEP-II Waveguide-Damped RF cavity

Omega3P - Mesh Refinement

Robert Ryne 21

Parallel Beam Dynamics Codes: Features

split-operator-based 3D parallel particle-in-cell canonical variables variety of implementations (F90/MPI, C++, POOMA, HPF) particle manager, field manager, dynamic load balancing 6 types of boundary conditions for field solvers:

open/circular/rectangular transverse; open/periodic longitudinal

reference trajectory + transfer maps computed “on the fly” philosophy:

do not take tiny steps to push particles do take tiny steps to compute maps; then push particles w/ maps

LANGEVIN3D: self-consistent damping/diffusion coefficients

Robert Ryne 22

Why is Large-Scale Modeling Needed? Example: Modeling Beam Halo in High Intensity Linacs

Future high-intensity machines will have to operate with ultra-low losses

A major source of loss: low density, large amplitude halo

Large scale simulations (~100M particles) needed to predict halo

Maximum beam size does not converge in small-scale PC

simulation (up to 1M particles)

0 100 200 300 400 500 600 700 800 900 1000 1100

Ws (MeV)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Bea

m S

ize

(cm

)

SNS CCDTL/CCLWith Errors, No Mismatch

1,000,000 100,00010,0001,000

Maximum Extent

RMS

Robert Ryne 23

Mismatched Induced Beam Halo

Matched beam.x-y cross-section

Mismatched beam.x-y cross-section

Robert Ryne 24

Vlasov Code or PIC code?

Direct Vlasov: bad: very large memory bad: subgrid scale effects good: no sampling noise good: no collisionality

Particle-based: good: low memory good: subgrid resolution OK bad: statistical fluctuations bad: numerical collisionality

Robert Ryne 25

How to turn any magnetic optics code into a tracking code with space charge

Split-Operator Methods

H=Hext H=Hsc

M=MextM=Msc

H=Hext+Hsc

M(t)= Mext(t/2) Msc(t) Mext(t/2) + O(t3)

MagneticOptics

Multi-ParticleSimulation

(arbitrary order possible via Yoshida)

Robert Ryne 26

Development of IMPACT has Enabled the Largest, Most Detailed Linac Simulations ever Performed

Model of SNS linac used 400 accelerating structuresSimulations run w/ up to 800M particles on a 5123 gridApproaching real-world # of particles (900M for SNS)

100M particle runs now routine (5-10 hrs on 256 PEs)Analogous 1M particle simulation using legacy 2D

code on a PC requires weekend 3 order-of-magnitude increase in simulation capability

100x larger simulations performed in 1/10 the time

Robert Ryne 27

Comparison: Old vs. New Capability

1980s: 10K particle, 2D serial simulations typical Early 1990s: 10K-100K particle, 2D serial simulations typical 2000: 100M particle runs routine (5-10 hrs on 256 PEs); more

realistic treatment of beamline elements

SNS linac; 500M particlesLEDA halo expt; 100M particles

Robert Ryne 28

Intense Beams in Circular Accelerators

Previous work emphasized high intensity linear accelerators

New work treats intense beams in bending magnets

Issue: vast majority of accelerator codes use arc length (“z” or “s”) as the independent variable.

Simulation of intense beams requires solving 2= at fixed time

The split-operator approach treated in linear and circular systems will soon make it possible to “flip a switch” to turn

space charge on/off in the major accelerator codes

x-z plot based on x- data from an s-codeplotted at 8 different times

Robert Ryne 29

Collaboration/impact beyond accelerator physics

Modeling collisions in plasmas new Fokker/Planck code

Modeling astrophysical systems starting w/ IMPACT, developing astrophysical PIC code also a testbed for testing scripting ideas

Modeling stochastic dynamical systems new leap-frog integrator for systems w/ multiplicative noise

Simulations requiring solution of large eigensystems new eigensolver developed by SLAC/NMG & Stanford SCCM

Modeling quantum systems Spectral and DeRaedt-style codes to solve the Schrodinger,

density matrix, and Wigner-function equations

Robert Ryne 30

First-Ever Self-Consistent Fokker/Planck

Self-consistent Langevin-Fokker/Planck requires the analog of thousands of space charge calculations per time step “…clearly such calculations are impossible….” NOT! DEMONSTRATED, thanks to modern parallel machines and intelligent

algorithms

Diffusion Coefficients Friction Coefficient / velocity

Robert Ryne 31

Schrodinger Solver: Two Approaches

).0,(),(2

22

2

2)

2(

2)

2(

xeeex mVm

.2

22

tiV

m

.2

V

i

dx

d

dx

d

m

iH

N

jjjj

N

j

jjjj Vi

m

iH

112

11 ,))((

2

Spectral:

Field Theoretic:

Discrete:

FFTs; global communication

Nearest-neighbor communication

Robert Ryne 32

Conclusion“Advanced Computing for 21st Century Accelerator Sci. & Tech.”

Builds on foundation laid by Accelerator Grand Challenge Larger collaboration:

presently LANL, SLAC, FNAL, LBNL, BNL, JLab, Stanford, UCLA

Project Goal: develop a comprehensive, coherent accelerator simulation environment

Focus Areas: Beam Systems Simulation, Electromagnetic Systems Simulation,

Beam/Electromagnetic Systems Integration

View toward near-term impact on: NLC, -factory (driver, muon cooling), laser/plasma accelerators

Robert Ryne 33

Acknowledgement

Work supported by the DOE Office of Science Office of Advanced Scientific Computing Research, Division

of Mathematical, Information, and Computational Sciences Office of High Energy and Nuclear Physics Division of High Energy Physics, Los Alamos Accelerator

Code Group