Upload
miles-strickland
View
218
Download
2
Tags:
Embed Size (px)
Citation preview
Tutorial: Computation for Advanced Accelerators
Thomas M. Antonsen Jr,
IREAP, University of Maryland
College Park MD, 20742
AAC 2012Work supported by NSF, and DOE
Computation has become integral to Accelerator Science
Why is Computation Important ?
- Understanding of basic physical processes- Understanding and diagnosing particular experiments- Designing improved experiments- Optimizing designs
Areas of Progress
Improved Physics Models
Find better descriptions of what’s going on inside a an accelerator
More Efficient Algorithms
Solve the governing equations with fewer operations and/or less memory usage.
New Computing Architectures
Restructure your code to take advantage of new hardware.
Examples of Areas where Computation is Important
Particle trajectories in analytic fields*Space charge fields* Magnetic fields due to currents in coils
RF fields in cavities and waveguidesRadiation generated by energetic particles Multipactor
Plasma-Based Accelerationand more
*See R. Ryne http://uspas.fnal.gov/materials/09UNM/ComputationalMethods.pdf
Meshes and Grids
Unstructured mesh
Structured mesh
Calculations are carried out on digital computersTime is discretized
Space may also be discretized - meshes or grids
If field is a dynamical variable (Maxwell’s Equations) a mesh is needed
If field describes pair-wise interaction forces (Coulombs law) it can be calculated with or without a grid.
Tree Code Calculation of Coulomb interaction without grids J. Barnes and P. Hut, “A hierarchical force-calculation algorithm,” Nature, vol. 324,
pp. 446–449, 1986
Direct application of Coulomb’s law to N particles requires O(N2) operations. Not practical for large N.
Tree code: Divide domain into cellsEach cell has fewer than NL particles
Particles within same cell interact pairwise
Particles in different cells interact through multipole moments
O(NlogN) operations possible
Mesh Types
Structured Mesh
Complex geometries may be represented using a multi-block representation:
Unstructured Mesh
conformal meshes: boundary of mesh conforms with physical boundary
Existing Discretization Techniques
Finite Integration TechniqueApplicable to orthogonal Cartesian meshDegrees of freedom are integrals of fields over appropriate mesh elements
Suffers due to non-conformal representation of geometry
Finite Element MethodApplicable to curvilinear mesh(structured or unstructured)
Field described by interpolating basis functions in each cell
Solves integral (weak) form of 2nd order differential equations
Evaluates matrix elements using overlap integrals of basis & test functions
Large number of coupling terms
Basis & test functions
1
-½ 0 ½
Fields and Structured Mesh for FDTD solution of Maxwell’s Equations
Electric fields defined on edgesMagnetic fields defined on faces Electric fields defined at integer time stepsMagnetic fields defined at half integer time steps
• Widely used• Simple (explicit)• Second order accurate• Stable if t <cdx
Yee Mesh(1966)
Leap Frog Explicit FDTDSolutions to Maxwell’s Equation
time differencing
Alternately apply Faraday’s law to faces
Explicit: knowing E at time t allows for direct calculation of H at time t+t/2
Finite Differencing Plane Waves
Assume all fields vary as
Dispersion relation for Plane waves
ContinuousFDTD
Stability - Courant Freidricks Lewy Condition (CFL)
Left side < Maximum of Right Side
Stable if
CFL Condition
Dispersion Relation
Dispersion and Accuracy
Dispersion in discrete system is periodic.
Departures from continuous system are quadratic in t, .
Waves have phase velocity <c
Numerical Cerenkov Radiation - Requires filter
More complicated evaluation of curl
Conducting Boundaries on Structured Grids
When boundary does not coincide with mesh FDTD must be modified.
Without modification errors of order are introduced.
Boundary
S. Dey, R. Mittra, A locally conformal finite-difference time-domain (FDTD) algorithm modeling modeling three-dimensional perfectly conducting objects, IEEE Microwave Guided Wave Lett. 7 (1997) 273
Problem: If cells are small CFL condition is violated
Separate modification to compute surface losses.
ADI (Alternating Dimension Implicit) Solutions to Maxwell’s Equation)
• Split time steps• Recently developed• Complex (implicit)• Not subject to Courant condition, t > cdx
time differencing
Second half of time step has x, y, and z permuted
Scalar: Douglas, Peaceman, and Ratchford (1955)
ME: Namiki T (1999), Zheng, Chen, Zhang (2000)
Leapfrog ADI-FDTD equationsS. Cooke et al (2009)
Electric and magnetic field terms are offset in timeElectric fields only appear on the half time stepsMagnetic fields only appear on the full time steps
Still need to solve six tri-diagonal equations per time step3 for the E-components, 3 for the H-componentsADI-FDTD uses 6 for the E-components (3 components, 2 half steps)
No additional explicit equations are necessary
Tri-diagonal operators “curl” terms
PIC Algorithm - From Plasma Physics via Computer Simulation C. K. Birdsall and A. B. Langdon
FDTD ME in here
Integration of Equations of Motion
Basic Leap Frog:
Modified for Lorenz Force:“Boris Push”
Half step with E
Full step with B
Half step with E
p and x known on staggered time grid
Interpolation of Fields and Accumulation of Currents and Charges
Forces interpolated from grid to location of particles
Particle contribution to current density accumulated on grid. Algorithm should conserve charge.
E and J are known on the same grid, but are staggered in time
J.Villasenor and O.Buneman. Computer Physics Communications, 69. 306-316, (1992)
Charge is not computed,only current is used.
Space charge fields occur only through integration of Maxwell’s equations.
Charge-conserving methodsdo this exactly – no need fordivergence-cleaning.
Advance H
Assign external current sources
Add new particles
Advance P, then X(accumulating J)
Remove dead particles
Advance E
HPEXJ
IVEC 2012 21
FDTD-PIC Time-stepping loop S. Cooke NRL
Plasma Based Accelerators
• Direct Acceleration in Modulated Channels
Physical Processes
Important
• Relativistic Motion
• Plasma Wave Generation / Cavitation
• Laser Self Focusing / Scattering
• Ionization
Not Important
• Turbulence - most plasma particles interact for a short time, several plasma periods, then leave
Our job is “relatively” easy.
Simulation has played an important role in the development of the field.
Guiding our understanding Designing new experiments
Relevant Time Scales (LWFA)
• Laser Period =800 nm TL = 2.7 fs
• Driver Duration ~ Plasma Period TD = 50 fs
• Propagation Time (driver evolution, L= 1 cm) TP = 3.3 104 fs
(driver evolution, L= 1 m) TP = 3.3 106 fs
• Disparity in time scales leads to both complications and simplifications
Propagation Time >> Driver Duration >> Laser Period
TP >> TD >> TL
Propagation Time >> Driver Duration >> Laser Period
TP >> TD >> TL
MODELS APPROACHES / APPROXIMATIONS
• LaserFull EM - Laser Envelope
• Plasma
Particles - Fluid
Full Lorenz force - Ponderomotive
Dynamic response - Quasi-static
Hierarchy of Descriptions
1. Full Format Particle in Cell (PIC)
- Relativistic equations of motion for macro-particles- Maxwell’s Equations on a grid- Most accurate but most computationally intensive
• LaserFull EM - Laser Envelope
• Plasma
Particles - Fluid
Full Lorenz force - Ponderomotive
Dynamic response - Quasi-static
Windows and Frames
z=0 z=L
t=L/cLab Frame - # Grids
NLF = (L/dz)2 ~ (L/)2
Moving Window - # Grids
NMW = (LD/L) NLF
grid must resolve
Three versions:- Lab Frame- Moving Window- Boosted Frame
- LD Propagation distancePulse length
dt=dz/c
Boosted FrameJ-L Vay, PRL98, 130405 (2007)
z=0 z=L
t=L/c
Boosted frame - # grids NBF = -2 NMW
Optimum based on 1Dmakes simulation “square”
Boost -
z'=0z' =-LD
t' =L/c
dz' = dz
Qualifications:-no backscatter. In boosted frame gives upshift. Requires smaller dz'-transverse grid size sets limit on dt' < dx' /c= dx /c
Lab Framedt' = dt
Reported speedups limited by various factors: • laser transverse size at injection, • statistics (trapped injection),• short wavelength instability (most severe).
Osiris: trapped injection
Vorpal: external injection w/ beam loading
Warp: external injection wo/ beam loading
Using conventional PIC techniques, 2-3 orders of magnitude speedup reported in 2D/3D by various groups
J-L Vay
2. Full EM vs. Laser Envelope Driver Duration >> Laser Period
TD >> TL
• Required Approximation for Laser envelope:
laser pulse >> 1, rspot >>
p /laser <<1
• Advantages of envelope model:
-Larger time steps
Full EM stability: t < x/c
Envelope accuracy: t < 2 x2/c
- No unphysical Cherenkov radiation
- Further approximations
• Advantages of full EM: Includes Stimulated Raman back-scattering
Also direct acceleration
Laser Envelope Approximation
• Laser Frame Coordinate: = ct – z
• Laser + Wake field: E = E laser + E wake
• Vector Potential: Alaser = A 0(,x,t) exp ik 0 + c.c.
• Envelope equation:
Necessary for: Raman ForwardSelf phase modulationvg< c
Drop(eliminates Raman back-scatter)
Validity of Envelope Equation
Extended Paraxial approximation- Correct treatment of forward and near forward scattered radiation - Does not treat backscattered radiation- If grid is dense enough can treat ~
Extended Paraxial– v g() = c / 1 +
k2c2 + p
2
2 2
True: v g() = c 1 –
k2 c2 + p
2
2
1 / 2
k2c2 , p
2 < < 2 Requires :
3. Full Lorenz Force vs. Ponderomotive Description
• Ponderomotive Equations
dpdt
= q Ewake +v Bwake
c + Fp
Fp = – mc 2
2
q A laser
mc 2
2
= 1 +
p 2
m2c2 +q A laser
mc 2
2
dpi
dt= q E +
vi Bc
dxi
dt= vi
• Full Lorenz:
E = Elaser + Ewake
x(t) = x(t) + x(t)
• Separation of time scales
x(t) Elaser < < Elaser
• Requires small excursion
Ponderomotive Guiding Center PIC Code: TurboWAVE
Fields are separated into high and low frequency components. The low frequency component is treated as in an ordinary PIC code. The high frequency component is treated using a reduced description which averages over optical cycles.
Deposit Sources j and
Advance Fields(Maxwell’s Equations)
Lorentz Push
Deposit Sourcenq2/<m>
Advance Laser(Envelope Equation)
Ponderomotive Push
+
Low Frequency Cycle High Frequency Cycle
D.F. Gordon, et al. , IEEE TRANSACTIONS ON PLASMA SCIENCE , V 28 , 1224-1232 ( 2000 )
35
INF&RNO(INtegrated Fluid & paRticle simulatioN cOde)
C. Benedetti at al. (LBNL)
2D cylindrical + envelope for the laser (ponderomotive approximation) full PIC/fluid description for plasma particle (quasi-static approx. is also available) switching between PIC/fluid modalities (hybrid PIC-fluid sim. are possible) dynamical particle resampling to reduce on-axis noise 2nd order upwind/centered FD schemes + RK2/RK4 (& Implicit) for time integration linear/quadratic shape functions for force interpolation/charge deposition high order low-pass compact filter for current/field smoothing “BELLA”-like runs (10GeV in ~ 1m) become feasible in a few days on small machines
FLUID PIC
4. Quasi - Static vs. Dynamic WakeP. Sprangle, E. Esarey and A. Ting, Phys. Rev. Lett. 64,
2011 (1990)
ddt
= t
+ c – vz
+ v
Laser Pulse PlasmaWake
Plasma electron
c
Trapped electron
c - vz
Electron transit time:
e = pulse
1 – vz / c
Electron transit time << Pulse modification time
Advantages: fewer particles, less noise (particles marched in ct-z)
Disadvantages: particles are not trapped
Quasi Static Simulation Code WAKE P. Mora and T. M. Antonsen Jr. - Phys Plasma 4, 217 (1997)
Particle trajectories
r
=ct-z
Two Time Scales1. “fast time” t ~ TD ~ p
-1
particle trajectories and wake fields determined
2. “slow time”
laser pulse evolvesdiffractionself-focusingdepletion
QuickPIC: 3D quasi-static particle code
New Features• Particle tracking• Pipelining• Parallel scaling to 1,000+
processors• Beta version of enhanced
pipelining algorithm: enables scaling to 10,000+ processors and unprecedented simulation resolution down to nm
Description
• Massivelly Parallel, 3D Quasi-static particle-in-cell code
• Ponderomotive guiding center for laser driver
• 100-1000+ savings with high fidelity• Field ionization and radiation reaction
included• Simplified version used for e-cloud
modeling• Developed by UCLA + UMaryland + IST
Examples of applications• Simulations for PWFA experiments,
E157/162/164/164X/167 (Including Feb. 2007 Nature)
• Study of electron cloud effect in LHC.• Plasma afterburner design up to TeV• Efficient simulation of externally injected
LWFA• Beam loading studies using laser/beam
drivers
Chengkun Huang: [email protected]://exodus.physics.ucla.edu/http://cfp.ist.utl.pt/golp/epp
QUASI-STATIC CODE STRUCTURE
Laser
Particles and Wake
“slow time - t”
“fast time =ct-z”
PARTICLES CONTINUEDconstant of motion
• Hamiltonian::
• Algebraic equation:
• Weak dependence on “t” in the laser frame
• Introduce potentialsTransverse Dynamics
LOCAL FREQUENCY MODIFICATIONM. Tsoufras, PhD Thesis, UCLA
neIntensity
Frequency shift is proportional to propagation distance
Relative frequency shift is unity at the dephasing distance
Assume complete cavitation
Quasi Static Modeling of Pulse DepletionW. Zhu Wednesday WG1
Comparison of Spectra; Modified Paraxial, Full-wave, and FDTD-PIC
Conclusions
• Numerical simulation of Laser-Plasma interactions is a powerful tool
• A variety of models and algorithms existfirst principlesreduced modles
• Field is still advancing with new developmentsBoosted Frame Calculations speed-up ~ 2
3D Parallel Quasi-static speed-up ~ [p2
GPUs
WAKE - CavitationP. Mora and T. Antonsen PHYSICAL REVIEW E, Volume: 53 R2068
(1996)
Intensity Density Trajectories
Complete cavitationSuppression of Raman instability Stable propagation for 30 Rayleigh lengths
Wake simulation of pulse Wake simulation of pulse propagation in corrugated plasma channelspropagation in corrugated plasma channelsSee WG #1 B. Layer Wed. PM, J. Palastro Thu. AM
150 m
3 cm ~ 90 corrugations ~ 40 TR
t=0 TR t=15 TR t=30 TRUpper: corrugatedLower: smooth
Suppression of Raman Instability
density
intensity
=800 nmwch=15 mao=.35t=1 ps
no=7x1018 cm-3
=.9m=.035 cm
QUICKPIC
Quasi-Static Plasma Particles
Dynamic Beam Particles
Wake Fields
Laser Field
Axial (dt) Axial (dt)Transverse (dt) Transverse (dt)
LaserAdvance
Axial (dt/2)
push particlespush particlesPlasma Particle Advance
time-st t+dt
Second Order Accurate Split-Step Scheme
time-tt t+dt
e- driver e+ driver
e- driver with ionization laser driver
Verification : Full PIC vs. Quasi-static PICBenchmark for different drivers:
QuickPIC vs. Full PIC
100 to 10000 CPU savings with “no” loss in accuracy
Excellent agreement with full PIC.
100 to 10000 times savings in CPU needs
No noise issues and no unphysical Cerenkov radiation
Iteration of Electromagnetic Field
Parallel electric field
Transverse electric field
Electromagnetic portion
Ampere’s law
Iterate to find
Equation of motion
Quasi-Static Field Representation
Lorenz
Pro:
Simple structure
Compatible with 2D PIC
Con:
A carries “electrostatic” field
Transverse Coulomb
Pro:
A = 0 in electrostatic limit
Con:
non-standard field equations
Particle Promotion in WAKES. Morshed PoP, to be published
Osiris
Wake
“Plasma particles” for which quasi-static violated are promoted to “beam particles”
54
Combine a half step forward with a half step back:
Subtracting equations gives a new scheme:
New “Leapfrog” ADI-FDTD Formulation
Tri-diagonal operator “curl H” term
Compare to FDTD:
S.J.Cooke, M.Botton, T.M.Antonsen, Jr., B.Levush, “A leapfrog formulation of the 3-D ADI-FDTD algorithm” Int. J. Numer. Model. 2009; 22:187–200
Simple source terms
TurboWAVE Framework
TurboWAVE BaseGrid geometry, regions, domain decomposition, diagnostics, input/output…
Numerical FrameworkLinear algebra, elliptical solvers, fluid advection…
PICEM pusher,
ponderomotive pusher, gather/scatter
“SPARC”Multi-species
hydrodynamics
Nonlinear Optics
Anharmonic Lorentz model, Quasistatic modelField Solver Modules
Explicit, envelope, direct fields, coulomb gauge, electrostatic.
Lindman boundaries, simple conducting regions, PML media.
ChemistryArbitrary species,
reactions
NRL Plasma Physics Division
Images from TurboWAVE
Gridded Gun ModelingContours are equipotential lines.Colors represent current density.Black circles represent grid wires.
Blowout WakefieldRed = ion richBlue = electron rich
ve
k0
Utilizes 2D electrostatic PIC (slab or cylindrical) with conducting regions
Utilizes 3D electromagnetic PIC
NRL Plasma Physics Division
Speedup of 50,000 shown for meter-scale parameters
Rigorously tested, showing second-order convergence and correct laser group velocity
Benchmarks against scaled simulations show excellent agreement
Envelope model in VORPAL enables full 3D simulations of meter-scale LPA stages
[1] B. Cowan et al., Proc. AAC 2008
[2] P. Messmer and D. Bruhwiler, PRST-AB 9, 031302 (2006)
[3] D. Gordon, IEEE Trans. Plasma Sci. 35, 1486 (2007)
[4] P. Mora and T. M. Antonsen, Phys. Plasmas 4, 217 (1997)
Ben Cowan
Charge Conserving Algorithm
Full Format: PIC Algorithm
VORPAL
OSIRIS 2.0
New Features in v2.0• Bessel Beams • Binary Collision Module• Tunnel (ADK) and Impact
Ionization• Dynamic Load Balancing• PML absorbing BC• Optimized Higher Order
Splines• Parallel I/O (HDF5)• Boosted Frame in 1/2/3D
osiris framework• Massivelly Parallel, Fully Relativistic
Particle-in-Cell (PIC) Code • Visualization and Data Analysis
Infrastructure• Developed by the osiris.consortium:
UCLA + IST• Widely used: UCLA, SLAC, USC,
Michigan, Rochester, IST, Imperial College, Max Planck Inst.
• Examples of applications• Mangles et al., Nature 431 529 (2004).• Tsung et al., Phys. Rev. Lett., 94
185002 (2004)• Mangles et al., Phys. Rev. Lett., 96,
215001 (2006)• Lu et al., Phys. Rev. ST: AB, 10, 061301
(2007)• I.A. Blumenfeld et al., Nature 445, 241
(2007)
Frank Tsung: [email protected] Fonseca: [email protected]://exodus.physics.ucla.eduhttp://cfp.ist.utl.pt/golp/epp
~300x fasterthan lab
simulation
+10GeV self-injection in nonlinear regimeControlled self-guided a0=5.8
Laser pulse
Laser pulse
Injected electrons
Injected electrons
Smooth accelerating
field
Smooth accelerating
field
Boosted frame7000x256x256
cells ~109 particles
3x104 timestepsΥ=10
7-12 GeV1-2 nC
7-12 GeV1-2 nC
Samuel F. Martins et al. Nature Physics V6, April 2010Nature Physics V6, April 2010