The use of ACE3P for the design and analysis HOM’s in SCRF ... · analysis HOM’s in SCRF accelerating structures. ... commercially available codes such as HFSS, CST etc) ... between

I.Shinton, HOMSC, Cockcroft Institute, Daresbury Laboratory, UK, 25th June 2012

1

The use of ACE3P for the design and analysis HOM’s in SCRF accelerating

structures

Dr Ian Shinton Honouree associate of the University of Manchester UK

HOMSC 2012

http://www.manchester.ac.uk/

I.Shinton, HOMSC, Cockcroft Institute, Daresbury Laboratory, UK, 25th June 2012 2

Overview of the presentation – the essential aspects

1) Overview of ACE3P – what can it do? 2) CUBIT

3) ACDTOOL

4) NERSC and ACE3P scripts

5) Important aspects – tips and tricks of using ACE3P for RF cavities (What you really need to know....)


ACE3P ACE3P is a large scale parallel EM suite of codes that have been developed at SLAC over a period of 20 years by SLACS ACD group (advance computing department)- Zenghai Li et al: http://www.slac.stanford.edu/grp/acd/ace3p.html ACE3P can solve a variety of problem types including eigen mode analysis (OMEGA3P), Scattering matrix analysis (S3P), Transient beam solver (T3P) and many other cases. It is a highly modular code, and very powerful. It currently hold the record for the largest full scale EM numerical problem, which was 3 TESLA cryomodules.

The ACE3P packages

Meshing - CUBIT for building CAD models and generating finite-element meshes. http://cubit.sandia.gov. Modeling and Simulation – SLAC’s suite of conformal, higher-order, C++/MPI based parallel finite-element electromagnetic codes https://slacportal.slac.stanford.edu/sites/ard_public/bpd/acd/Pages/Default.aspx Postprocessing - ParaView to visualize unstructured meshes & particle/field data. http://www.paraview.org/.

ACE3P (Advanced Computational Electromagnetics 3P) Frequency Domain: Omega3P – Eigensolver (damping) S3P – S-Parameter

Time Domain: T3P – Wakefields and Transients Particle Tracking: Track3P – Multipacting and Dark Current EM Particle-in-cell: Pic3P – RF gun (self-consistent) Multiphysics: TEM3P – Thermal, RF and Structural

4 I.Shinton, HOMSC, Cockcroft Institute, Daresbury Laboratory, UK, 25th June 2012

http://cubit.sandia.gov/

http://www.paraview.org/

Reference: End-to-End Electromagnetic Simulation of ILC Cryomodule Investigator(s): L.-Q. Lee, K. Ko, Z. Li, C.-K. Ng, L. Xiao, SLAC; E. Ng, LBNL

ACE3P: OMEGA3P SCRF simulations

Omega3P was applied to end to end simulation of the TESLA superconductive modules at FLASH. Using the cavity parameters as inputs the deformed cavity shapes were recovered by solving the inverse problem through an optimization method. This same method was later used to identify the BBU in CEBAF



ACE3P: S3P SCRF simulations

3 3.5 4 4.5 5 5.5 6 6.5 7-800

-700

-600

-500

-400

-300

-200

-100

0

ω/2π: GHz

S21

DB

c1p1,c1p2c1p1,c2p1c1p1,c2p2c1p1,c3p1c1p1,c3p2c1p1,c4p1c1p1,c4p2

C1 C2 C3 C4

p1 p2 p1 p2 p1 p2 p1 p2

9.061GHz

9.062GHz

Trapped modes located in the 5th dipole band

3 4 5 6 7 8 9 100

20

40

60

80

100

ω/2π: GHz

R/Q

: Ω/c

avity

7 7.5 80

2

4

6

ω/2π: GHz

R/Q

: Ω/c

avity

ACE3P: T3P SCRF simulations T3P was applied to end to end simulation of the TESLA superconductive modules for the proposed ILC to check on the damping of the HOM’s


Calculations by X.Ling

ACE3P: TRACK3P & PIC3P SCRF simulations

Left, Geometry of a coupler design generated by rescaling of a TESLA-style coupler. Right, Geometry of the alternative design. The regions indicated relate to simulation domains used in their analysis. For proposed ESS coupler

Impact energy of each electron vs the gradient of the accelerating field within the cavity.

Impact energy of each electron vs the gradient of the accelerating field within the cavity

“Enhanced counter function” for alternative “Hook & Plate” design

Reference: MULTIPACTING ANALYSIS FOR THE SUPERCONDUCTING RF CAVITY HOM COUPLERS IN ESS, S.Molley et al



The ACE3P timeline – in terms of solving a large problem 1) Writing a parameterized script within the CUBIT meshing program ~1 week (comparable to commercially available codes such as HFSS, CST etc)

2) Meshing the problem within CUBIT, this is a semi manual process it is up to the user to manually correct any elements that are not within tolerance by adjusting the mesh itself ~1 weeks (can be difficult if your structure contains small features that require web-cutting etc) 3) Converting the CUBIT mesh to ACE3P mesh using ACDtool ~ 10mins

4) Writing ACE3P and job submission scripts ~ 30mins

5) Transferring files to a high performance cluster (using GLOBUS or SCP etc) ~ 1hr

6) Submitting jobs to the HOPPER queue system ~ 1min

7) Wait for job to go to the front of the queue ~ ?????

8) Wait for job to run – 1024 nodes will take about 12hours for a big problem (here jobs much larger than commercially available codes can be run – this is the advantage of ACE3P)

9) Analyse data using Paraview by either running locally on FRANKLIN or transferring data back.

ACE3P simulations Tips/Tricks/Workaround solutions

Things to be aware of......



Setting up an ACE3P simulation requires many steps Many of them manual and error-prone Running many simulations produces a lot of les; easy to loose track of differences No simple way to compare results between simulations Difficult to use for work requiring fast turn-around, such as design and optimisation of RF components As ACE3P stands it is best suited to large problems that cannot be solved using the standard commercially available software.

Basically you have to write/develop your own scripting process to run/extract/analyse the ACE3P suite ACDOPTI written by K.Sjobaek (CERN) is just such a set of scripts developed to automate the arduous ACE3P process: https://github.com/kyrsjo/AcdOpti

Create geometry

CUBIT

Create Mesh CUBIT

Mesh OK?

Convert Mesh ACDTOOL

Create ACE3P scripts

Upload to NERSC: GLOBUS

The ACE3P timeline continued....

Run scripts on NERSC Scratch

Wait...

Convert data on NERSC ACDTOOL

Transfer Results: GLOBUS

Visualise results:

PARAVIEW

CUBIT – the meshing program


CUBIT meshing Tips/Tricks/Workaround solutions



DIRECT IMPORT FROM HFSS OF COMPLETE ACC39 INTO CUBIT

Direct import of sat file generated from HFSS into cubit results in a meshing failure… Attempts to heal import in CUBIT have failed… Redraw required….

SIMPLER IMPORT TO IDENTIFY IMPORT ISSUE

Here is half the ACC39 structure i.e. the simplest repeating unit cavities C1 and C2. Here we see that the cubit meshing fails halfway through the bellows.


1) You are always better off parameterising your model and drawing it directly in CUBIT rather than trying to import and “heal” i.e. You will save alot of time 2) When drawing cavity shapes in CUBIT it is better to “deform” a pillbox cavity into the shape you require rather than trying to “sweep” your 2D shape around an axis 3) Most of us are use to sweeping geometry in other packages into the cavity shapes we require; however in CUBIT if you decide to use this method make sure that you sweep a closed parameterised curve rather than relying on boolean operations (which tend to fail in CUBIT). 4) CUBIT is probably the only mesher I have come across (there are many available e.g. tetgen etc...) where it is still possible for the mesher to incorrectly mesh a shape that is directly drawn in it’s GUI... It is a quirk of CUBIT, basically do not initially mesh your geometry too finely, instead adaptively refine it manually.... 5) Generating mesh in CUBIT is a black art that can only be learned through experience. 6) Generating the mesh for your ACE3P problem in CUBIT is probably the most difficult stage of the ACE3P calculation.....

CUBIT meshing quirks....

CUBIT parameterised macro for a TESLA like cavity

reset #Re = 35.787 #Aarc = 13.6 #Barc = 15 #SweepAngle = 90 #Ri = 15 #b = 6 #a = 4.6 #L = 19.2167 #Theta1L = 76.147 #Theta2L = 73.399 #Theta1yL = Barc*cos(Theta1L/180*PI) #Theta1xL = Aarc*sin(Theta1L/180*PI) #Theta2yL = b*cos(Theta2L/180*PI) #Theta2xL = a*sin(Theta2L/180*PI) ##create main body create surface ellipse major radius Aarc minor radius Barc xplane move surface 1 y Re-Barc . .

I have written a generic TESLA like cavity CUBIT journal file macro generator. This allows any TESLA like cavity to be generated within cubit. These can be quickly linked together to form cavities. This is a Python script so the entire process can be automated internally or externally. The macro is based on sweeping 2D curve of the cavity around the “z” axis. The macro is extendable to do end cells – so point 1 “Draw out the overall 9-cell 3.9 GHz structure with cubit” is complete – except for couplers…

I.Shinton, EuCARD-SRF Annual review 2011, Paris, 5th May 2011 18

Drawing ACC39 in CUBIT – in real time

Meshing ACC39 ACE3P is not the major problem… it is getting CUBIT to mesh your problem properly – NOT EASY FOR LARGE COMPLEX STRUCTURES The trick to proper meshing of large complex structures is to “webcut” and imprint sections manually. Manual adjustment of the mesh to then remove “bad elements”

Meshing ACC39 sequentially So one sequentially goes through the structure individually meshing each section individually. Checks are made at each stage that there are no bad elements – if there are then manual adjustment of surfaces/nodes/virtual surfaces etc are made.

The complete ACC39 mesh Here is the complete ACC39 mesh There are only two bad elements (out of over 16million…) - these bad elements I have not been able to fix; however they are very close to the acceptable limit – and as long as the Jacobean of their transform is not negative then they will be ok for FEM…

Adding the boundary surfaces in theACC39 mesh Boundary surfaces are added to the completed mesh, before the mesh is scaled to the units of meters (from mm) Once a volume block has been added (for material assignment) the mesh can be exported as a genesis mesh file.

CAUTION about adding boundary conditions to a webcut and imprinted structure

If you have a webcut and imprinted structure then between the individual pieces you have “cut” to mesh there will be surfaces… These will be assigned E boundaries by default… hence wrong answers will be obtained… It is important not to include these surfaces – make a separate sideset without these…. I can make a nice flythrough video of ACC39

ACDtool

ACDtool

Before you can use the mesh file in ACE3P you have to convert the genesis mesh into the ncdf format used in ACE3P. You do this using a program called ACDtool. I’ve put various versions of this SLAC program on afs: acdtool-v.3.2.2 works the best. So here is one of the simple test cases being converted

ACDtool checking the mesh

After you have converted the genesis CUBIT mesh to an NCDF mesh you have to check that all the elements are ok

ACDtool and ACC39

The ACC39 mesh that was generated by CUBIT and converted by ACDtool…~200MB mesh file… with 16million elements. As a comparison the NLSF HFSS mesh that Nawin is using (9cells with couplers) is approximately 1million elements.

ACDtool checking the ACC39 NCDF mesh

The ACC39 mesh that was generated by CUBIT and converted by ACDtool shows that there are 7 bad elements (probably and after effect from the 2 non-tolerant elements in the CUBIT mesh)

ACDtool fixing the bad ACC39 NCDF mesh

Because I was clever and used a high order tetrahedral meshing scheme (Tet10) I can get ACDtool to fix the bad elements So now we have a good mesh (according to ACDtool) that can be used in ACE3P Note if ACDtool cannot fix the bad elements it is up to the user to go back into CUBIT and re-mesh everything….

ACDTOOL Tips/Tricks/Workaround solutions



Here are some of the output files

The rf postprocessor package

kickFactor ionoff = 1 modeID1 = -1 modeID2 = -1 x0 = 0.00100 y0 = 0.00000 z1 = -0.00547 z2 = 0.00547 [kickFactor] // Ks=V^2/(4omega/c*U*r^2) Integral: x0 = 1.0000e-03, y0 = 0.0000e+00, r0= 1.0000e-03 z1 =-5.4672e-03, z2 = 5.4672e-03 ModeID Frequency (V_r, V_i) |V| Ks(V/C/m/cavity) 0 1.13612e+10 (-7.9267e+00, -1.4669e+01) 1.66736e+01 1.64829e+16

Need to be a bit careful with what acdtool gives you i.e. look at the definition for kickfactor – dipoles only….

The rf postprocessor package outputs

1) The kickfactor has the standard definition – for dipoles only:

ACE3P acdtool definitions

( )224/ UrVcKickfactor ω=

2) The Transverse R/Q in acdtool is for dipoles only

( )( )22// crUVRQT ωω=

3) The energy in the cavity is actually re-normalised to being equal to ε0/2!!!

4) The definition for R/Q is somewhat tricky in terms of the operation of acdtool (see later slides) and needs to be treated with caution…

( )UVRQ ω/2=

ACE3P acdtool R/Q things to be aware of: 1) The R/Q definition is not the standard definition, there is also no radial dependence in either the formula nor the outputted values for the voltage (hence no radial dependence at all…). This suggests that the voltage has been calculated along the axis and not at a given offset. 2) The energy in the cavity is actually re-normalised to being equal to ε0/2!!!

[RoverQ] // RoverQ=V^2/(omega*U) Integral: x1 = 0.0000e+00, y1 = 1.0000e-03, z1 =-1.3687e-01 x2 = 0.0000e+00, y2 = 1.0000e-03, z2 = 6.0000e-02 ModeID Frequency (V_r, V_i) |V| RoQ(ohm/cavity) 0 4.04964e+09 (-5.4596e-03, 1.4940e-02) 1.59065e-02 2.24612e-03 1 4.08524e+09 ( 1.1438e-02, 5.2390e-02) 5.36236e-02 2.53044e-02 2 4.19833e+09 ( 1.1260e-02, -4.4800e-03) 1.21186e-02 1.25756e-03 3 4.22633e+09 ( 1.1989e-02, -8.2007e-02) 8.28786e-02 5.84281e-02 4 4.44110e+09 (-1.7574e-02, 9.9309e-02) 1.00852e-01 8.23344e-02 5 4.49891e+09 ( 4.2391e-03, 8.4849e-02) 8.49548e-02 5.76727e-02 6 4.54641e+09 ( 2.0115e-03, -2.6613e-02) 2.66886e-02 5.63226e-03 7 4.85144e+09 (-5.5678e-02, -4.1836e-02) 6.96442e-02 3.59418e-02 8 4.88997e+09 (-1.8937e-02, 1.2829e-02) 2.28735e-02 3.84645e-03 9 5.12475e+09 ( 1.0486e-01, -3.0820e-02) 1.09294e-01 8.37960e-02 10 5.15125e+09 (-4.6929e-01, 3.1983e-01) 5.67910e-01 2.25086e+00

[RoverQT] // RoverQT=|V_T|^2/(omega*U)=|V_L|^2/(omega*U* (r0*omega/c)^2 ) Integral: x0 = 0.0000e+00, y0 = 1.0000e-03, r0= 1.0000e-03 z1 =-1.3687e-01, z2 = 6.0000e-02 ModeID Frequency (V_r, V_i) |V| RoQT(ohm/cavity) 0 4.04964e+09 ( 4.8616e-02, 5.7736e-02) 7.54784e-02 7.02065e+00 1 4.08524e+09 ( 4.1304e-02, 1.8236e-01) 1.86982e-01 4.19692e+01 2 4.19833e+09 ( 1.3754e-01, -2.9259e-02) 1.40622e-01 2.18703e+01 3 4.22633e+09 (-2.6599e-01, -5.2112e-01) 5.85077e-01 3.71123e+02 4 4.44110e+09 (-2.2147e-01, 1.0368e-01) 2.44536e-01 5.58726e+01 5 4.49891e+09 ( 1.9941e-01, 1.0495e+00) 1.06825e+00 1.02567e+03 6 4.54641e+09 (-9.9883e-03, -4.3234e-01) 4.32456e-01 1.62877e+02 7 4.85144e+09 ( 5.4903e-03, -2.8941e-01) 2.89460e-01 6.00547e+01 8 4.88997e+09 ( 1.2223e-02, -4.5690e-02) 4.72972e-02 1.56579e+00 9 5.12475e+09 (-2.6038e-01, -2.1255e-02) 2.61249e-01 4.15025e+01 10 5.15125e+09 (-2.2872e-01, 3.8556e-01) 4.48297e-01 1.20331e+02

[kickFactor] // Ks=V^2/(4omega/c*U*r^2) Integral: x0 = 0.0000e+00, y0 = 1.0000e-03, r0= 1.0000e-03 z1 =-1.3687e-01, z2 = 6.0000e-02 ModeID Frequency (V_r, V_i) |V| Ks(V/C/m/cavity) 0 4.04964e+09 ( 4.8616e-02, 5.7736e-02) 7.54784e-02 3.79044e+12 1 4.08524e+09 ( 4.1304e-02, 1.8236e-01) 1.86982e-01 2.30592e+13 2 4.19833e+09 ( 1.3754e-01, -2.9259e-02) 1.40622e-01 1.26908e+13 3 4.22633e+09 (-2.6599e-01, -5.2112e-01) 5.85077e-01 2.18235e+14 4 4.44110e+09 (-2.2147e-01, 1.0368e-01) 2.44536e-01 3.62792e+13 5 4.49891e+09 ( 1.9941e-01, 1.0495e+00) 1.06825e+00 6.83441e+14 6 4.54641e+09 (-9.9883e-03, -4.3234e-01) 4.32456e-01 1.10835e+14 7 4.85144e+09 ( 5.4903e-03, -2.8941e-01) 2.89460e-01 4.65338e+13 8 4.88997e+09 ( 1.2223e-02, -4.5690e-02) 4.72972e-02 1.23261e+12 9 5.12475e+09 (-2.6038e-01, -2.1255e-02) 2.61249e-01 3.58838e+13 10 5.15125e+09 (-2.2872e-01, 3.8556e-01) 4.48297e-01 1.05119e+14

Why won’t acdtool do my bidding?

1) Acdtool commands (a beit using the infinite monkey approach) have worked for ACE3P mesh conversion, mode conversion and wakefield conversion. So why does it fail when trying to use the rf post processor?

[ishinton@pc135 ACE3P_acdtool_sw_test_1]$ ./acdtool-v3.2.3 postprocess rf input.in ************************************************************* *** acdtool V3.2.0 08/18/2009 $ *** ------------------------------------------------------------- Copyright 2007, Advanced Computations Department Stanford Linear Accelerator Center ************************************************************* ****************************************************************** o3pp-5-20 Use eigenvector files from Omega3P V5.20!! ****************************************************************** RFField O3PMode = vector1 ModeRangeID1= -1 ModeRangeID2= -1 ModeID = 0 xsymmetry = magnetic // [none, electric, magnetic] ysymmetry = magnetic // [none, electric, magnetic] gradient = 1.00000e+06 cavityBeta = 1.00000 //for R/Q, V integral reversePowerFlow= 0 // [1=charging 0=decaying] x0 = 0.00000 y0 = 0.00000 gz1 = -0.00547 gz2 = 0.00547 fx1 = -10000000.00000 fx2 = -10000000.00000 fy1 = -10000000.00000 fy2 = -10000000.00000 fz1 = -10000000.00000 fz2 = -10000000.00000 npoint = 300 fmnx = 10 fmny = 10 fmnz = 50 Time for calculating adjacency: 0.020447 Partitioning Method: parmetis terminate called after throwing an instance of 'pncdf_error' what(): open [pc135:24948] *** Process received signal *** [pc135:24948] Signal: Aborted (6) [pc135:24948] Signal code: (-6) [pc135:24948] [ 0] [0xffffe600] [pc135:24948] *** End of error message *** Aborted

How to make acdtool work for the rf postprocessor package

Acdtool actually needs a copy of ACE3P to read the raw eigen vector files in order to process the data. So at the moment the only way to use the acdtool rf postprocessor is to analyze all the data on NERSC. So here’s what to do….

1) Transfer the entire data files (if stored) onto a NERSC machine 2) Create a pbs acdtool script to initially analyze the data there: ~candel/franklin…… acdtool rf postprocess 3) Take the output script and modify it for the values and offset you want 4) Put the modified script back onto a NERSC machine 5) Create another pbs script to use the modified output script 6) Run the final script and export the results as usual using scp to the data export node….

Basically a lot of effort is required to do this step (especially since a lot of the data I have run is now stored externally)

ACE3P scripts and the NERSC job queue

HOPPER

Hopper is a very large cluster about 6362 nodes in total. It is setup somewhat different from Franklin with the emphasis being on more MPI per node. Each Hopper node has 24 cores and a total of 128GB of hard RAM The T3P module from ACE3P should be better suited to running on Hopper (due to the large number of CPU’s per node)

Running scripts on Hopper is very similar to running scripts on Franklin (i.e. a few changes in the pbs files) – refer to CW11 or CW10 SLAC’s ACD website The scripts for a OMEGA3P, S3P, T3P problem are all very similar – refer to refer to CW11 or CW10 SLAC’s ACD website.

ACE3P ACC39 scripts continued Here is an example of an ACC39 OMEGA3P pbs script for looking at the eigen values of the trapped beam-pipe modes just above 4GHz

NERSC job queue Tips/Tricks/Workaround solutions



PBS Job Id: 7202977.nid00003 Job Name: ACC39_HOM_A_Scatter_debug2 Exec host: nid00576/6 An error has occurred processing your job, see below. Post job file processing error; job 7202977.nid00003 on host nid00576/6 Unable to copy file /global/homes/m/mewgroup/ACC39_Scattering_2cell_HOM_A_Scatter_debug_2/7202977.nid00003.ER to /global/homes/m/mewgroup/ACC39_Scattering_2cell_HOM_A_Scatter_debug_2/ACC39_HOM_A_Scatter_debug2.e7202977 *** error from copy /bin/cp: cannot stat `/global/homes/m/mewgroup/ACC39_Scattering_2cell_HOM_A_Scatter_debug_2/7202977.nid00003.ER': No such file or directory *** end error output

ACE3P usage issues…..

Need to watch out for maintenance on NERSC machines – scratch files are regularly wiped. Best to transfer data to either a local machine using the GLOBUS program (as recommended by NERSC – nice GUI, basic idiot SCP program) or transfer your results to HPC NERSC storage system

PARAVIEW – Visualising the ACE3P data


Paraview is well documented and easy to use Can be scripted in Python – has a python interface As long as you have enabled the “SLAC” tab in the filter section you will be able to effortlessly import ACE3P geometry and files

ACE3P data visualisation using PARAVIEW

Documents

The use of ACE3P for the design and analysis HOM’s in SCRF ... · analysis HOM’s in SCRF accelerating structures. ... commercially available codes such as HFSS, CST etc) ... between