TRANSP and Frameworks

TRANSP and Frameworks

Presented by D. McCune at the Joint ORNL/IU Workshop on Computational

Frameworks in Fusion,

June 6, 2005

Outline of Talk

• TRANSP Background Information:– What is TRANSP; Applications of TRANSP.– Users, Developers, Funding profile.

• FusionGrid TRANSP.

• TRANSP Code Generators – Python.

• TRANSP Makefile Generators – Development Environment.

• Requirements for Frameworks.

What is TRANSP?• “1.5D” time dependent tokamak transport

simulation tool:– 2d axisymmetric tokamak reduced to 1d by

using standard “flux coordinate” methods.

• Major application: analysis of experimental data from tokamaks.

• Major strengths:– Physics: e.g. validated fast ion models.– Operations: effective workflow & support.

• Under continuous development since 1978!

Applications of TRANSP

• Transport and confinement studies (with or without ITB, H-mode, etc.).

• Neutral beam heating simulation; computation of fast ion distributions.

• RF heating and current drive simulations.

• Diagnostics simulations.

• Predictive calculations for experimental proposals.

Applications of TRANSP (cont.)

• Many groups report use of TRANSP results in conference papers and refereed journal articles.

• Two examples follow:– Tritium Transport on JET (I. Voitsekovitch).– NPA Simulations on NSTX (S. Medley).

• Numerous other examples could have been chosen.

TRANSP at JET

Trace Tritium: Thermal Tritium Transport:

14 MeV neutrons measured along 19 chords (left) are reproduced in TRANSP by using a proper adjustment of Tritium transport coefficients (right, 6 bottom chords are shown)

2 techniques have been used: (1) Dt and Vt are adjusted in TRANSP; (2) novel use -

combined TRANSP/SANCO analysis -plasma reactivity is extracted from TRANSP using 20 runs and transmitted to SANCO for automatic estimation of Dt and Vt by

minimising 2-value

Publications: K.-D. Zastrow and JET team (EPS 2004, PPCF, 2004 in press), I. Voitsekhovitch, et

al., Trace Tritium transport in H-mode JET plasma with different density, JET pinboard, waiting for clearance, to be submitted to Phys. Plasmas and also in EPS 2004; J. Mailloux et al, EPS 2004; P. Belo et al., EPS 2004; D. Stork and JET team, IAEA 2004; D. Stork and JET team, APS 2004

MHD-Induced Fast Ion Losses in NSTX

SN108730

ln( N

PA F

lux/

Ene

rgy

1/2

) (

ster

-1cm

-2eV

-3/2

s-1

)

Energy (keV)0 20 40 60 80 100

t = 400 ms During H-mode

TRANSP

NPA

Ion Loss Region

15

19

18

17

16

20

21

14

13

12

TRANSP WITH FAST ION DIFFUSION - 108730M07

TRANSP WITHOUT FAST ION DIFFUSION - 108730M08

TRANSP

Normalization

Sn

(x10

13 s

-1)

Time (s)

SN108730

H-Mode Onset

0 0.2 0.4 0.6 0.8

MHD Onset

2

0

4

8

6

10

12

14

16

18

20

22

Measure Neutron Rate

TRANSP with Fast Ion Diffusion

TRANSP without Fast Ion Diffusion

S. Medley et al., Nucl. Fusion 44(2004)

TRANSP Developers and Users

• PPPL TRANSP team:– R. Andre, A. Brooks, F.

Dahlgren, E. Feibush, K. Indireshkumar, S. Klasky, L. P. Ku, C. Ludescher, D. McCune, L. Randerson

• ~5 FTE for TRANSP & related efforts.– Major focus on

production & support…

• User Sites:– Culham (MAST)– GA (DIII-D)– IPP/Garching (Asdex-

U)– JET– MIT (C-Mod)– PPPL (NSTX)– PPPL

(Collaborations)

FY05 PPPL TRANSP Funding

• NSTX/MAST (approx. 2.15 FTE, 60%)• DIII-D collaboration (0.9 FTE, 25%)• C-Mod collaboration (0.25 FTE, 7%)• JET collaboration (0.25 FTE, 7%)• Tokamak Projects Total: 3.55 FTE• R&D Projects (1.45 FTE):

– PTRANSP (predictive upgrade), 0.6 FTE– SciDAC FusionGrid (2 year renewal grant),

0.85 FTE

Outline of Talk

• TRANSP Background Information.

• FusionGrid TRANSP:– TRANSP as a Computational Service.– TRANSP Workflow; Workflow Automation.– Future: Linked, Distributed Services.

• TRANSP Code Generators – Python

• TRANSP Makefile Generators – Development Environment.

• Requirements for Frameworks.

FusionGrid TRANSP

• TRANSP runs as a service on PPPL Linux cluster machines.– User authentication via Globus.– Credit to SciDAC Fusion Collaboratory Project.

• Users submit run requests via supported client software tools.

• Users can monitor system load and watch progress of runs, access log files, etc.

• Website: http://w3.pppl.gov/TRANSP ...

TRANSP Client Tools, Services

• Prepare Input Data.

• Submit runs.

• Monitor run progress; view log files.

• Fetch interim data.

• Option to cancel run prior to completion.

• Explore, visualize output data.

• Crashed runs debugged by TRANSP support staff at PPPL.

access via http://w3.pppl.gov/TRANSP

FusionGrid TRANSP Annual Run Production

0

500

1000

1500

2000

2500

3000

Total Runs

through10/31/2002*

11/1/2002 --10/31/2003

11/1/2003 --10/31/2004

*FusionGrid TRANSP Operational Since September 2002

FusionGrid TRANSP Workflow

Preliminary dataAnalysis andPreparation

(largely automated)

DiagnosticHardware

Experiments (CMod, DIII-D, JET, MAST, NSTX)

20-50 signals {f(t), f(x,t)}Plasma position, Shape,Temperatures, DensitiesField, Current, RF andBeam Injected Powers.

TRANSP Analysis*:Current diffusion, MHD equilibrium, fast ions,

thermal plasma heating; power, particle and

momentum balance.

TRANSP Analysis*:Current diffusion, MHD equilibrium, fast ions,

thermal plasma heating; power, particle and

momentum balance.

Experiment simulationOutput Database

~1000-2000 signals{f(t), f(x,t)}

Visualization

Load RelationalDatabases

Detailed (3d) time-slice physics simulations (GS2, ORBIT, M3D…)

*FusionGrid TRANSP now executes on PPPL servers;GS2 and other grid services are planned for the future.

Pre- and Post-processing at the experimental site…

Workflow Automation

• User-programmable automated data preparation is key to system productivity.– IDL (many legacy tools in fusion program).– UREAD/SGLIB and other legacy fortran tools*.

• Also important: – convenient tools for submission and web-

based monitoring of runs.– User-programmable automated visualization,

database loading, and further post-processing.

D. McCune 23 Apr 2004

*available at http://w3.pppl.gov/NTCC

TRANSP Workflow Caveats

• The software technology used is ancient.– But: familiar to current users.

• The workflow volumes (dataset sizes) are small, so far.– So, no motivation yet for transition to high

performance solutions.

• The potential for generation of larger datasets exists.– Current practice may be a limiting factor.

TRANSP FusionGrid Future: Linked, Distributed Services.

Serial TRANSP at PPPL

Parallel RF Wave Solver

(TORIC at MIT)

Parallel MonteCarlo Fast Ion

Model(NUBEAM at

PPPL)

• Multi-site Globus Forwarded Authentication• Possible issues:

• Firewalls• Resource sharing

• Queue wait time.• Data transfer times.

Outline of Talk

• TRANSP Background Information.

• FusionGrid TRANSP.

• TRANSP Code Generators – Python:– Ad hoc data specification languages.– Python code generators.

• TRANSP Makefile Generators – Development Environment.

• Requirements for Frameworks.

Code Generators

• Many i/o interfaces of TRANSP are written by code generators (Python scripts).– Measured once to be ~25% of fortran lines.

• Code generator input: list of data items.

• Code generator output:– Anything needing to be driven by the list:

• Fortran namelists, “state” or restart file i/o routines, documentation, allocate/deallocate routines, default/initialization routines, plot labeling routines, Fortran-90 type definition modules, debugging…

Ad Hoc Languages

• Custom, ad hoc, requirements-driven “language” for each list.

• Example: NTCC NUBEAM “nbspec.dat”*:– Sections: Array_Dimensions, Constants,

Inputs, Internal, Outputs.– “nbigen.py” writes code for internal data

structures, public modules defining input/output data structures, input default settings, state file i/o, html documentation, …

*see http://w3.pppl.gov/NTCC

Ad Hoc LanguagesSection: Inputs # from nbspec.dat… … …Block: Atomic ! Atomic physics controls (defaults shown).D xdepmod = 1.0d0 ! Anomolous deposition opacity adjust.L nlbbcx = .TRUE. ! .FALSE. to disable beam-beam ch.ex. … …Section: Outputs … …orb%DAS sorbn0(mj,mig,mibs) ! Thermal neutral source ! due to deposition or charge exchange recapture of fast ! neutrals, vs. x, thermal ion, fast ion; “orb” MPI_reduce.

Key: D means REAL*8, L means LOGICALA means ARRAY, S means “in State File”,…

Code Development Automation

• Code generators organize i/o and “book-keeping” aspects of code.– They make it easy to add inputs & outputs:

• (a) edit specification file.• (b) run generator python script.

– Less error prone than hand-maintained methods, where list-driven requirements exist.

– Allows greater focus on physics coding.

• Key to long term success of TRANSP.

Python Chosen for Code Generators

• Early TRANSP code generators were written in fortran-77 or C, but…

• Much easier / faster to work with Python.– Python “dictionary” and “list” data structures

perfect for code generator application.– Fast prototype / debug cycle.– Python has excellent error reporting.– Python has excellent tutorial and reference

documentation (e.g. O’Reilly books).

Outline of Talk

• TRANSP Background Information.

• FusionGrid TRANSP.

• TRANSP Code Generators – Python

• TRANSP Makefile Generators – Development Environment:– Makefile generator system.– Resulting benefits for code developer.– Issues / Limitations.

• Requirements for Frameworks.

TRANSP Developers do not edit Makefiles…

• Makefiles are written by makefile generator for:– Subroutine libraries (179).– Executables (220).

• Supported by script database with information on various fortran compilers…

• Library makefiles based on contents of library source directory:– F90 module dependency analysis included.

TRANSP Makefile Generator

• Linkage for executables defined by separate file containing ordered list of objects and libraries:

> cat foo_exe.linkfoo_main.ofoo_sub1.afoo_sub2.a$L_NETCDF$L_LAPACK$L_FFTW

Elements:• .o – from TRANSP source.• .a – from TRANSP source.• $L_<name> -- non-TRANSP dependency.

Command Line Interface

• cmsmms <libname> -- generate makefile for library.

• uplib <libname> -- update (build) library.• makelink <progname> -- generate

makefile for executable.• uplink <progname> -- update executable.• These commands build “standard” (i.e.

non-debug) libraries and executables.

Debug builds• dbxadd <source-name> -- add single

source to debug list.– F90 module dependencies added

automatically.

• dbxaddlib <libname> -- add entire library to debug list.

• uplink <progname> debug – build debug executable.– Makefile for custom debug executable is built;

only named items are debug compiled.

Benefits

• Relieves developer of makefile maintenance requirement.– Hand built makefiles become nightmares…

• Precise control of generation of debug executables.

• Easy to learn.

• Saves time.

Drawbacks / Limitations

• Monolithic – not easily transferable outside the TRANSP project.

• Too much dependency on ancient csh scripts (awkward to maintain).

• Certain peculiarities, due to a lapsed VMS compatibility requirement, persist…

• We are looking at SCONS for possible upgrade or replacement…

• Advice of CS experts would be welcome!

Outline of Talk

• TRANSP Background Information.

• FusionGrid TRANSP.

• TRANSP Code Generators – Python

• TRANSP Makefile Generators – Development Environment.

• Requirements for Frameworks.– Generic Requirements.– Requirements Checklist.– Compatibility with CS Research (?).

Generic Requirements

• Non-invasive; light-weight.– No motivation for massive project to convert

an existing, working system.– Evolutionary method needed for introduction

of new technology.

• Simple things must be simple, complicated things must be possible.

• Well engineered.– Robustness, ease-of-use, documentation…

Requirements Checklist

• Technical assessment:– Computational resources needed.– Data management issues to be faced.

• Sociological assessment:– Define benefit to end user.– Define cost to end user (e.g. skills to learn).– Define benefit to code developer.– Define cost to developer (e.g. skills to learn).

Requirements vs. CS Research

• Are the deployment requirements for a production scientific application compatible with CS research constraints?– How are engineering issues handled:

robustness, error handling, documentation…– Who finances such expensive work?

• Shouldn’t production deployment be viewed as the “experimental program” for software technology research?

Example of adoption of new software technology in the

TRANSP project: introduction of Python for code generator

applications.

The End.

Hopefully, the perspective derived from TRANSP experience will be helpful. But, it is only one perspective…