Cactus Workshop - NCSA Sep 27 - Oct 1 1999

Cactus For Relativistic Collaborations

Ed SeidelAlbert Einstein Institute

eseidel@aei-potsdam.mpg.de

Cactus for Relativistic CollaborationsCactus for Relativistic Collaborations Back to Physics (the main goal of most of us today)…

Yesterday will make achieving this easier But don’t lose sight of our physics goals...

What we hope to achieve through community code Question: is it a good idea to have one code for a whole community?

Yes! But Cactus is a metacode; it is many codes; it a glue for codes…

Grand Challenges First attempt to get large scale collaborations across disciplines to work Now can really do it with such a collaborative, connective infrastructure to

support entire communities!

Present overview today of: What physics we have done with Cactus What modules have been created and tested Which are already in Cactus 4, what is coming from Cactus 3, what is NOT

there yet

The Promise of a Community Code for Relativity and AstrophysicsThe Promise of a Community Code for Relativity and Astrophysics

Intellectual Property Philosophy Use cactus to solve your own problems through standards for the

community Use publicly available thorns as you need them Develop your own private thorns for your own research Make them public when and only when you are ready (e.g., maybe you

want to publish first!)

Sharing of work, modules, results much easier Providing numerical relativity functionality to groups previously

working in other areas, from mathematics to astrophysics Much more activity in the field!!

Advanced developments in computational science made available through KDI, other research projects

Cactus Again, a new community code for 3D simulations

Publications so far with/about CactusPublications so far with/about Cactus “The Cactus Computational Collaboratory: Enabling Technologies for Relativistic

Astrophysics, and a Toolkit for Solving PDE’s by Communities in Science and Engineering”, G. Allen, T. Goodale, and E. Seidel, 7th Symposium on the Frontiers of Massively Parallel Computation-Frontiers ’99, IEEE, (1999).

“Technologies for Collaborative, Large Scale Simulation in Astrophysics and a General Toolkit for solving PDE's in Science and Engineering”, E. Seidel, to appear in “Forschung und wissenschaftliches Rechnen”, T. Plesser and P. Wittenburg, eds., (Max-Planck-Gesellschaft, 1999).

“Numerical Relativity in a Distributed Environment”, W. Benger, I. Foster, J. Novotny, E. Seidel, J. Shalf, W. Smith, and P. Walker, SIAM PPP, (1999).

“Three Dimensional Numerical Relativity with a Hyperbolic Formulation”, C. Bona, J. Massó, E. Seidel, P. Walker, Physical Review D15, in press…

“Numerical Relativity As A Tool For Computational Astrophysics”, E. Seidel and Wai-Mo Suen, Journal of Computational and Applied Mathematics”, (1999).

“Test-beds and applications for apparent horizon finders in numerical relativity”, M. Alcubierre, S. Brandt, B. Brügmann, C. Gundlach, J. Masso, E. Seidel, and P. Walker, gr-qc/9809004 Physical Review D, (1998).

More publications...More publications...

“Collapse of 3D Gravitational Waves to form a Black Hole”, M. Alcubierre, G. Allen, B. Brügmann, G. Lanfermann, Edward Seidel, W.-M. Suen, and Malcolm Tobias, submitted to Physical Review Letters, (1999).

“Axisymmetry without Axisymmetry: A New Method for Spherical and Axisymmetric Simulations without Coordinate Singularities”, M. Alcubierre, B. Brügmann, E. Seidel, and J. Thornburg, in preparation for Computer Physics Communications, (1999).

“Towards an understanding of the stability properties of the 3+1 evolution equations in general relativity”, Miguel Alcubierre, Gabrielle Allen, Bernd Bruegmann, Edward Seidel, Wai-Mo Suen, submitted to Physical Review D, gr-qc/9908079, (1999).

“The Shapiro Conjecture: Prompt or Delayed Collapse in the head-on collision of neutron stars?”, Mark Miller, Wai-Mo Suen, Malcolm Tobias, gr-qc/9904041(1999).

“A Conformal Hyperbolic Formulation of the Einstein Equations”, Miguel Alcubierre, Bernd Brugmann, Mark Miller, Wai-Mo Suen, submitted to PRD, (1999) gr-qc/9903030.

“Three Dimensional Numerical General Relativistic Hydrodynamics I: Formulations, Methods, and Code Tests”, J. A. Font, M. Miller, W. Suen, M. Tobias, submitted to PRD, (1998), gr-qc/9811015.

“Robust evolution system for Numerical Relativity”, A. Arbona, C. Bona, J. Masso, J. Stela, submitted to PRD, (1999), gr-qc/9902053.

… (probably forgot a few…)

Many others in the pipeline, hopefully soon some from you...

Recent Large Scale Simulations of CactusRecent Large Scale Simulations of Cactus Black Holes (prime source for GW)

Largest Ever Prod. Simulations: 3843 (80GB Memory Required) Full 3D Distorted BH Evolutions, Extracting Waves Increasingly complex collisions: now doing full 3D grazing collisions

Gravitational Waves Study linear waves as testbeds Move on to fully nonlinear waves Interesting Physics: BH formation in full 3D!

Neutron Stars Developing capability to do full GR hydro Now can follow full orbits!

Evolving Pure Gravitational WavesEvolving Pure Gravitational Waves

Probe GR in highly nonlinear regime Form BH? Critical Phenomena in 3D?

– Choptuik, many others in 1D now– 2D example of Abrahams and Evans

Naked singularities? … Little known about generic 3D behavior

Take “Brill Wave” Initial data ds2 = 4 (e2q(d2+dz2)+ 2d2) q = A f(,z,) Choose Kij (take time symmetry for now…) Solve constraints

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Highly nonlinear waves

Subcritical Waves: Everything radiates away...Subcritical Waves: Everything radiates away...

Newman-Penrose 4 (showing gravitational waves)with lapse underneath

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Supercritical Waves (form Black Hole!)Supercritical Waves (form Black Hole!)

4 showing collapse to BH, AH with gaussian curvature

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Comparison of full non-axisymmetric and axisymmetric supercritical collapse to BHComparison of full non-axisymmetric and axisymmetric supercritical collapse to BH

l=2,m=2 wave extraction

3D

Axisym. simulationQNM fit to two lowestmodes for BH ofappropriate mass

BH forming Formed BH Ringing

Now try first 3D “Grazing Collision”: Spinning, “orbiting”, unequal mass BHs merging.Now try first 3D “Grazing Collision”: Spinning, “orbiting”, unequal mass BHs merging.

Preliminary evolution results Can be evolved

beyond through coalescence

AH’s merge early 3843 (largest we’ve

attempted)

QuickTime™ and aVideo decompressor

are needed to see this picture.

But just grazing collision: have a long way to go to get full orbits...

Apparent Horizon of Coalescing HolesApparent Horizon of Coalescing Holes

z-axis

x-axis

t=5M

Merging of AH’s

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Without excision, BH’s willnot move across grid...

Gravitational Radiation from the Grazing Collision (inner 2563 of 3843 simulation shown)Gravitational Radiation from the Grazing Collision (inner 2563 of 3843 simulation shown)

Re(4) (red: even-parity radiation) Im(4) (blue: odd-parity radiation)

QuickTime™ and aMotion JPEG B decompressor

are needed to see this picture.

More than a pretty movie: Extracting WaveformsMore than a pretty movie: Extracting Waveforms

l=2,m=0 even-parity l=2,m=2 even-parity

l=2,m=0 odd-parity

• Waves extracted very close to the final BH: 8M• Very preliminary, but modes show QNM-like ringing…• Each carry < 0.001Madm in energy

(MAH)

2 = (Mir)2 + J2/(4 (Mir)2 ) MAH = 3.08

Growth in Area of Apparent HorizonGrowth in Area of Apparent Horizon

Mir = (AAH/16 ≈

SpuriousGrowth (error)

Merged AH Appears

(AAH/16

Time (M)

Total Mass of Spacetime: MADM = 3.11

Total Mass of Final BH: MAH = 3.08

MADM - MAH = 0.03 = 0.0097MADM

Compare to…

Total energy radiated in all modes (integrated only to t=30M):

MRAD ~ 0.007 - 0.008MADM

MRAD ≈ MADM - MAH !!

Total Energy Accounting of Grazing CollisionTotal Energy Accounting of Grazing Collision

Specific Thorn of interest: “Cartoon 2D”Specific Thorn of interest: “Cartoon 2D” Axisymmetric systems difficult in axisym coords.

Singularities (e.g. 1/(r sin))

Special coordinate systems introduce problems to be solved only there…don’t carry over to 3D (e.g. special “diagonal gauge”)

Routines develop for 3D cartesian systems will usually not work in 2D axisymmetric coordinate systems, and vice versa.

Would like to be able to study axisymmetric systems as testbeds in 3D, yet cartesian simulations take forever, require huge amounts of memory (~N3)…

Would like to use the SAME tools in 2D or 3D, SAME equations, SAME gauge conditions, etc...

Cartoon 2DCartoon 2D Basic Idea:

Axisymmetric system lives in a single plane Can do a “thin slab” in x-z plane, 3 zones thick Then need boundary conditions on “ghost” zones, which are

provided by rotations of scalar, vector, and tensor quantities being evolved

Now implemented in Cactus: use on, e.g., BH problems studied yesterday!!

All 3D cartesian modules

should work...

x-z plane

Ghost boundaryzones

Rotate interior data after each time step to boundaries

SchedulingScheduling

Basic Skeleton idea Initial data Evolution loop

– Evolve– Analysis and IO

Extend it with scheduling groups A certain thorn A must be called before another one B and after C

– Example: Initialization of different pieces• some elliptic solve to perform after initializing the fields• Then, some other fields are initialized based on the elliptic solution

Some thorn must be called while some condition is met

Future redesign The scheduler is really a runtime selector of the computation flow. We can add much more power to this concept