SENR/NRPy: A Next-Generation, Dynamical Reference Metric ...astro.phys.wvu.edu/zetienne/SENR/Etienne_ET_Workshop_2017__SENR_NRPy.pdfSENR/NRPy: Motivation SENR/NRPy = Simple, efficient,

SENR/NRPy: A Next-Generation, Dynamical SENR/NRPy: A Next-Generation, Dynamical Reference Metric Numerical Relativity CodeReference Metric Numerical Relativity Code

Zachariah B. EtienneZachariah B. EtienneIan RuchlinIan Ruchlin

in collaboration within collaboration with

Thomas W. BaumgarteThomas W. Baumgarte

SENR/NRPy: Code OverviewSENR/NRPy: Code Overview

● NRPy: Python+sympy code generation for NRNRPy: Python+sympy code generation for NR● Similar to Kranc, but with no Mathematica!Similar to Kranc, but with no Mathematica!

– Equations Equations at your fingertipsat your fingertips, even on HPC systems!, even on HPC systems!– Input: Einstein notation + simple syntax Python codeInput: Einstein notation + simple syntax Python code– Output: Efficient, compiler-vectorizable C code (AVX)Output: Efficient, compiler-vectorizable C code (AVX)

● SENR: Simple, Efficient Numerical Relativity codeSENR: Simple, Efficient Numerical Relativity code● Contains NRPy wrappers, diagnostics, MoL, BCs for Contains NRPy wrappers, diagnostics, MoL, BCs for

solving BSSN equations in arbitrary coord systemssolving BSSN equations in arbitrary coord systems– Log-Spherical Polar, Cylindrical, Cartesian, Bispherical-likeLog-Spherical Polar, Cylindrical, Cartesian, Bispherical-like

https://tinyurl.com/senrcode

SENR/NRPy: MotivationSENR/NRPy: Motivation

● SENR/NRPy = Simple, efficient, open (BSD-licensed, SENR/NRPy = Simple, efficient, open (BSD-licensed, Python-based) infrastructure for numerical relativity codes Python-based) infrastructure for numerical relativity codes and code generationand code generation

● Goal: When solving problem, choose the best coordinate Goal: When solving problem, choose the best coordinate grid for the task!grid for the task!● Black hole, neutron star: Log-Spherical polar coordsBlack hole, neutron star: Log-Spherical polar coords● Compact binary: Dynamical, Bispherical-like coordsCompact binary: Dynamical, Bispherical-like coords

● Better coordinate grids = Giant efficiency gain over AMR!Better coordinate grids = Giant efficiency gain over AMR!● At least At least ~160x decrease in # of gridpoints → use desktop for BHB~160x decrease in # of gridpoints → use desktop for BHB● Single grid patch = ~25x better scalability than AMR!Single grid patch = ~25x better scalability than AMR!

https://tinyurl.com/senrcode

Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations

BHBHBHBHdxdx dxdx

2 dx2 dx

4 dx4 dx

8 dx8 dx

16 dx, etc16 dx, etc

AMRAdaptive Mesh Refinement

(Most Popular Method in NR)

BHBHBHBHdxdx dxdx

2 dx2 dx

4 dx4 dx



8 dx8 dx

16 dx, etc16 dx, etc


BHBHdxdx


BHBHBHBHdy=dx=drdy=dx=dr

Near-Spherical Near-Spherical ObjectObject

● Highest res needed Highest res needed in radial dirn, need in radial dirn, need ~1/3 points in ~1/3 points in angular directions angular directions Cost: Cost: Nr*Ntheta*Nphi ~ Nr*Ntheta*Nphi ~ 1/9 Nr1/9 Nr33

● Cartesian grid: Cartesian grid: need dx=dy=dz=dr. need dx=dy=dz=dr. Cost:Cost:Nx*Ny*Nz ~ NrNx*Ny*Nz ~ Nr33

● So far, spherical So far, spherical polar grid ~polar grid ~9x 9x more more efficient than efficient than CartesianCartesian




What about dr What about dr along diagonal?along diagonal?

● Cube diagonal = Cube diagonal = √√3*sidelength → to 3*sidelength → to get dr resolution in get dr resolution in all directions, need all directions, need to reduce dx,dy,dz to reduce dx,dy,dz by by √√33

● Since cost in Since cost in memory ~1/dxmemory ~1/dx33, , “fitting the round “fitting the round peg in a square peg in a square hole” increases hole” increases cost by another cost by another factor of factor of ((√√3)3)33~5.2x!~5.2x!






Inefficiencies so far: Inefficiencies so far: ~47x~47x


What about dr What about dr along diagonal?along diagonal?

● Cube diagonal = Cube diagonal = √√3*sidelength → to 3*sidelength → to get dr resolution in get dr resolution in all directions, need all directions, need to reduce dx,dy,dz to reduce dx,dy,dz by by √√33

● Since cost in Since cost in memory ~1/dxmemory ~1/dx33, , “fitting the round “fitting the round peg in a square peg in a square hole” increases hole” increases cost by another cost by another factor of factor of ((√√3)3)33~5.2x!~5.2x!





BHBHBHBH AMR Box side-AMR Box side-length = 2length = 2

AMR Box AMR Box Boundary is a Boundary is a

Cube...Cube...

● … … but fields fall off but fields fall off radially!radially!

● →→region outside region outside orange circle is orange circle is over-resolved by 2xover-resolved by 2x

● Total volume of Total volume of over-resolved over-resolved region = 8-4/3 pi ~ region = 8-4/3 pi ~ 3.8 = about half the 3.8 = about half the cube!cube!

● Bispherical Bispherical coordinate system: coordinate system: Gain another ~1.7xGain another ~1.7x


BHBHBHBH AMR Box side-AMR Box side-length = 2length = 2


Cube...Cube...





Inefficiencies so far: Inefficiencies so far: ~80x~80x


BHBHBHBHdxdx dxdx

2 dx2 dx

4 dx4 dx



8 dx8 dx

AMRAMR

● Information must be interpolated Information must be interpolated across refinement boundaries.across refinement boundaries.

● Interpolation → grids must overlap Interpolation → grids must overlap ● Overlap regions (grey) can take up Overlap regions (grey) can take up

50% of overall computational domain!50% of overall computational domain!


BHBHBHBHdxdx dxdx

2 dx2 dx

4 dx4 dx



8 dx8 dx

AMRAMR

● Information must be interpolated Information must be interpolated across refinement boundaries.across refinement boundaries.

● Interpolation → grids must overlap Interpolation → grids must overlap ● Overlap regions (grey) can take up Overlap regions (grey) can take up

50% of overall computational domain!50% of overall computational domain!


BHBHBHBH


Cube...Cube...





High-order finite High-order finite difference with difference with

AMRAMR

● → → Enormous Enormous number of ghost number of ghost zones at refinement zones at refinement boundaries!boundaries!

● Ghost zones can Ghost zones can take up 50% of take up 50% of overall overall computational computational domain!domain!

● Bispherical Bispherical coordinate system: coordinate system: Gain another ~2xGain another ~2x


BHBHBHBH


Cube...Cube...





AMR Inefficiencies: AMR Inefficiencies: ~160x~160x

High-order finite High-order finite difference with difference with

AMRAMR

● → → Enormous Enormous number of ghost number of ghost zones at refinement zones at refinement boundaries!boundaries!

● Ghost zones can Ghost zones can take up 50% of take up 50% of overall overall computational computational domain!domain!

● Bispherical Bispherical coordinate system: coordinate system: Gain another ~2xGain another ~2x


AMR Inefficiencies: AMR Inefficiencies: ~160x~160x


AMR Inefficiencies: AMR Inefficiencies: ~160x (estimated)~160x (estimated)




Inefficiency Inefficiency MeasurementMeasurement

● Single black holeSingle black hole● Moderate spin:Moderate spin:

● a/M = 0.5a/M = 0.5● Set up AMR Set up AMR

(Carpet) grid, (Carpet) grid, measure H measure H constraint violationconstraint violation

● Adjust SENR grids:Adjust SENR grids:● H_SENR < H_AMR H_SENR < H_AMR

at all pointsat all points









AMR Grid:AMR Grid:● 10 GB10 GB

SENR's Log-SENR's Log-Spherical Spherical Polar Grid:Polar Grid:

● 40 MB40 MB● (un-optimized grid (un-optimized grid

structure, another structure, another 4-10x drop 4-10x drop possible)possible)













AMR Inefficiencies: AMR Inefficiencies: 250x (measured)250x (measured)













AMR Inefficiencies: AMR Inefficiencies: 250x (measured)250x (measured)

Stable long-term BH evolutions!Stable long-term BH evolutions!But does it converge?But does it converge?

SENR Results: SENR Results: Exponential convergence of Exponential convergence of

numerical errors!numerical errors!

Simulating Simulating black black hole without hole without

excisionexcision: :

Numerical errors Numerical errors converge to zero converge to zero

exponentiallyexponentially with with increased increased

polynomial polynomial approximation order!approximation order!

SENR/NRPy: SummarySENR/NRPy: Summary● Open Source, Open DevelopmentOpen Source, Open Development → → Greater AdoptionGreater Adoption

● http://tinyurl.com/senrcodehttp://tinyurl.com/senrcode● Algorithmic SimplicityAlgorithmic Simplicity → → More Science FasterMore Science Faster

● Easier to debug & extendEasier to debug & extend● Build on tried & true algorithmsBuild on tried & true algorithms

– BSSN in Spherical Polar Coords techniques pioneered by T. BSSN in Spherical Polar Coords techniques pioneered by T. Baumgarte et alBaumgarte et al

● SENR: Extend ideas to support arbitrary, SENR: Extend ideas to support arbitrary, dynamicaldynamical coords coords● Memory Efficiency Is Key FocusMemory Efficiency Is Key Focus: : Unlock the DesktopUnlock the Desktop

● Get public involved → ~10,000x more GW throughput!Get public involved → ~10,000x more GW throughput!

● Bottom line: Bottom line: Maximize science with minimal human & computational Maximize science with minimal human & computational resourcesresources

Documents

SENR/NRPy: A Next-Generation, Dynamical Reference Metric ...astro.phys.wvu.edu/zetienne/SENR/Etienne_ET_Workshop_2017__SENR_NRPy.pdfSENR/NRPy: Motivation SENR/NRPy = Simple, efficient,