Dey-Mittra ADI: An absolutely stable method for Maxwell’s Equations with Embedded Boundaries Travis Austin John Cary, David Smithe Tech-X Corporation Funded

Dey-Mittra ADI: An absolutely stable method for Maxwell’s Equations

with Embedded Boundaries

Travis AustinJohn Cary, David Smithe

Tech-X Corporation

Funded by US-DOE grants DE-FG02-05ER84172, DE-FC02-07ER41499, and FA9451-06-D-0115/002.

ComPASS MeetingBoulder, CO

Tuesday, October 6, 2009

Overview

• Background• Motivation

– Dey-Mittra Cut-Cell Stability

• Alternating-Direction Implicit (ADI) Methods– Divergence-preserving– Tridiagonal Solves (Smithe, Cary, Carlsson, Ovtchinnikov)

• Dey-Mittra ADI Method– Implementation– Frequency Extraction Results– Performance

• Future Work

Ex

Ex

Ez Ez

EyEy

Ey

Bx

By

Bz

Ez

Background

Maxwell’s Equations:

Yee Method:

Background

Courant-Friedrichs-Lewy Stability Condition:

Yee Method (Faraday’s Law):

MotivationDey-Mittra Cut-Cell Stability

(a) Dey-Mittra Approach (b) Stairstep Approach

- Only change Faraday update

MotivationDey-Mittra Cut-Cell Stability

Dey-Mittra-Induced Stability Condition:

Determination of fDM:

• fractional value between 0 and 1• based on stability derived from Gershgorin circle theorem• too small cut-cells make time step prohibitively small• an implicit method would overcome these restrictions

• There are four 2nd-order accurate variants of the ADI algorithm, depending on the order of the operands:

• ZCZ is the first. We investigated the last …

Alternating-Direction Implicit Methods

Divergence-Preserving Form

• The operator P+M is the curl operator, so for Yee-cell

• Of the four ADI combinations, only the last form, DP, can be algebraically manipulated to show that its final operation is equivalent to a finite-difference curl

• Thus it is divergence preserving, for source S.



• Full details are in the paperD. N. Smithe, J. R. Cary, J. A. Carlsson, ”Divergence preservation in the ADI algorithms for electromagnetics,” J. of Comp. Physics 228, 7289 (2009).



• Each domain does forward-solve over its domain

• Passes boundary data to single process global solve

• Receives data, then back-solves over its domain


Tridiagonal Solves

• Remedy is Concurrent Divide & Conquer

• In 2-D and 3-D there are multiple 1-D solves. Global solves are distributed across the processes.


Tridiagonal Solves

• For good scaling, the local backward solve must cover latency.

N # cells in process• Longer 1-D

dimension, N1/2 rather than N1/3, means more time to cover latency.


Tridiagonal Solves

• There was good scaling as long as Ncells

(1/Ndim) 64.

Implies typically good scaling for ADI in both 2-D and 3-D.

• On office linux cluster, needed Ncells

(1/Ndim) 128 for good scaling

Implies good scaling for 2-D, and marginal scaling for 3-D.

(Ideal = dotted line)


Tridiagonal Solves

• Accelerator devices have boundaries which are nonconvex. This breaks each row tri-diagonal solve into several solves.

• Simply unit-fill the diagonals of rows for fully exterior field components, and set RHS source to zero.

Dey-Mittra ADIImplementation

• Dey-Mittra is a metallic cut-cell algorithm giving 2nd-order accurate global solutions.

• Modify Faraday’s law to use non-metallic electric line length and magnetic flux area.

Anon-metallic

lnon-metallic


Division by small area limits algorithm. Reduction in time step is significant, 0.5t for

decent results, and 0.25t or even 0.10t for excellent results.

Throws away small cells, leading to occasional “pits” and “scratches” in geometry including particle creation/destruction surfaces.

Using ADI can eliminate both these inconveniences.


• Technically, the stability of ADI requires that the two alternating-direction curl matrix operators, P and M, be anti-symmetric.

• The Dey-Mittra length/area factors appear to destroy this anti-symmetry.

• However, anti-symmetry can be recovered by solving for re-scaled fields. Then removing scale factors.

• Bottom-line: OK to use Dey-Mittra difference matrix even in non-anti-symmetric matrices.E.g., it’s still stable. (Even when matrix element )


Testing modes of A6 magnetron.

Dey-Mittra ADIFrequency Extraction Results

Fre

qu

ency

(H

z)

1/Nx2

2nd-order accuracy verified, even for t that is 8 times the normal Courant limit.



Method investigated in 3D for A15 cavity

Magnetic Field (z-component)

tf = 343.06 psTime Steps = 2000

Electric Field (x-component)

• Time Step = 2.0 DtCFL

• fDM = 0.0000001 => all cells kept in the simulation

Dey-Mittra ADIPerformance: 2D

Simulation Parameters:

• Single CPU Results• 2D A6 Magnetron Benchmark• 2500 (50x50) cells• GMRES w/ Jacobi preconditioner• 1.358 ns of simulation time

• Explicit: 1280 time steps• ADI-1.0: 320 time steps• ADI-2.0: 160 time steps• ADI-4.0: 80 time steps• ADI-8.0: 40 time steps• ADI-16.0: 20 time steps• ADI-32.0: 10 time steps• ADI-64.0: 5 time steps

• MaxIts raised at 8x CFL from 100 to 500 and then to 1000 at 32x CFL and then to 5000 at 64x CFL

Dey-Mittra ADIPerformance: Strong Scaling

• Full details are in the paperT. M. Austin, J. R. Cary, D. N. Smithe, C. Nieter, ”Alternating Direction Implicit Methods for FDTD using the Dey-Mittra Embedded Boundary Method,” accepted in The Open Plasma Physics Journal.

Dey-Mittra ADI

Summary

• Motivation for ADI methods was discussed in the context of Dey-Mittra method.

• Efficient parallel tridiagonal solves were presented and performance was verified.

• Implementation of ADI for the Dey-Mittra method was introduced and results showed stability beyond CFL.

• Argument made for fast tridiagonal solves.– Multiprocessor?– GPUs?

Documents

Dey-Mittra ADI: An absolutely stable method for Maxwell’s Equations with Embedded Boundaries Travis Austin John Cary, David Smithe Tech-X Corporation Funded