Domain Decomposition Methods for FEM Modeling of Large ...€¦ · Interconnecting method...

Domain Decomposition Methods

for FEM Modeling of Large-Scale

Phased Arrays

J.-M. Jin

Center for Computational Electromagnetics

Department of Electrical and Computer Engineering

University of Illinois at Urbana-Champaign

Urbana, Illinois 61801-2991

July 30, 2012

Antennas Mini-Symposium

AN/FPS-85 Spacetrack Radar Ballistic Missile Early Warning

System

• Extremely high directivity/resolution

• Rapid electronic scanning over 120o

• Simultaneous tracking of multiple

targets

13 stories high

5,134 transmitters/5,928 antennas

4,660 receivers/19,500 antennas

Phased-Array Antennas

X-band Phased-Array Antenna Phased-Array Antenna with

Distributed Feed Network

MMIC phase-shifters & T/R modules

256 elements

Applied Radar, Inc.

New applications:

• Wireless communications

• Video games

Phased-Array Antennas

Very large-scale inhomogeneous problem with non-uniform excitation

Requires an excessively long computation time and extensive resources

Can only handle problems with a limited size on a single computer

Requires parallel computers to handle relatively large problems

Phased-Array Antennas

Computational Challenges

FEM Basics

Vector wave equation:

Boundary conditions:

on

1imp00

2

0 JEE Zjkk r

r

Dn on0ˆ E

NN

r

nnjk

n

on)ˆ(ˆ

1ˆ 0 KEE

2

0

00 0 imp

1 1ˆ( ) ( ) ( ) ( )

ˆ ˆ( ) ( )

D

N

r

r r

N

k d n d

jkn n d jk Z d

W E W E W E

W E W K W J

Weak-form representation:

FEM Basics

Spatial discretization:

edge

1

N

i

iiENE

FEM matrix equation:

where

[ ]{ } { }K E b

2

0

0

1( ) ( )

1ˆ ˆ( ) ( )

N

ij i j r i j

r

i j

K k d

jk n n d

N N N N

N N

0 0 imp

N

i i i Nb jk Z d d

N J N K

Decompose the computational domain

into many small subdomains

Reduce a 3D global problem to an explicit

interface problem for a dual unknown

with a much smaller size

Construct a coarse grid problem defined

at the subdomain corner edges

Suitable for general 3D problems using

parallel computing

Utilize geometrical redundancies in

finite array-like structures

The ElectroMagnetic Dual-Primal Finite Element Tearing &

Interconnecting method (FETI-DPEM)

FETI-DPEM Overview

Advantages of FETI-DPEM

1. Able to solve very large 3-D problems because it reduces

a 3-D global problem to a much smaller explicit interface

problem.

2. Extremely suitable for parallel computation because

subdomains are fully decoupled.

3. Fast convergence rate for solving the interface problem by

enforcing the continuity at the corner edges.

4. Able to solve finite periodic problems with billions of

unknowns very efficiently on a single workstation.

Domain Decomposition Methods

i j

i j

FETI-DPEM

w/ weak BC

FETI-DPEM

w/ strong BC

FETI-DPEM

w/ mixed BC

• Corner unknowns

belong to interfaces

• Field continuity is

enforced implicitly

• Interface meshes

don’t have to match

• Corner and interface

unknowns are separated

• Field continuity is

enforced explicitly

• Interface meshes

have to match

• Corner and interface

unknowns are separated

• Field continuity is

enforced explicitly at the

corners and implicitly at

the interfaces

c

c

c

c

FETI-DPEM1 Formulation

Computational domain

partitioned in a check-board

fashion.

ˆ in

ˆ jnij

c

c

c

c c

c

c

c

c

All degrees of freedom related

to the corner-edges are global

primal variables: 1[ ... ]sN T

r r cE E E E

[ ] [ ]s s s s T s s T

V I c r cE E E E E E

In the sth subdomain the primal

variables are grouped as:

ji

Introducing a Boolean matrix ,

which extracts the corner dofs of

the sth subdomain as:

s

cB

s s

c c cB E E

FETI-DPEM1 Formulation

s 2 s

0 0 0

1in s s

r imps

r

E k E jk Z J

Vector wave equation:

FEM matrix equation: s s s sK E f

s

c

ˆ in

s

I

s

impJ

s

I

s

I

c

c

c

1 1ˆ ˆ ˆ ˆ on s s q q s s q q sq

Is q

r r

n E n E n E n E

Boundary conditions:

0

ˆ 0 on

ˆ 0 on

1ˆ ˆ ˆ 0 on

s s s

PEC

s s s

PMC

s s s s s s

ABCs

r

n E

n E

n E jk n n E

Interface condition:

Analysis of subdomain

problem.

s

c

ˆ in

s

I

s

impJ s s 2 s s

0

s s

0

1( ) ( )

ˆ ˆ( ) ( )

T

s

sI

s T

r

r

T

K N N k N N dV

jk n N n N dS

s

0 0 s

s s

impf jk Z N J dV

Subdomain FEM matrix :

Subdomain excitation

vector:

Interface contribution:

1ˆ

T

I

s s s s s

rs

r

N n E dS B

is a signed Boolean matrix that

extracts the interface dofs.

s

rB

s

I

s

I

c

c

c SubdomainFEM matrix equation:

s s s sK E f

FETI-DPEM1 Formulation

Using the “r” and “c” notations:

T

s s

rr rcs

s s

rc cc

K KK

K K

s

rs

s

c

ff

f

The subdomain problem in the matrix form: T

T

s s s s srr rc r r r

ss s scrc cc c

K K E f B

EK K f

Note: λ is unknown

1

( )Ts s s s s s s s

r r r rr r r rc cB E B K f B K E

The electric field at the interfaces of the sth subdomain:

The subdomain level corner dofs related system:

1 1 1

( )T T T Ts s s s s s s s s s s s

cc rc rr rc c c rc rr r rc rr rK K K K E f K K f K K B

FETI-DPEM1 Formulation

Subdomain system equations are assembled using

the electric field continuity equation:

1

0sN

s s

r r

s

B E

The corner-related dofs are assembled globally as a

super finite element system, which couples the

whole computational domain:

1 1 1

1 1 1

( ) ( ) ( )s s s

T T T TN N N

s s s s s s s s s s s s s s s T

c cc rc rr rc c c c c rc rr r r rr rc c

s s s

B K K K K B E B f K K f B K K B

T

cc c c rcK E f F

FETI-DPEM1 Formulation

where

By eliminating and , the interface equation for

solving for the dual variable λ is given as:

1 1T

rr rc cc rc r rc cc cF F K F d F K f

1

1

1 1

1 1

s s

s s

N Ns s s sT

rr rr r rr r

s s

N Ns s s s s

rc rc r rr rc c

s s

F F B K B

F F B K K B

1

1

1 1

1 1

( )

s s

s sT T T

N Ns s s s

r r r rr r

s s

N Ns s s s s s s

c c c c c rc rr r

s s

d d B K f

f f B f B K K f

1

1 1

( ) ( )s s

TN N

s s s s s s T s s s

cc cc c cc c rc c rr rc c

s s

K K B K B K B K K B

rE cE

FETI-DPEM1 Formulation

( ) ( ) ( ) ( )

( ) ( )

1 1ˆ ˆi i j j

i j

r r

n n

E E Λ

( ) ( ) ( ) ( )ˆ ˆi i j jn n E E

Subdomain interface conditions:

Vector wave equation:

FETI-DPEM1 Formulation

2

0 0 0 imp

1r

r

k jk Z

E E J

i j

c

c

c

c

Y. Li and J. M. Jin, “A vector dual-primal finite element tearing and interconnecting method for solving 3-

D large-scale electromagnetic problems,” IEEE Trans. Antennas Propagat., vol. 54, no. 10, pp. 3000-

3009, Oct. 2006.

Final interface system for dual unknowns:

1 1

rr rc cc cr r rc cc cK K K K f K K f

Global interface system:

rr rc c rK K E f

Global corner system:

cc c c crK E f K

FETI-DPEM1 Formulation

i j

c

c

c

c

31x31 Vivaldi Antenna Array

Convergence history for solving

the interface problem of various

Vivaldi antenna arrays at 3 GHz.

Numerical scalability Radiation patterns

Radiation patterns of the 100 x

100 Vivaldi antenna array at 3

GHz for θ= 45º, φ= 0º scan.

Finite Vivaldi Antenna Array

Size of array

Total number of unknowns

(Million)

Total number of

dual unknowns

Memory

(MByte)

Interface solving time

(h:min:s)

Iteration number

Total time

(h:min:s)

10 x 10 5.7 0.1M 500 00:00:10 19 00:06:36

31 x 31 39.7 0.7M 541 00:01:21 19 00:11:44

50 x 50 100.9 1.9M 685 00:03:02 18 00:20:23

75 x 75 224.2 4.4M 1300 00:06:01 17 00:40:11

100 x 100 396.2 7.8M 2100 00:10:54 17 01:12:30

150 x 150 887.3 17.6M 3800 00:28:58 20 02:18:42

200 x 200 1572.2 31.5M 6700 01:09:36 20 04:16:35

250 x 250 2452.3 49.4M 10000 02:35:07 20 08:34:19

300 x 300 3528.4 70.8M 15000 05:27:15 20 12:27:14

Computation time & memory usage of the FETI-DPEM method for

various Vivaldi antenna array simulations

Finite Vivaldi Antenna Array

Numerical Scalability

Numerical scalability of the FETI-DPEM1 method. (a) Numerical

scalability with respect to frequency for the 100-by-100 array. (b)

Numerical scalability with respect to the array size at different

frequencies.

(a) (b)

100x100 array

( ) ( ) ( ) ( ) ( ) ( )

0( )

1ˆ ˆ ˆj j j j j j

j

r

n jk n n

E E Λ

( ) ( ) ( ) ( ) ( ) ( )

0( )

1ˆ ˆ ˆi i i i i i

i

r

n jk n n

E E Λ

Subdomain interface conditions:

Equivalent interface conditions: ( ) ( ) ( ) ( ) ( )

0

( ) ( ) ( ) ( ) ( )

0

ˆ ˆ2

ˆ ˆ2

i j j j j

i j i i i

jk n n

jk n n

Λ Λ E

Λ Λ E

Vector wave equation:

FETI-DPEM2 Formulation

2

0 0 0 imp

1r

r

k jk Z

E E J

Final interface system for dual unknowns:

1 1

rr rc cc cr r rc cc cK K K K f K K f

Global interface system:

rr rc c rK K E f

Global corner system:

cc c c crK E f K

FETI-DPEM2 Formulation

Y. Li and J. M. Jin, “A new dual-primal domain decomposition approach for finite element simulation of

3D large-scale electromagnetic problems,” IEEE Trans. Antennas Propagat., vol. 55, no. 10, pp. 2803-

2810, Oct. 2007.

Numerical Scalability

Numerical scalability of the FETI-DPEM2 method. (a) Numerical

scalability with respect to frequency for the 100-by-100 array. (b)

Numerical scalability with respect to the array size at different

frequencies.

(a) (b)

100x100 array

Mutual Coupling Between Arrays

Geometry of two 9 x 9 patch antenna

arrays recessed in dielectric cavities

on a planar platform

Normalized received power

patterns

23.9 million primal unknowns

1.2 GB RAM, 5 hours on a

single 1.5-GHz processor

Photonic Crystal Nanocavity

x

y

Number of layers

Number of subdomains

Number of unknowns

Memory

(MByte)

Simulation time

14 919 7.2M 1100 14m57s

Photonic Crystal Nanocavity

Geometry of the PhC nanocavity analyzed (H. Y. Ryu, 2002).

The yellow region is the dielectric slab with the refractive index

of n = 3.4 and the red circles represent the air holes. The slab

thickness t = 0.4a.

Energy stored in the cavity as a function of

frequency. The locations of the energy

peaks represent the resonant frequencies of

the resonant modes.

Dipole Hexapole

Quadrupole

Monopole

Photonic Crystal Nanocavity

Q ~ 100,000

Q ~ 5,000

Improved Waveguide Bend

Problem analyzed

Number of subdomains

Number of unknowns

Memory

(GByte)

Simulation time

Original 487 11.7M 1.7 47m34s

Modified 487 11.7M 1.8 47m51s

Patch Antenna on a Platform

Normalized radiation patterns in the

H-plane at 3.3 GHz for a patch

antenna on a cylinder with a wing.

Speedup = 4 ×

T4/TNp

Non-realistic Scattering Example

Original object

Subdomain decomposition

Parallel speedup

Current distribution

at 300MHz

Geometry of a 31x31 cavity-backed patch antenna array on an infinitely

long PEC cylinder. (a) The patch antenna array on platform. (b) The

geometry of the array element.

(a) (b)

31-by-31 Patch Array on Curved Surface

Radiation patterns and active

reflection coefficients (mid-row)

for the 31x31 cavity-backed patch

antenna array on various

platforms.

31-by-31 Patch Array on Curved Surface

E-plane

H-plane

Patch Antenna Arrays on Battleship

Phased Array on a Platform

Nonconformal FETI-DPEM

(1) Lagrange-multiplier (LM)-based nonconformal FETI-DPEM

(2) Cement-element (CE)-based nonconformal FETI-DPEM

Nonconformal FETI-DP Methods for Large-Scale Electromagnetic

Simulation

Features:

• Both methods implement the Robin-type transmission condition at the subdomain

interfaces;

• Both methods formulate a global coarse problem related to the degrees of freedom

at the subdomain corner edges to propagate the residual error to the whole

computational domain in the iterative solution of the global interface equation;

• The first method extends the conformal FETI-DP algorithm, which is based on two

Lagrange multipliers, to deal with nonconformal interface and corner meshes;

• The second method employs cement elements on the interface and combines the

global primal unknowns with the global dual unknowns.

LM-based Nonconformal FETI-DPEM

s ss s si iii ib ic

s s s s s s s

bi bb bc b b bb

s s s s sci cb cc c c

E fK K K

K K K E f B

K K K E f

1ˆ ˆ ˆ( )s s s s s s s

s

r

n n n

E E Λ

sΛ

{ } { }s s T s

b Λ N

[ ]s

bbB

Note: Different from the conformal FETI-DPEM method, the dual unknown

expanded in terms of a set of curl-conforming vector basis functions as

Therefore, is no longer a Boolean matrix.

Vector wave equation:

Subdomain interface conditions:

2

0 0 0 imp

1 s s s

r

r

k jk Z

E E J

Subdomain matrix equation:

here is explicitly

Equivalent interface conditions: ˆ ˆ( ) ( )

ˆ ˆ( ) ( )

s q s q s s s

q s q

s q s q q q q

q s s

n n

n n

Λ Λ E

Λ Λ E

Note: The global system matrix related to the global interface and corner unknowns is

quite similar to that of the conformal FETI-DPEM scheme, except for some Boolean

matrices are replaced by real-valued sparse matrices.

CE-based Nonconformal FETI-DPEM

Subdomain interface conditions:

Subdomain matrix equation:

Equivalent interface conditions:

0 0

1 1ˆ ˆ ˆ ˆ ˆ ˆ( ) ( )s s s s s q q q q q

s q

r r

n jk n n n jk n n

E E E E

Cement auxiliary variable:

ˆ ( )s s sn b E00

0

0 0 0

0 0

s s ss s si i iii ib ic

s ss s s s sb bbi bb bb bc b

s s ss sbb bb

s s s ss sci cb cc cc c

E f fK K K

E fK K B K f

D C gb g

K K K fE f

neighbor( )

{ } [ ]

q

bs q q

s s qq s

Eg U V

b

[ ]{ } [ ]{ } { }rr rc c rK u K E f

[ ]{ } { } [ ]{ }cc c c crK E f K u

Note: The global system matrix equation is written in the following form, which can be solved

by eliminating the global corner unknowns first.

Nonconformal Global Corner System

slave

cNmaster

cN

ccorner

slavemastercorner

cornerslave

cornerslave

slave

cNslave

cN

slave master

c cN N

Master and slave corners associated with one

shared cross point, and we choose

master slave

t tE = E

slave

master

slave slave slave

, ,

1

master master master

, ,

1

c

c

N

t c n c n

n

N

t c n c n

n

E

E

E N

E N

slv-slv slave slv-mst master[ ]{ } [ ]{ }cc c cc cG E H E

slv-slv slave slave

, , ,

slv-mst slave master

, , , .

c

c

cc mn c m c n

cc mn c m c n

G dl

H dl

N N

N N

slave slv-slv 1 slv-mst master{ } [ ] [ ]{ }c cc cc cE G H E

In order to remove the conformal-mesh restriction on the geometrical crosspoints, we

impose the Dirichlet continuity condition at the corner in a weak sense as

Basis expansion:

Galerkin testing:

where

Represent slave corner unknowns by

master ones:

Eigenspectra and Convergence Test

Conformal interface and corner

meshes Nonconformal interface and

conformal corner meshes Nonconformal interface and

corner meshes

0 20 40 60 80 10010

-10

10-5

100

Number of Iterations

Resid

ue

LM FETI-DP

CE FETI-DP

0 20 40 60 80 10010

-10

10-5

100

Number of Iterations

Re

sid

ue

FETI-DPEM2

LM FETI-DP

CE FETI-DP

0 20 40 60 80 10010

-10

10-5

100

Number of IterationsR

esid

ue

LM FETI-DP

CE FETI-DP

Finite Vivaldi Array Revisit

Simulation of the 100-by-100 Vivaldi antenna

array at 3 GHz. (a) Broadside scan E-plane

relative pattern. (b) Broadside scan H-plane

relative pattern. (c) Convergence history.

-90 -60 -30 0 30 60 90-60

-50

-40

-30

-20

-10

0

10

(degrees)

Re

lative

Pa

tte

rn (

dB

)

E-plane Relative Pattern

FETI-DPEM2

LM FETI-DP(w/ nonconformal mesh)

CE FETI-DP(w/ nonconformal mesh)

-90 -60 -30 0 30 60 90-60

-50

-40

-30

-20

-10

0

10

(degrees)

Re

lative

Pa

tte

rn (

dB

)

H-plane Relative Pattern

FETI-DPEM2

LM FETI-DP(w/ nonconformal mesh)

CE FETI-DP(w/ nonconformal mesh)

0 10 20 30 40 5010

-4

10-3

10-2

10-1

100

Number of Iterations

Re

sid

ue

FETI-DPEM2

LM FETI-DP(w/ nonconformal mesh)

CE FETI-DP(w/ nonconformal mesh)

105

106

107

108

109

101

102

103

104

105

Total Number of Unknowns

Com

puta

tion T

ime (

Second)

Interface Time

Total Time

105

106

107

108

109

101

102

103

104

105

Total Number of Unknowns

Co

mp

uta

tio

n T

ime

(S

eco

nd

)

Interface Time

Total Time

Scalability of LM-based Nonconformal FETI-DPEM

Vivaldi Antenna Array Photonic Crystal Cavity

9 subs

100 subs

961 subs

10000 subs

81 subs

841 subs

8281 subs

40401 subs

Conclusion

• Domain decomposition methods offer a most

promising approach to modeling large-scale and

multi-scale EM and multi-physics problems

• FETI-DPEM is a unique method, which achieves a

faster convergence through the formulation of a

coarse grid system

• FETI-DPEM is highly parallelizable and has an

excellent scalability

• Both conformal and nonconformal FETI-DPEM have

been developed to model large phased arrays with a

fully exploitation of geometry repetition

• FETI-DPEM has been validated numerically and

experimentally using NRL’s dual-polarized Vivaldi

phased array

Conclusion (cont’d)

• Good agreement has been achieved for the radiation

patterns for all the frequencies and scan angles

considered

• FETI-DPEM has been shown to be a powerful

numerical method for simulating large phased-array

antennas

• FETI-DPEM is also applicable to many other array-

type problems and general problems including

antenna-platform interaction analysis