Computational ChallengesIn

Electromagnetics

Computational ChallengesIn

Electromagnetics

Prof. N. BalakrishnanAssociate Director

Indian Institute of ScienceBangalore

Prof. N. BalakrishnanAssociate Director

Indian Institute of ScienceBangalore

ATIP 1st HPC in India Workshop 2009Supercomputing 2009

Portland, OR, USANovember 20th, 2009

ATIP 1st HPC in India Workshop 2009Supercomputing 2009

Portland, OR, USANovember 20th, 2009

Traditional View of Computational Electromagnetics

• It is a technology for the defence • It is more about waveguides, Circulators,

cables, big antennas, radars• The mathematics is tough and hence most

of the people in microwaves are “hardcore” tinker/ mechanics

• You should be a plumber before you take up microwaves

CEM are part of your life more than IT now

CEM becomes quantitative and adds to Grand Challenges in Computer Science

• Numerical Circuit Simulations- EMI inside a chip

• EM Modelling• Multitude of simulations encompassing PDEs

and IEs in Frequency Domain and Time Domain- Birth of Compuatational Electromagnetics- even before CFD and FEM

• And now the new buzz words- TeraHetrz

CEM drives the craving for more compute power

Computational Techniques in EM

DifferentialEquation

FD

FEMFEM

FDTD

AsymptoticTechniques

GO

GTD

UTD

PO

PTD

SBR- Frequency domain- Time domain

IntegralEquation

MOM

MOT

TLM

T-Matrix

FMM

Finite Difference Time Domain(FDTD)• 3D discretization

• Results in Sparse Matrix

• Gives excellent visualizaion

• Good for high resolution mapping of the aircraft

• Absorbing boundary condition

• Our contribution- PML and Lossy Media

Discritisation Details for Representative Aircraft

Frequency 1 GHzTotal No. Layers in X 183 Total No. Layers in Y 379 Total No. Layers in Z 539 No. PML Layers used 18

Discrimination Discrimination

ofof

Radar Targets Radar Targets

with with

Minor VariationsMinor Variations

Triangular Patch modelling of an aircraft with stores

• missile length = 3.2 m

A/c with stores modeled as 63 surfaces

48585 triangular patches

Convolution OutputConvolution Output

Aircraft with missile: Te = 26 lmAircraft with missile: Te = 26 lm

Aircraft with missile: Te = 30 lmAircraft with missile: Te = 30 lm

Convolution output

in late-time

is minimum

for the basic aircraft

Radar Cross Section EstimationAnd

Control

Method of Moments(MOM)

Hotspot Analysis

Discritisation Details forRepresentative Aircraft

Element type: TriangleNo. of Edges 62202 (Unknowns)Criterion 4 cells /lambda

On a 256 node Cluster 3 hrs

Today we need to solve 15 million elements-

RCS Prediction of a Model Aircraft Using Method of Moment (MoM)

Surface Area of the Aircraft : 47.0135851 sq. meters

Wing Span: 10.8 metersAircraft is along the x-axis:Front angle of the wing w.r.t x-axis : Rear angle of the wing w.r.t x-axis :

For all the frequencies the aircraft body is discretized with lambda by 5

deg122deg142

Complexity of Integral Eq. Techniques

• Surface Area of a typical Aircraft : 40 sq. meters

• Wing Span: 2.6 m (Approx)

• At 10 GHz the wave length is 3 cm

• Cell size at lambda/10 discretization is 0.04 Sq cm

• Number of cells = 10 Million

• At 3 GHz = 100,000

• The matrix size at 10 GHz is 15M X 15M

HARDWARE OVERVIEW

IBM Bluegene:

•4096 2-way SMP nodes (8192 processors)

• IBM PowerPC processors operating at 700 MHz

• 1 GB main memory per node with a total of

4 TB for the cluster.

• Gigabit network with Cisco 6500 Gigabit switch.

No Frequency

(GHZ)

Number of Nodes

Number of Edges

Number of Faces

Number of unknowns

Number of processors

used

Time

1 0.25 3417 10245 6831 10245 512 3 min.

2 1 21653 64953 43304 64953 512 1.1 hrs

3 2 52230 156683 104456 156683 1024 5.5 hrs

4 2.5 77379 232131 154754 232131 1024 13.5 hrs

Table for MoM Technique:

Alternative Techniques

• Finite Element Method

• Physical Theory of Diffraction with Shooting and bouncing of Rays

• Multilayer Fast Multipole Method

Machine Specification:

Tyrone systems

Two physical CPUs with a total of 8 cores.

Intel® Xeon® CPU

CPU GHz 2.88

Main Memory : 32GB

Alternative- MLFMMAlternative- MLFMM

Frequency Number of unknowns

Method Number of processors

Clock rate

(MHz)

Total CPU Time (hrs)

250 MHz 14682 MLFMM 8 2883.503 0.028

Frequency Number of unknowns

Method Number of processors

Clock rate

(MHz)

Total CPU Time (hrs)

1 GHz 55260 MLFMM 8 2833.503 0.593

Frequency Number of unknowns

Method Number of processors

Clock rate

(MHz)

Total CPU Time (hrs)

2 GHz 223440 MLFMM 8 2833.503 2.623

Frequency Number of unknowns

Method Number of processors

Clock rate

(MHz)

Total CPU Time (hrs)

2.5 GHz 345771 MLFMM 8 2833.503 5.097

Physical Theory of Diffraction with Shooting and Bouncing Rays

• Asymptotic technique

• Not rigourous

• Works well at high frequencies

• Not computationally expensive

Discritisation Details

Triangular Elements 104948

No. of Nodes 54365 No. of Edges 158318

Others work – current state of the art

Indian Institute of Science

object Size Freq. Type No of unknowns Method

NASA Flamme

880λ 440 GHz Metallic 203,664,320 MLFMM

Polarization Θ

Illumination 90˚

No of Iteration 154

Total Time(min.) 2922

Computing platform

Intel Xeon Dunnington processors with 2.40 GHz clock rate. 16 computing nodes, each node has 48 GB of memory and multiple processors, four cores per node (a total of 64 cores)

Indian Institute of Science

object Size Freq. Type No of unknowns Method

Aircraft carrier

128λ 150 MHz Metallic 415,316 MoM

Length 256m

Width 66m

Height 47m

Total Time(min.) 946

Indian Institute of Science

Computing platform

One head node, 64 compute nodes and three Infiniband switches, head node has two quad-core Intel Xeon E5450 3.0 GHz processors, 16 GB of RAM, Each compute node has two quad-core Intel XeonE5450 3.0 GHz processors, 16 GB of RAM

How do we scale to 10’s of GHzHow do we scale to 10’s of GHz

CEM: Present days Requirements

* Electrically large in size Complex geometrical Shape High Frequency Analysis Multi-layered composite body Tera Hertz thru Human Body

LAMENT• Before an aircraft design is complete, we may need around 1000’s

of runs at various frequencies of large complex objects- coated, penetrable etc

• Rigorous Methods such as MoM are not known to scale to the 15 M variables problems in accuracy nor do we have machines that can be used for computing RCS at > 10 GHz frequencies

• The Asymptotic Techniques require extensive validation with real measurements

• For Homeland Security applications and sensing for materials – it is high frequency, layered though small in size

• Large levels of validation needed• More importantly new architectures, new physics and newer

techniques are needed