Vortex Particle Simulations of Aircraft Wake Instabilities ... · • sheds a vortex sheet • rolling up into a/several counter-rotating vortex pair(s) • Vortex pair induces •

Vortex Particle Simulations of Aircraft Wake Instabilities on

Massively Parallel Architectures

with : M. Bergdorf, D. Rossinelli, P. Koumoutsakos A. Curioni, W. Andreoni (IBM Research Zurich)

CSE LabComputational Science & Engineering Laboratoryhttp://www.icos.ethz.ch/cse

Philippe Chatelain

VecPar’08

http://www.icos.ethz.ch/cse


http://www.cse-lab.ethz.ch/ June 26 2008

Motivation






Motivation

• COMPUTATION• Massively Parallel Simulations Using Particles

• IBM/BG: Distributed memory (104 - 105 CPUS), with relatively low RAM per node• Achieve sustained O(Tflops) performance






Motivation

• COMPUTATION• Massively Parallel Simulations Using Particles

• IBM/BG: Distributed memory (104 - 105 CPUS), with relatively low RAM per node• Achieve sustained O(Tflops) performance

• PHYSICS• Optimization of aircraft wake decay

• aircraft & lift devices design

• High Reynolds DNS• Capture accurate decay of vortical flows• Physical insight - Basis for turbulent models






• Lifting wing (with finite span)

• sheds a vortex sheet

• rolling up into a/several counter-rotating vortex pair(s)

• Vortex pair induces

• downwash

• rolling moment

➡ Safety hazard

➡ Currently, the limiting constraint on the scheduling of take-offs and landings in airports

Motivation: Aircraft Wakes







• Decay accelerationOutboard loading

Inboard loading







• Decay acceleration

• Modify lift distribution

Outboard loading

Inboard loading









• change number and configuration of vortex pairs

Outboard loading

Inboard loading










• affects trajectories, growth rate of instabilities and decay

Outboard loading

Inboard loading











• Actively perturb lift distribution, flow

• trigger instabilities

Outboard loading

Inboard loading

Leweke & Williamson, 1998













• Decay prediction

Outboard loading

Inboard loading














• Decay prediction

• Vortex tracking

Outboard loading

Inboard loading







Outline

• Motivation

• Vortex Particle Method

• Implementation

• Medium Wavelength Instability

• Conclusions and Outlook






Vortex Particle Method






Vortex Particle Method• GOVERNING EQUATIONS

• Navier-Stokes, incompressible! · u = 0DuDt

= "1!!p + "!u







• Navier-Stokes, incompressible

• Vorticity form

! · u = 0DuDt

= "1!!p + "!u

D!

Dt= (! ·!)u + "!!

! · u = 0! = !" u








• Vorticity form

• with velocity field

! · u = 0DuDt

= "1!!p + "!u

D!

Dt= (! ·!)u + "!!

! · u = 0

u = !" !!! = !"

" · ! = 0

! = !" u








• Vorticity form


• DISCRETIZATION

! · u = 0DuDt

= "1!!p + "!u

D!

Dt= (! ·!)u + "!!

! · u = 0

u = !" !!! = !"

" · ! = 0

! = !" u








• Vorticity form


• DISCRETIZATION

• Particles: position xp and strength

! · u = 0DuDt

= "1!!p + "!u

D!

Dt= (! ·!)u + "!!

! · u = 0

u = !" !!! = !"

" · ! = 0

!p =!

Vp

" dV ! "p Vp

! = !" u








• Vorticity form


• DISCRETIZATION

• Particles: position xp and strength

• EVOLUTION EQUATIONS

! · u = 0DuDt

= "1!!p + "!u

D!

Dt= (! ·!)u + "!!

! · u = 0

u = !" !!! = !"

" · ! = 0

!p =!

Vp

" dV ! "p Vp

dxp

dt= u(xp)

d!p

dt=

!(" ·!)u(xp) + !!2"(xp)

"Vp

! = !" u












• Particle methods ARE NOT MESH-FREE








➡ Particle distortion = loss of convergence

Vortex Particle MethodACCURATE







➡ Particle distortion = loss of convergence

Vortex Particle MethodACCURATE

Particle Method Particle MethodDistortion handled by periodic regularization

Incompressible Euler Equations in 2D:(u-ω) formulation,evolution of an axis-symmetric vortex patchexact solution is steady

dxp

dt= u(xp, t)

d!p

dt= 0






Remeshed Particle Methods

Qnewp =

!

p!

Qp!M(j h! xp!)

M(x)Interpolation Kernel• Moment conserving• Tensorial Product of 1D kernelse.g. high order B-splines






• Remesh : reinitialize particles onto regular locations


Qnewp =

!

p!

Qp!M(j h! xp!)









Qnewp =

!

p!

Qp!M(j h! xp!)









Qnewp =

!

p!

Qp!M(j h! xp!)









Qnewp =

!

p!

Qp!M(j h! xp!)


• Mesh also used for the efficient computation of Right-Hand Side






Vortex Particle Mesh Methods






• Hybrid scheme

• Mesh: RHS evaluations







• Hybrid scheme

• Mesh: RHS evaluations• Differential operators (F.D.)

• Fast Poisson solver (Fourier)


!ij

uij

!!ij






• Hybrid scheme



• Particles only handle advection


D!

Dt=

!!

!t+! · (!u)

!ij

uij

!!ij






• Hybrid scheme




• Particles and Mesh communicate through interpolation


D!

Dt=

!!

!t+! · (!u)

!ij

uij

!!ij

!ij!p






• Hybrid scheme






D!

Dt=

!!

!t+! · (!u)

!ij

uij

!!ij

!ij!p

uij(! ·!)uij

!!ij !!p

(! ·!)up

up






• Hybrid scheme






D!

Dt=

!!

!t+! · (!u)

!ij

uij

!!ij

!ij!p

uij(! ·!)uij

!!ij !!p

(! ·!)up

up

•Stability properties of particle method unaffected: relaxed advection CFL

•Particle-Mesh interpolation: Efficient vectorizing loops, data alignment






Implementation www.cse-lab.ethz.ch/software.html





http://www.cse-lab.ethz.ch/software



Implementation• Code is client of the PPM library,

Parallel Particle Mesh Library (Fortran 90, MPI)

www.cse-lab.ethz.ch/software.html

PPMers: I. Sbalzarini, J. Walther, M.Bergdorf, P. Chatelain, S. Hieber, E. Kotsalis, P. Koumoutsakos, F. Milde, M. Quack, B.

Hejazi Alhosseini

Sbalzarini et al, JCP 2006








• Here:

• Mesh-based domain decomposition



Hejazi Alhosseini









• Here:


• Data mappings

• particles: advected between subscommunication is flow dependent

• mesh values: ghosts for FD stencils



Hejazi Alhosseini









• Here:


• Data mappings





Hejazi Alhosseini









• Here:


• Data mappings



• Poisson solver

• 3D FFT

• Data Transposition uses PPM decompositions and mappings



Hejazi Alhosseini









• Here:


• Data mappings



• Poisson solver

• 3D FFT

• Data Transposition uses PPM decompositions and mappings



Hejazi Alhosseini


local:send

global:alltoallv

Chatelain et al, CMAME 2008






Implementation• Client and library tuned for BG/L• Weak and strong scalabilities

• Drop at 4k: Poisson solver not sole culprit• Overhead in data mappings, latency in particle advection?

IBM T. J. Watson Center, Yorktown Heights, NJIBM Zurich Research Laboratory

Per-CPU problem sizeWithout Poisson solver

512 1024 2048 4096 8192 163840

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

NCPUS

ηwe

ak

2048 4096 8192 163840

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

NCPUS

ηst

rong

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Mpe

r CPU

/ 10

6

Weak scal. Strong scal.

~ 400000 particles per CPU






512 1024 2048 4096 8192 163840

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

NCPUS

ηwe

ak

2048 4096 8192 163840

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

NCPUS

ηst

rong

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Mpe

r CPU

/ 10

6



1010






512 1024 2048 4096 8192 163840

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

NCPUS

ηwe

ak

2048 4096 8192 163840

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

NCPUS

ηst

rong

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Mpe

r CPU

/ 10

6



1010First prod. run


• An interesting candidate...

• Counter-rotating vortex pairs

• generated by outboard loading

Aircraft Wakes









• or large horizontal tail plane

Aircraft Wakes










• Medium wavelength instability λ ~ span

Aircraft Wakes










• Medium wavelength instability λ ~ span

• secondary vortices wrap around primary ones

• reconnections of vortices with different circulations

Aircraft Wakes

Ortega, J.M. et al., JFM 2003

Durston D.A. et al., J Aircraft 2005






• Long domain: length = long wavelength • Problem is periodic in all directions• Initial condition

• Vortices straight• Initiation by ambient noise

Medium Wavelength Instability






• Long domain: length = long wavelength • Problem is periodic in all directions• Initial condition

• Vortices straight• Initiation by ambient noise


Physical parameters

• ReΓ = 6,000• Circulation ratio Γ2/Γ1 = -0.35• Span ratio b2/b1 = 0.5• Domain Lx = 10 b1

• Time = 0.35• Ambient white noise uRMS= 0.5%

CPU parameters

• IBM BlueGene/L• 4096 CPUS• 100 hours

Numerical parameters

• ~1.6 109 particles• 1024 x 768 x 2,048 grid• 10,000 time steps• RK3 Low-storage• 4th order FD







0 0.05 0.1 0.15 0.2 0.25 0.3 0.35−6

−5

−4

−3

−2

−1

0

t/to

E(k x)/E

0

kx = 0kx = 1kx = 8kx = 9kx =10kx =11kx =12

• Cross-flow energy

0 0.05 0.1 0.15 0.2 0.25 0.3 0.350.4

0.5

0.6

0.7

0.8

0.9

1

t/t0

E transverse(t)/E

transverse(0

)

Long

Medium






• Energy spectra

• Mode λ/b1 = 0.86 - 0.943

• Experimental λ/b1 = 0.9 - 1.3• Bursts when Ω-loop feet come together


0 0.05 0.1 0.15 0.2 0.25 0.3 0.35−6

−5

−4

−3

−2

−1

0

t/to

E(k x)/E

0

kx = 0kx = 1kx = 8kx = 9kx =10kx =11kx =12


0 0.05 0.1 0.15 0.2 0.25 0.3 0.350.4

0.5

0.6

0.7

0.8

0.9

1

t/t0

E transverse(t)/E

transverse(0

)

Long

Medium






• Energy spectra

• Mode λ/b1 = 0.86 - 0.943

• Experimental λ/b1 = 0.9 - 1.3• Bursts when Ω-loop feet come together

• Visualization: volume rendering of |ω|• Parallel CPU-GPU ray-casting


0 0.05 0.1 0.15 0.2 0.25 0.3 0.35−6

−5

−4

−3

−2

−1

0

t/to

E(k x)/E

0

kx = 0kx = 1kx = 8kx = 9kx =10kx =11kx =12


0 0.05 0.1 0.15 0.2 0.25 0.3 0.350.4

0.5

0.6

0.7

0.8

0.9

1

t/t0

E transverse(t)/E

transverse(0

)

Rossinelli D. et al., SIGGRAPH08

Long

Medium






Instability : Linear phase t=0.21






Instability : Linear phase t=0.21






Instability : Reconnections t=0.25






Instability : Reconnections t=0.25






Instability: Propagation t=0.27






Instability: Propagation t=0.27






Instability: Decay t=0.35






Instability: Decay t=0.35

Large Vortex Pair

Series of Vortex Rings






Conclusions & Outlook

• Vortex Particle - Mesh Methods• Efficiency• Scalability on Massively Parallel Architectures...

• is good but could be better...• Mapping latency for particles• 3D FFT...

• Reference large-scale DNS of wake instabilities






• Development







• Development

• Optimization/clean-up of global data mappings







• Development


• Unbounded Poisson solvers in Fourier space







• Development



• Applications







• Development



• Applications

• Spatially developing flow







• Development



• Applications


• forcing term accounts for wing lift, wake and jets







• Development



• Applications



• capture roll-up, merger of vortex pairs







• Development



• Applications




• Large Eddy Simulation







• Development



• Applications




• Large Eddy Simulation

• Optimization of wake configurations with Evolution Strategies






Documents

Vortex Particle Simulations of Aircraft Wake Instabilities ... · • sheds a vortex sheet • rolling up into a/several counter-rotating vortex pair(s) • Vortex pair induces •