Daresbury Laboratory Science & Technology Facilities Council

High-Speed Flow Dynamic Research Through High-Order Numerical

Simulation on ARCHER

Scientific Computing Department Science and Technology Facilities Council

Daresbury Laboratory, UK

Dr. Jian Fang

Daresbury Laboratory Science & Technology Facilities Council Outlines

• Background • CFD Code • Shock-wave/turbulent boundary layer interactions • Flow Mechanisms • Next-Gen Code for UKCTRF • Conclusions • Acknowledgements

Background

Shock-Wave/Turbulent Boundary Layer Interaction

Background

Computational Fluid Dynamics

RANS

DNS

LES

DES

DDES x

<u>

• Mean flow field, overall performance. • Small amount of data, very efficient. • Highly relied on turbulence model

• Fully temporal/spatial resolved flow field. • High-accuracy, Lots of data. • Very expensive, Not ready for industrial

applications. • Highly relied on HPC resource and CFD code.

CFD Code • ASTR (Advanced flow Simulator for Turbulence Research)

– High-order FDM on generalized mesh • High-order Dealiasing Compact Central scheme • High-order Low-Dissipative Shock-Capturing Scheme • 3rd-order Runge-Kutta time scheme

– Modern Fortran 90 – Parallelized by using MPI and Hybrid MPI-OpenMP – Collective HDF5 I/O – Tested in ARCHER, Tianhe, Hector, Blue Genes…

PP Solver

Mesh Generation Software Initial Field

grid.h5 flowini3d.h5

flowstate.dat monitor.dat tecfield.plt

flowfield.h5 meanflow.h5 budget.h5 supply.h5

Results

Tecplot/Origin/Excel

To analyze

Impinging Shock-Wave/Flat-Plate Boundary Layer Interaction Flow Passing a Cylinder

-2.0

-1.6

-1.2

-0.8

-0.4

0.0

0.4

0.8

1.2

1.6

2.0

-0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4

Present Results PIV of Parnaudeau et al.

x/D=2

.02

v'v'

y/D

x/D=1

.53

x/D=1

.06

0 20 40 60 80 100 120 140 160 180-1.5

-1.0

-0.5

0.0

0.5

1.0

Present Result Experiment of Norberg

Cp

θ

Turbulent Channel Flow Tip-Leakage Vortex Well resolved wall-turbulence

Benchmarks

Performance on ARCHER

Cores CPU Time I/O Time

120 105.41 5.49 240 53.36 20.08 480 29.44 41.87

1200 12.1 77.74 2400 7.41 122.27 4800 4.39 255.47 9600 2.74 -

ARCHER Nodes

Test Case: 3D Taylor-Green Vortex Mach:0.1, Re=1600 Mesh size: 16.8M=256×256×256

100 1000 100001

10

100

Computing Time HDF5 I/O Time

Tim

e

Cores

100 1000

10

100

1000 Comuting Time HDF5 I/O Time

Cpu

time

Cores

Perfect Acceleration

Performance on ARCHER

Cores CPU Time I/O Time

120 455.64 2.74

240 248.36 8.29

480 138.96 15.3

1200 82.81 10.80

2400 60.65 14.48

4800 51.63 97.22

ARCHER Nodes Cores HyperThread MPI OpenMP CPU Time

256 4 256 4 523.05

256 4 8 128 1629.69

8 - 8 - 17076.09

256 - 256 - 513.281

KNL Nodes

Test Case: DNS of Turbulent Boundary Layer Mach:0.2, Reδ=7500 Mesh size: 71.2M=1980×120×300

Shock-wave/turbulent boundary layer interactions

• Cases studied

Impinging Shock-Wave/Flat-Plate Boundary Layer Interaction

Transonic Airfoil

Supersonic Backward Step

Compression Corner

Transonic Flow Passing a Cylinder

3D Single-Fin Flow

• Mach=2.25；Impinging Angle: α=33.2° • Reδ=41167, Reθ=3148

IM×JM×KM ∆x⋅∆y⋅∆z (10−4) 2800×150×256 4.0×1.00×7.81 ∆xmin

+×∆y1+×∆z+ Lz+

3.9×0.97×7.6 1950

Leading Edge

Impinging Shock

Reflected Shock

Comput. Domain

α=33.2°

Blow & Suction

Ma=2.25

Impinging Shock-wave/flat plate boundary layer interaction

Impinging Shock-wave/flat plate boundary layer interaction

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

0.0 0.2 0.4 0.6 0.8 1.0

Shutts et al. 1955 Bookey et al. 2005 DNS

y/δ

u/u01 10 100 1000

0

5

10

15

20

25

30

u+ VD

y+

DNS

u+ =y+

u+ =1/κ•ln(y+ )+5.0

•Velocity Profile

0 10 20 30 40 50 60 70 80 90 100

0.0

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

2.7

3.0

RMS

Velo

city

Fluc

tuat

ion

y+

u'DRMS

w'DRMS

v'DRMS

Present DNS Wu and Moin, 2009 Spalart, 1981 Pirozzoli et al, 2010 Purteld et al., 1981

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4

0.0

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

2.7

3.0 Present DNS Wu and Moin, 2009 Pirozzoli, et al., 2010 Erm & Joubert, 1991 Purteld et al., 1981

RM

S Ve

locit

y Fl

uctu

atio

n

y/δ

u'DRMS w'

DRMS

v'DRMS

•RMS Velocity Fluctuation Intensity Profile

-1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5-0.5

0.0

0.5

1.0

1.5

2.0

2.5

Experiment of Dupont at Ma=2.3, α=32.4Ο

Experiment of Dupont at Ma=2.3, α=33.2Ο Present Case S

p*

X*

•DNS .vs. PIV Humble, R.A., et al., Exp. Fluids, 2007, 43:173-183.

•Wall pressure

Flow Mechanisms

Turbulent Kinetic Energy

Turbulent structures under zero and adverse pressure gradient

Jian Fang, Zhaorui Li, and Lipeng Lu, An Optimized Low-Dissipation Monotonicity- Preserving Scheme for Numerical Simulations of High-Speed Turbulent Flows, Journal of Scientific Computing, 2013, Vol. 56, pp. 67–95.

Expansion-Compression Corner • Expansion-Compression Corner

– Mach=2.9；Deflection Angle: β=25° – Reδ=20000/40000/80000 – Mesh size: 1420×120×256/2020×120×300/2620×200×400

25°

Density Schlieren Pressure Field

Expansion-Compression Corner

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0 0.2 0.4 0.6 0.8 1.0

Case 1 Case 2 Case 3 Experiment at Re=133000 at x=2δ Experiment at Re=80000

<u>

y/δ

1 10 100 10000

5

10

15

20

25

30 Case 1 at Reθ =1571 Case 2 at Reθ =3171 Case 3 at Reθ =6140Murlis et al. (1982) at Reθ =791Murlis et al. (1982) at Reθ =1368Erm et al. (1985) at Reθ =617

u+ vd

y+

u+vd=1/0.41y+ +5.2

u+ vd=y

+

6140

Mean Velocity Profiles in the Undisturbed BL

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4

0.0

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

2.7

3.0 Case 1 Case 2 Case 3 Purteld et al., 1981 Erm & Joubert, 1991 Wu and Moin, 2009 Pirozzoli, et al., 2010

RM

S Ve

locit

y Fl

uctu

atio

n

y/δ0 10 20 30 40 50 60 70 80 90 100

0.0

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

2.7

3.0 Case 1 Case 2 Case 3

RMS

Velo

city

Fluc

tuat

ion

y+

Purtell et al., 1981 Spalart, 1981 Wu and Moin, 2009 Pirozzoli et al, 2010

Turbulence Velocity Intensities in the Undisturbed BL

0 5 10 15 200.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Case 1 Case 2 Case 3 Experiment at Reδ=40700 Experiment at Reδ=67600 Experiment at Reδ=80000 Experiment at Reδ=194000 LES of El-Askary 2011 LES of Knight et al. 2001

PW/P

∞

xEC CC

0 5 10 15 20

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014 Case 1 Case 2 Case 3 LES of El-Askary 2011 LES of Knight et al. 2001 Experiment at Reδ=194000 Experiment at Reδ=80000

Cf

xEC CC

Wall Pressure

Skin friction

Flow Mechanisms

Iso-surface of |ω| colored with ux

Large-scale Görtler Vortex

Formation of Görtler vortices

Jian Fang, Yufeng Yao, Alexandr A. Zheltovodov, Zhaorui Li, and Lipeng Lu, Direct numerical simulation of supersonic turbulent flows around a tandem expansion-compression corner, Physics of Fluids, 2015, 27, 125104

• Gortler Vortex

3D SWTBLI Hypersonic Single Fin

￭ Mach=5.0；Deflection Angle: β=23° ￭ Re=37×106/m, Reδ=1.4×105

￭ Mesh: 1040×240×1420

•Sketch map of the domain

Fin

3D SWTBLI

Turbulent Coherent Structures & Shock Surface

Near Wall Velocity Streaks

3D SWTBLI

0.4 0.6 0.8 1.0 1.2 1.4 1.6

-3

0

3

5

8

10

13

15

18

20

S1

LES Results at x=83 mm LES Results at x=93 mm LES Results at x=123 mm LES Results at x=153 mm LES Results at x=183 mm Experiment Measure at x=83 mm Experiment Measure at x=93 mm Experiment Measure at x=123 mm Experiment Measure at x=153 mm Experiment Measure at x=183 mm

p W /p

∞

z/x

R1

S2

R2

0 5 10 15 20 25 300

10

20

30

40

50

60

70

80

90 Separation Line Angle ϕ1 of Experiment Reattachment Line Angle γ1 of Experiment Secondary Separation Line Angle ϕ2 of Experiment Separation Line Angle ϕ1 of LES Reattachment Line Angle γ1 of LES Secondary Separation Line Angle ϕ2 of LES

β

•Characteristic Angles •Wall Pressure

Flow Mechanisms

•Instantaneous Vorticity Fluctuations Zone 1: Undisturbed Wall Turbulence Zone 2: Separated Free-Shear Turbulence Zone 3: Jet Flow Zone 4: Regenerated Wall Turbulence Zone 5: Low-Turbulence Region

1 2

3

4

5

5

Jian Fang, Yufeng Yao, Alexandr A. Zheltovodov, and Lipeng Lu, Investigation of Three-Dimensional Shock Wave/Turbulent-Boundary-Layer Interaction Initiated by a Single Fin, AIAA Journal, 2017. Vol. 55, No. 2: 509-523.

Next-Gen Code for UKCTRF

Interface •Flame front •Gas/Liquid interface •Droplets surface

Scale-variety

•Turbulence •Flame •Shock-wave •Droplets

Geometry Complexity

•Swirler •Piston •Igniter •Spray

Next-Gen Code for UKCTRF • HAMISH Code

– First developed at CUED. – To solve the compressible Navier-Stokes equations with species

mass fraction equations and chemical reactions. – Finite volume method using a 2nd-order spatial scheme. – Runge-Kutta time stepping with sub-steps. – Morton-code based unstructured dataset. – Octree-based AMR. – MPI+domain decomposition paralization.

Next-Gen Code for UKCTRF

M-Code M-Code+Tree Tree data structure

• AMR in HAMISH (h-refinement)

Binary tree/Quadtree/Octree

Next-Gen Code for UKCTRF

Temperature Density

Next-Gen Code for UKCTRF

HAMISH with Fixed Mesh of 6400 cells

HAMISH with Adaptive Mesh of 3676 cells

Temperature at t=0.001

Next-Gen Code for UKCTRF

HAMISH with Fixed Mesh of 6400 cells

HAMISH with Adaptive Mesh of 3676 cells

Temperature at t=0.001

Next-Gen Code for UKCTRF

mccomp 19%

mcxyzc 10%

mcci2o 9%

ocfind 5%

mcglvl 2% mccxyz

2% mcrtrv

1% mclowr

1%

steppr 25%

adaptm 12%

enthpy 2%

sortem 2%

trebal 2% discon

2% strcel

1%

OTHER 5%

Profile by Function

MCCOMP Compares two Morton codes in their entirety MCXYZC Converts xyz coordinates into a Morton code at the specified level MCCI2O Converts an encoded integer array to an octal string OCFIND Searches the local Octree using a given Morton code STEPPR Time stepping of the solution, including calculating RHS ADAPTM Adapts the spatial mesh

More than 60% of the computation is cost for M-Code, Octree, AMR related subroutines.

Conclusions • Thanks to ARCHER that a series of DNS/LES of SWTBLI flows can be done

and the quality of the results is proved to be good. • New flow structures and mechanisms are discovered based on the analysis

of the data. • AMR solver could be a game changer, although it is hard in term of coding.

Future Plans • Improve performance of ASTR in terms of I/O and vectorization • Improve the parallel performance and load balance of HAMISH • Add more functionalities to HAMISH.

Daresbury Laboratory Science & Technology Facilities Council

• UKTC (EPSRC – EP/L000261/1)

• UKCTRF (EPSRC – EP/K026801/1)

• Hartree Centre for using their machines

• EPSRC ARCHER Leadership

Acknowledgements

Daresbury Laboratory Science & Technology Facilities Council

Thank you very much