Center for Hybrid Rocket Exascale Simulation Technology

Center for Hybrid Rocket Exascale Simulation Technology

Team: Varun Chandola, James Chen, Paul DesJardin, Matt Jones, Matt Knepley, AbaniPatra, and Mark SwihartTST: John Hewson (SNL), Bob Anderson (LLNL), Marianne Francois (LANL), Fady Najjar (LLNL), Sriram Swaminarayan (LANL), Greg Weirs (SNL) NNSA Lab Collaborators: Greg Burton (LLNL) Matthew McNenly (LLNL), Roger Pawlowski (SNL), Tom Smith (SNL), Jim Stewart (SNL), Cosmin Safta (SNL), Brian Williams (LANL)

Team Members MAE CBE

CSE CCR

Paul DesJardin James Chen

Varun ChandolaMatt Knepley Matt Jones Abani Patra

Mark Swihart

Tufts

Letitia Thomas

SEAS

Team MembersProfessional Staff

• 3 post doctoral scholars• 6 Ph.D. students, 3 UG

students

Software Architect (identified)

Rubik Asatryan(chemistry)

Marianne Sullivan(Administration Support)

Louis Llanos

Kenny Budzinski

Jake Henry Sophia Matla Nick McNallyGabe Surina

Mae Sementilli Venoos Amiri

Darsh Nathawani

Outline

• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Technical Challenges

Outline

• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Technical Challenges

NASA Peregrine Motor

Advantages• Safe – no TNT equivalent• Cost effective for handling, storing, and

environmentally friendly

• Throttled like a liquid but has energy density of a solid

Disadvantages• Historically low regression rate and

therefore lower specific impulse

• Difficulty with multiple port designs

• Scaling less understood because of turbulent boundary layer

• O/F shift with constant ox mass flux

Cantwell, B., "Aircraft and Rocket Propulsion", Stanford University, 2019, Ch. 11 Hybrid Rockets

Application: Hybrid Rocket Motors

• Response of ablating solid fuels – Phase transformations and mechanical deformations– Liquid layer formation, growth and atomization– Mixed modes of burning, individual droplet vs. group combustion– Mixed modes of heat transfer from reacting slurry to surface

Cantwell, B., "Aircraft and Rocket Propulsion", Stanford University, 2019, Ch. 11 Hybrid Rockets

Hybrid Rocket: High Regression Rate

• Turbulent reacting multiphase

interfaces

– Shear-induced liquid layer

instability growth / atomization

– Secondary atomization

processes and droplet burning

– Gas phase combustion

• Coupled thermal and mechanical

response of fuel grain

– Heat transfer of the melting fuel

– Dynamically changing interface

Gas

Solid

Liquid

Liquid

&

Gas

Turbulent Oxidizer

Simulation

Slab Burner Exp. (20k fps)

Wakes

BL’sGrid

Channels

Cut off limit ~1/D

Resolved scales

Modeled scales

Wave number

Ener

gy s

pect

ra

Curran, Henry. “Developing detailed chemical kinetic mechanisms for fuel combustion.” Proceedings of the Combustion Institute 37 (2019): 57-81

Turbulent Length Scales Chemical Reactions

DOF Scaling Challenges

Ti

me

Sca

le, s

econ

ds

Engineering Scale of Interest

Convective Transport Nozzle

Expan

sion

10-10

102

10-8

10-6

10-4

10-2

100

Length Scale, meters

10-12

10-9

10-6

10-3

100

103

Fuel Decomposition & Liquid Formation

Wave Instability Growth & Shear Atomization

Paraffin wax, ~ C35H72

Multiphase C

ombustion

Scaling Challenges

Ti

me

Scal

e, s

econ

ds

10-10 102

10-8 10-6

10-4 10-2

100

Length Scale, meters

10-12

10-9

10-6

10-3

100

103

For every decade in length scale: - 3 orders of magnitude increase

in nodes (for uniform mesh) - 1-2 order of magnitude increase

in time steps

106 cells 103 steps (megaflop)

109 cells 104 steps (petaflop)

1012 cells 105 steps (teraflop)

1015 cells 106 steps (exaflop)

Subgrid Scale (SGS) Modeling

Exascale SGS Modeling Development

Scaling Challenges

ML Driven Model ReductionTurbulence Chemistry

• Identifying optimal flow regime (SGS Model) for LES

• Using Gaussian Process Classification• Non-linear models• Able to propagate input

uncertainty to output• Trained from DNS-LES-regime

outputs

• Mapping high-dimensional data to low-dimensional non-linear manifold

• Flamelet Generated Manifolds (FGM)

Z

Temperature(K)

0 0.2 0.4 0.6 0.8 1300

600

900

1200

1500

1800

2100

2400 Zst = 0.149Λ = Cst

x (m)Temperature(K)

0 0.003 0.006 0.009 0.012 0.015300

600

900

1200

1500

1800

2100

2400

2700

3000 t = 0st = 7.69e-4st = 5.09e-3st = 0.015st = 0.029st = 0.052st = 0.086st = 0.242s

Manifold prior for LES

Outline

• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges

Data Development and ExperimentsSource / Activity Scale (m) Measurements V&V Outcomes

fuel characterization 10-9 - 10-4 wax viscosity, vapor pressure, specific heat, conductivity, etc.

burning droplet models

slab motor / DNS 10-4 - 10-2 regression rates, solid/gas temp., particle dynamics

chemical kinetics, atomization, droplet dynamics

sounding rocket / LES 10-2 - 10-1 specific impulse, coef. of thrust, characteristic velocity

LES near-wall subgridscale (SGS) models

NASA Perigrine / VLES 10-1 - 101 specific impulse, coef. of thrust, characteristic velocity

VLES boundary layer subgrid scale (SGS) models

Data Gathering/Experiments guided by QOI UQ outcomes

Physical Property Data Sources

Melting point, vapor pressure, viscosity, thermal conductivity, heat capacity, enthalpies and entropies of

formation, fusion, and vaporization

Literature

Quantitative Structure-Property Relations

Computation

Experiment

Slab Burner ExperimentGlow Plug Ignition (60 fps) Wollaston Prism Interferometry (8k fps)

Two-color Pyrometry

Slab Exp: Regression Rates

Wax raw image during burn Binary image of wax sample

• RAW image converted to binary• Height of wax sample from each image• Difference in heights divide by time

interval• Obtain experimental global and local

regression rates

Change in height over time for and oxidizer mass flux of 14.959 k'/)*+

2% Error

Global Regression Rate

Local Regression Rate

Slab Exp: Two-Color PyrometrySoot Formation Model

T = 2800K

• Ratio of detector signal at two narrowband wavelengths

! =#$( 1'( −

1'*)

,n.('*, !).('(, !) + ,123('()23('*) + ,1

'*'(

4+ ,1 53,(53,*

• Radiometric self-calibration using low and high temperature black body sources

≈ 7879

Ray-Tracing

Camera Response Function

3-D Reconstructed Temperature Profile

*Aphale, Siddhant S., and Paul E. DesJardin. "Development of a non-intrusive radiative heat flux measurement … based two-color pyrometry." Combustion and Flame 210 (2019): 262-278.

Radiative Heat Flux

Slab Exp: Thermocouples & Heat Flux GaugesEmbedded thermocouples

Response time: 3msHukseflux Schmidt-Boelter gauge

Response time: 250msSacrificial micro heat flux gauge

(Thin skin calorimeter)

10 m

m

10 mm

Radiative heat flux comparison

!. # $$

Thermocouple and micro heat flux gauge location

Thermocouple locations

Micro heat flux gauge

Front

Middle

Back

Slab Exp: Particle TrackingSlab Burner – Phantom Camera (11k fps)

Droplet - Particle Tracking (13k fps)

Droplet Velocity

Droplet Diameter

Outline


Exascale Challenge

The Main Challenge is to run the entire analysis pipline at

scale.

Exascale Challenge


scale.

Exascale Challenge


scale.

Exascale PlanA library compatibility layer will enable low-level optimizations:• Support Vendor Standards (CUDA, SYCL, HIP)• Leverage Vendor Libraries (cuBLAS/cuSparse, ACML, MKL, Kokkos)• Node-aware Communication (batch messages, use PDE structure)• GPU-aware Communication (custom message pack/unpack)

Composable abstractions will enable high-level optimizations:• Develop library of ML primitives (Isomap, MSVM, k-means, DNN surrogates)• Computation-Aware Partitioning (for heterogeneous machines)• Robust solvers for multiphysics (FAS, nonlinear elimination, bootstrap)• Optimization and Eigensolves (one-shot multigrid, bootstrap, integrated design)

Outline


To build composable libraries,

based on PETSc,

for the entire analysis pipeline.

Software Plan

CCR DeploymentCCR scalable development pipeline enables:• rapid prototyping on desktop,• deployment on CCR resources,• scaling up to national facilities.

CCR provides early access to:• XMS/XDMoD metrics service could be used for arch-aware

partitioning• “Ookami” ARM64-SVE testbed

Why Libraries?

Libraries Hide

Hardware Details

Why Libraries?

Libraries Hide

Hardware Details

Why Libraries?

Libraries Hide

Hardware Details

Why Libraries?

Libraries Hide

Implementation Complexity

Why Libraries?

Libraries Hide


Why Libraries?

Libraries Hide


Why Libraries?

Libraries Disseminate

Project Advances

Validated Ablation and Droplet Burning Models

Scalable Isomap and Physics-Based Encoders

Scalable Nonlinear Solvers

Why PETSc?• Dependability & Maintainability

• Portability & Robustness

• Performance & Scalability

• Optimality & Robustness

• Flexibility & Extensibility


• Nearly 30 years of continuous development• Portability & Robustness





• Portability & Robustness• Tested at every supercomputer installation and 10,000+ users






• Performance & Scalability• Many Gordon Bell Winners






• Optimality & Robustness• State-of-the-Art Linear and Nonlinear Solvers, Eigensolvers,

Optimization Solvers, Timesteppers• Flexibility & Extensibility

Why PETSc?

• Dependability & Maintainability




• Flexibility & Extensibility• More than 8000 citations and hundreds of application packages• Aerodynamics, Arterial Flow, Corrosion, Combustion, Data Mining,

Earthquake Mechanics, . . .

Outline

• Center Vision Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges

Center Overview

Data Development

- Experiments- Exascale DNS

- Small, Intermediate and

System level milestones

Field Modeling

- DNS, LES, VLES- a posteriori ML

based SGS assessment

Computational Framework- PETSc Exascale

simulation framework

Prediction Assessment- ML based UQ

- a priori ML based SGS assessment

- V & V

Center for Exascale Simulation of Hybrid Rocket Motors

Experimental Design

Multiscale modeling & simulation

Large scale system-level testing

Small scale slab experiments

Uncertainty Quantification

DNSv.s.UQ

model

ML

DynamicSGS

model

LES

Slab motorexperiment

Validation

Exascale Sim.

Roadmap

Years 1-2 (Small spatial/temporal scale – DNS – Slab motor)

Years 2-4 (Intermediate spatial/temporal scale – LES – Sounding rocket)

Roadmap

Years 4-5 (Full spatial/temporal scale – VLES – Peregrine rocket motor)

Peregrine sounding rocket test at NASA Ames

Outline


Key Challenges• COVID-19 impediments which impacts all aspects of the

center beyond “normal challenges”• Technical Challenges

– Resolution requirements for resolving liquid fuel instability and atomization processes

– Soot kinetics with wax/GOX systems, what level of detail is adequate to sufficiently describe conjugate heat/mass transfer processes?

– Scaling of mixed Eulerian/Lagrangian formulations on exascaleplatforms with mixed hardware requirements

– Classification of burning regimes, fidelity and robustness of ML inspired FGM

– Propagation of uncertainty from 1D models into multi-dimensional contexts

– V&V using optical measurements, e.g., two-color pyrometry, Schlieren and other spectroscopy measurements

Documents

Center for Hybrid Rocket Exascale Simulation Technology