Jacqueline Chen, Evatt Hawkes, David Lignell and Chunsang Yoo

Combustion Research FacilitySandia National Laboratories

jhchen@sandia.gov

Ramanan SankaranOak Ridge National Laboratory

Supported by Division of Chemical Sciences, Geosciences, and Biosciences,

Office of Basic Energy Sciences

Computing: ORNL NLCF, NERSC, PNNL, SNL

Data Management Challenges in Petascale Simulations of Turbulent CombustionData Management Challenges in Petascale Simulations of Turbulent Combustion

Combustion and energy securityCombustion and energy security

• Combustion accounts for 3/4 of energy used in U.S. manufacturing• Ground transportation accounts for 2/3 of petroleum usage

– Potential for improvement in thermal efficiency (30%45%)• Low temperature combustion (LTC) concepts for automobiles• Savings of 3 million barrels of oil per day (out of 20M)

• Design improvements are difficult– Low hanging fruits have already been picked– Advanced concepts require combustion operating at the edge

• Sound scientific understanding is necessary

Turbulent combustion is a grand challenge!Turbulent combustion is a grand challenge!

Diesel Engine Autoignition, Soot IncandescenceChuck Mueller, Sandia National Laboratories

• Stiffness : wide range of length and time scales

– turbulence– flame reaction zone

• Chemical complexity– large number of species

and reactions (100’s of species, thousands of reactions)

• Multi-Physics complexity – multiphase (liquid spray,

gas phase, soot, surface)– thermal radiation – acoustics ...

• All these are tightly coupled

Direct Numerical Simulation (DNS)Direct Numerical Simulation (DNS)

• DNS is a tool for fundamental studies of the micro-physics of turbulent reacting flows– Full access to time resolved fields

– Physical insight into chemistry turbulence interactions

• A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems

DNSDNS PhysicalPhysicalModelsModels

EngineeringEngineeringCFD codesCFD codes

(RANS, LES)(RANS, LES)

DNS capability at SandiaDNS capability at Sandia

• Solves compressible reacting Navier-Stokes equations.• High fidelity numerical methods.

– 8th order finite-difference– 4th order explicit RK integrator

• Hierarchy of molecular transport models• Detailed chemistry• Multi-physics (sprays, radiation and soot)

– From SciDAC-TSTC (Terascale Simulation of Turbulent Combustion)

• Fortran90 and MPI• Highly scalable and portable

S3D is a state-of-the-art DNS code developed with 13 years of BES sponsorship.

S3D parallel performance S3D parallel performance

•S3D scales with 90% parallel efficiency on 10000 cores on CrayXT3 (ORNL)

Achieving quantitative predictability requires petascale computingAchieving quantitative predictability requires petascale computing

Petascale computers needed to achieve relevant parameters spaces for turbulent combustion: N ~ Re9/4 turbulence plus flame scales (3-4 decades of scales)

Relevant parameter regimes of real devices and laboratory-scale flames Re>15,000.

Terascale computing: Re~O(10,000), fully-developed turbulence

Turbulence-chemistry interactions requires transporting 20-80 species plus turbulence

DNS of canonical laboratory configurationsDNS of canonical laboratory configurations

• Lean premixed flames

• Extinction and reignition in nonpremixed flames

• Flame stabilization in autoigniting flames

Temperature and heat Temperature and heat release, Akiba and Marelease, Akiba and Ma

DNS of lean premixed methane/air combustionDNS of lean premixed methane/air combustion

• Goals

– Better understanding of lean premixed combustion in natural-gas based stationary gas turbines

– Model validation and development

• Simulation details

– Detailed CH4/Air chemistry (18 D.O.F.)

– Slot burner configuration with mean shear

– Spatially developing and statistically stationary simulation.

• Better suited for model development

– Parametric study: 3 simulations at increasing turbulence intensities

• to understand the effect of turbulence on flame structure and burning speed.

Sankaran et al. 2006

Heat release in a methane-air slot Bunsen Heat release in a methane-air slot Bunsen flame, animation by H. Akiba and K. L. Maflame, animation by H. Akiba and K. L. Ma

Parametric study with increasing Reynolds numberParametric study with increasing Reynolds number

u’/SL 3 6 10

Mesh 52M 88M 200M

Performed on X1E 512 MSPs

XT34800 CPUs

XT37200 cores

CPU-hours 200k 500K 1.2M

•Small-scale intense turbulence leads to flame broadening in the preheat zone and increased flame shortening

Effect on flame structureEffect on flame structure

• Progress variable defined based on O2 mass fraction

• Instantaneous slices shown on the left for Case 1

- Progress variable from 0 (blue) to 1 (red) in color

- Heat release rate as line contours

• Considerable influence on preheat zone

• Reaction zone relatively intact

1/4

1/2

3/4

FreshBurned

u’/SL=3 u’/SL=6

Turbulent nonpremixed combustionTurbulent nonpremixed combustion

• Fuel and air segregated

• Mixing limited

• Extinction

• Reignition

• Flame stabilization

Extinction and reignition in a CO/H2 jet flameExtinction and reignition in a CO/H2 jet flame

Understanding extinction/reignition in non-premixed combustion is key to flame stability and emission control in aircraft and power producing gas-turbines

Discovered dominant reignition mode is due to engulfment of product gases, not flame propagation

Scalar dissipation rate

BurningExtinguished

The largest ever simulations of combustion have been performed to advance this goal:

500 million grid points 11 species and 21 reactions 16 DOF per grid point 512 Cray X1E processors 30 TB raw data 2.5M hours on IBM SP NERSC (INCITE); 400K hours on Cray X1E (ORNL)

Hawkes, Sankaran, Sutherland, Chen – 2006, DOE INCITE 2005, early user LCF /ORNL 2005

Description of runs- Temporally Evolving Non-premixed Plane Jet Flame

Description of runs- Temporally Evolving Non-premixed Plane Jet Flame

• Jet develops temporally.• Shear-driven turbulence interacts with the flame.

Fuel

Air

Air •Initial condition•Later time

Mixing,Reaction

Mixing,Reaction

Fuel

Air

Air

Spanwise BC:periodic

Streamwise BC:periodic

Volume rendering of scalar dissipationYu, Ma, Chen, Chen, Hawkes, StorCloud demo at SC05. Volume rendering of scalar dissipationYu, Ma, Chen, Chen, Hawkes, StorCloud demo at SC05.

• Hardware accelerated parallel volume rendering (nearly interactive – read and render in 2 sec).

• 4D time-varying terascale data

• Multi-variate volume rendering

• Intelligent visualization

Reynolds number effects on Reynolds number effects on

• Higher Re:– more fine-scale intermittent structure

– higher fluctuations of

Case L

Case HCase M

•Increasing Reynolds number

Reynolds number effects on extinctionReynolds number effects on extinction

20tj S

imu

lation

Tim

e 40tj

•Increasing Reynolds number results in more extinction

Re=2500 Re=4500

Re=9000

How is extinction correlated with local mixing rates?How is extinction correlated with local mixing rates?

•Scalar Dissipation (mixing rate) •Extinguished Regions

Quantification of Extinction- Extinguished Flame AreaQuantification of Extinction- Extinguished Flame Area

Extinguished<0

Burning>0

Flameedge

Mixture Fraction IsosurfaceZ=Z0

• Reaction rate related to the conditional fine-grained surface-density of the stoichiometric surface

• Isosurface extraction from volume data through triangulation– Data analysis on iso-surface and local

normal vector

• Identify flame holes– Flame edge analysis

• Edge propagation speed

– Study reignition mechanisms

Analysis code has to be parallel, suitable for large data and reside within S3D

Se

Joint PDF of edge speed and mixing rateJoint PDF of edge speed and mixing rate

• Color scale: Joint PDF, Black line: conditional mean speed.• First, mainly negative speeds, strong negative correlation with .• Then, broader PDF, with 2 branches

– negatively correlating branch at very high – positively correlating branch at low-intermediate

• Peak positive edge speed occurs at quite high !!!

Simulation Time

Physical interpretationPhysical interpretation

2 basic ideas for reignition:

• Propagating edges along the stoichiometric contour.

•Burning

•Extinguished

Z=Zst

Tentative InterpretationTentative Interpretation

2 basic ideas for reignition:

• Propagating edges along the stoichiometric contour.

•Burning

•Extinguished

Z=Zst

O(SL)

Tentative InterpretationTentative Interpretation

2 basic ideas for reignition:

• Turbulence folds burning flames onto non-burning areas.

•Burning

•Extinguished

Z=Zst

Tentative InterpretationTentative Interpretation

2 basic ideas for reignition:

• Turbulence folds burning flames onto non-burning areas.

•Burning

•Extinguished

Z=Zst

•Burning

•Extinguished

Z=Zst

•Burning

•Extinguished

Z=Zst

Tentative InterpretationTentative Interpretation

• u’>>sL indicates laminar edge flame propagation unimportant.

• Expect reignition by turbulent flame-folding.

• To bring burning and non-burning surfaces together, compressive strain is required, leading to high dissipation.

• Consistent with our result – but more work needed to confirm.

•Compressive strain leads to high

Understanding the effect of ignition on lift-off stabilization of n-heptane diesel jet requires petascale computingUnderstanding the effect of ignition on lift-off stabilization of n-heptane diesel jet requires petascale computing

•How is combustion stabilized at the lift-off length in a n-heptane diesel jet?

-Flame propagation-Autoignition (1st stage cool flame)

•Is lift-off stabilization supported by premixed flame propagation into a cool flame mixture, or by second-stage autoignition?

•Is lift-off scaling parameterized by ignition or by flame propagation?

•How does turbulent mixing affect the transition from first-stage to second-stage, high temperature ignition?

•How are soot precursors affected by ignition quality?

High-speed Chemiluminescence Imaging in a combustion vesselnear the lift-off length, Lyle Pickett,SNL

Stabilization of Lifted Autoigniting FlamesStabilization of Lifted Autoigniting Flames

• Stabilization mechanisms in lifted, vitiated flames

• Hydrogen lifted flame– High flow velocity to lift off flame– Auto-ignition may occur below the base of

the lifted flame due to high co-flow temperature

– Approximately 1.2 million CPU hours per a simulation on Jaguar in NCCS

• 9 species with 21 elementary reactions• 24 x 24 x 5.4mm3 domain size with 1600

x 1600 x 270 grid resolution ~0.7 billion grids

• 6~8 flow-though times w/ Umean = 347 m/s

• Total 3~4 simulations

H2(65%)/N2(35%)400K

Air1100K

Highest heat release rate (green), YHO2(red), and vorticity (aqua) of hydrogen lifted jet flame at t = 0.3ms

Liftoffheight

Animation of hydroxyl radical in lifted hydrogen/air lifted flameAnimation of hydroxyl radical in lifted hydrogen/air lifted flame

n-Heptane Lifted Autoigniting Flamen-Heptane Lifted Autoigniting Flame

• CPU hour estimation of n-heptane lifted flame– Skeletal n-heptane mechanism

• 88 species; 384 reaction steps

– 40 x 30 x 30 mm3 domain size– 2000 x 1200 x 1200 grid resolution (3

billion grids)– 12 flow-though time (2.4ms) calculation

200-300 m/s mean flow velocity– CPU hours

• (2000 x 1200 x 1200 grid points) x (240,000 time steps) x (92 equations) x (3.0 x 10-9 hour/NgridNtimeNeq) 240 million CPU hours (on Jaguar in NCCS)

– Requires petascale computer!

Propane lifted flames in hot coflow (from Kim et. al, Proc. Combust.

Inst. 31, 2007 to appear)

50 m/s

80 m/s

110 m/s

140 m/s

Community Data SetsCommunity Data Sets

• Precedents for comparison of measured and modeled results– TNF workshop

http://www.ca.sandia.gov/TNF/abstract.html

– Premixed flame workshops http://eetd.lbl.gov/aet/combustion/workshop/workshop.html

• Addition of high-fidelity numerical benchmarks for model validation and development

Data SharingData Sharing

What is the best model for sharing data, especially large data?– Reduced data on the web?

• RANS, LES means, conditional means, slices, chunks• What is the basic set of data to make available?

– Visiting research projects?

– Giving out whole data?

– What, if any, tools/procedures are needed to enable this?• Visualization• Data movement• Generic data-processing

Challenges of Petascale Computation: Mountains of DataChallenges of Petascale Computation: Mountains of Data

• Data storage:– Long term: where do we put 500TB?

– Short term: scratch ~ 1TB, but need ~ 10TB!

• Data movement:– Archive to scratch (~ 1 week to move 10TB)

– HPC facility to local analysis cluster (longer)

• Data processing:– Everything must be parallel, scalable.

– IO speed, memory are the bottlenecks.

• Visualization:– Parallel, Multi-variate

– Multi-scale phenomena

•HPSS storage facility at NERSC

• Interpretation:–Physics are more complex

–Wider range of scales, manual sifting is impossible.

–Multi-scale analysis methods

–Feature detection, growing, and tracking

S3D I/O RequirementsS3D I/O Requirements

I/O sizes of current terascale S3D combustion simulations

Grid points Platform Size per

dump

Jet, Re=3,000 150M XT3 (NCCS) 19GB

Jet, Re=5,000 350M SP (NERSC) 45GB

Jet, Re=10,000 500M X1E (NCCS) 64GB

Bunsen, u’/Sl=3 52M X1E (NCCS) 8GB

Bunsen, u’/Sl=6 88M XT3 (NCCS) 13GB

Bunsen, u’/Sl=10 200M XT3(NCCS) 29GB

• Jet simulation (Re=10,000) on X1E (20TF). – At the rate of one data dump every hour, I/O rate is 64GB/hour

• On a Petaflop system, required I/O rate is 3.2TB/hour

• To achieve 5% maximum overhead, I/O has to occur at 64TB/hour or 17GB/s

• It will be useful to dump data more often than once an hour if higher I/O rates are available.

Data Movement and WorkflowData Movement and Workflow

• 40TB of data generated last year (excluding prep runs and analysis output)

• Geographically distributed HPC resources/archives– ORNL, NERSC, PNNL, Sandia (CA)

• Need automatic handling of data– Kepler workflow for S3D, Ramanan Sankaran and Scott Klasky (ORNL)

and Norbert Podhorszki (UC Davis) (SDM Center)

Parallel feature detection and tracking - gleaning insight from large dataParallel feature detection and tracking - gleaning insight from large data

• Large and complex multiscale data

• Interesting regions sparse

• Temporal evolution of features

Interactive Scalable Feature Detection and Tracking Interactive Scalable Feature Detection and Tracking

DNS of Autoignition of H2/Air in Inhomogeneous Mixtures, Echekki and Chen, 2001

Feature definition: threshold on HO2 concentration

Feature detection and tracking:

Born

Merge

Die

Grow

Split

Feature statistics

Feature-based analysis

W. Koegler 1996 IEEE Vis.

Feature StatisticsFeature Statistics

Feature Graph

time

Koegler 1996 IEEE Vis.

Future of automated data discoveryFuture of automated data discovery

• Novel approach for data discovery:– Hierarchical features allow humans to frame hypotheses at

different levels of detail. – Quantitative feature definitions – beyond static thresholds

• Complex isosurfaces• Topological methods for scalar and vector definitions

– Online distributed feature detection and tracking; computational steering of I/O and analysis

• FastBit technology for efficient search and query– Frame hypotheses as parametric relationships between feature

properties.– New rendering tools allow humans to visually and quantitatively

review multiple features in context and provide training data to machine learning.

Scalable Interactive Combustion Analysis ToolkitScalable Interactive Combustion Analysis Toolkit

Raw DNS Data Feature Definition

Demand-driven I/O preprocessor

Feature detection

Feature tracking

Feature-based analysis

Flame surface statistics Chemical Analysis

Conditional statistics Spectra – Fourier, wavelets

Averaging and filtering Reduced order representation- POD, topology

Analysis Libraries

Feature Detection/Tracking Demand-driven I/O Libraries

AcknowledgmentsAcknowledgments

• Chunsang Yoo (SNL)• Ramanan Sankaran (ORNL)• Scott Klasky (ORNL)• Evatt Hawkes (SNL)• Mark Fahey (ORNL)• David Skinner (LBNL)• David Lignell (SNL)• Andrea Gruber (SINTEF, U. Trondheim)• Tianfeng Lu (Princeton U.)• Chung Law (Princeton U.)• Kwan-Liu Ma (U. C. Davis)• Hiroshi Akiba (U. C. Davis)• Hongfeng Yu (U. C. Davis)• Scott Klasky (ORNL)• Wendy Doyle (SNL)• John Wu (LBNL)• Arnaud Trouve (U. Maryland)• Hong Im (U. Michigan)• Chris Rutland (U. Wisconsin)

ContactsContacts

Jacqueline ChenCombustion Research FacilitySandia National LaboratoriesLivermore, CA 94550jhchen@sandia.gov

Ramanan SankaranNational Center for Computational SciencesOak Ridge National Laboratorysankaranr@ornl.gov

41 Presenter_date

DNS of turbulent flame-wall interactionDNS of turbulent flame-wall interaction

- DNS of a premixed flame interacting with turbulence in a wall-bounded channel flow

- Detailed H2/air chemistry (14 D.O.F.)

- Inflow turbulence obtained from a separate simulation of inert channel flow

- Cost: 100K hours on Cray X1E

- Goals- Material failure from fluctuating

thermal stress in micro-gas turbines

- Study of spatial and temporal patterns of wall heat flux

- Discovered correlation of near-wall coherent turbulence structures and exothermic radical recombination reactions with wall heat flux

Wall heat fluxes in turbulent flame wall Wall heat fluxes in turbulent flame wall interaction at low Reynolds numberinteraction at low Reynolds number

A. Gruber, R. Sankaran, E. R. Hawkes, and J. H. Chen - 2006

Flame stabilization in lifted autoigniting hydrogen flamesFlame stabilization in lifted autoigniting hydrogen flames

• Goal: determine stabilization mechanisms in lifted, autoigniting hydrogen/air flames

• Hydrogen lifted flame– Important submechanism for n-heptane– Comparison with experiment (Chung, Cabra)– Auto-ignition may occur below the base of the

lifted flame due to high co-flow temperature– Approximately 1.5 million CrayXT3 CPU hours – 9 species; 21 elementary reaction steps (Li et al.

2006)• 24 x 20 x 3.6mm3 domain size; 1600 x 1000 x

240 grid resolution=~400M grids

• 4 flow-though times; Umean = 347 m/s, Re=7000

• Parametric study with coflow temp, flow velocity, additives

• Total 3~4 simulations (~3.0 million cpu-hrs

on CrayXT3 ORNL 2006-2007)

Heat release rate and vorticity in a H2/air lifted slot jet flame in heated coflow

FuelFuel

AirAir

AirAir

fuelfuel

airair

Motivation• Soot is a pollutant and lowers combustion

efficiencies in diesel engines• Soot radiation is the major heat transfer mode

Goals• Quantify soot formation and transport

mechanisms in turbulent flames Approach

• DNS of turbulent sooting flames with detailed chemistry, transport, radiation

• 2–3 moment soot particle model with semi-empirical soot chemistry

• 3M cpu-hours on CrayXT3 at ORNL to observe slow soot processes

D. O. Lignell, P. J. Smith, and J. H. Chen

DNS of non-premixed sooting ethylene flames

DNS of stabilization in lifted auto-igniting flames (planned)DNS of stabilization in lifted auto-igniting flames (planned)

Goal: Determine stabilization mechanisms in lifted, auto-igniting flames relevant to efficient, clean burning low-temperature compression ignition engines and flashback in aircraft gas turbine engines

Hydrogen lifted flame• 300K hydrogen turbulent jet; 1100K air coflow

jet• Important submechanism for n-heptane• Comparison with experiment (Chung, Cabra)• Auto-ignition may occur below the base of the

lifted flame due to high co-flow temperature • Approximately 1.2 million CPU hours per

simulation on Jaguar at NCCS

- 9 species; 21 elementary reaction steps (Li et al. 2006)

- 24x20x3.6 mm3 domain size; 1600x1000 x240 grid resolution= ~400M grids

- 4–6 flow-though times; Umean = 350 m/s

- Total 3~4 simulations (~3.0 million cpu-hours on CrayXT4 at ORNL 2006–2007)

Heat release (gold) and vorticity in a lifted autoigniting H2/air jet flame

Transport Effects on Mixing TimescaleTransport Effects on Mixing Timescale

• Mixing time scale is key element of combustion models (transported PDF method)

• Most models assume mixing time scale same as mechanical time scale for all species

• Detailed transport and chemistry effects can alter the observed mixing timescales

Increasing diffusivity

AcknowledgmentsAcknowledgments

• Chunsang Yoo (SNL)

• Ramanan Sankaran (ORNL)

• Evatt Hawkes (SNL)

• Mark Fahey (ORNL)

• David Lignell (SNL)

• NLCF Computational Combustion End Station (3.6M cpu-hours, 2006)

• NERSC Incite Award 2005 (2.5M cpu-hours)