Irregularly Portioned Lagrangian Monte-Carlo for Turbulent Reacting Flow Simulationsposter presentation by,

Server Levent Yılmazslyilmaz@pitt.eduCenter for Simulation and Modeling

Mehdi B. Nik Patrick H. Pisciunerismb51@pitt.edu php8@pitt.eduMechanical Engineering and Materials Science

University of PittsburghPittsburgh, PA, USA

National Energy Technology LaboratoryMorgantown, WV, USA

Computational resources by,

AbstractHigh fidelity, feasible simulation methodologies are indispensable to modern gas-turbine design, and here we take on a novel methodology, Filtered Density Function (FDF) for large eddy simulation of turbulent reacting flow. FDF is a robust methodology which can provide very accurate predictions for a wide range flow conditions. However, it involves an expensive particle/mesh algorithm where stiff chemical reaction computations cause quite interesting, problem specific, and in most cases extremely imbalanced (a couple of orders of magnitude) computational load distribution. While FDF is established as an indispensible tool in fundamental combustion research, these computational issues prevent it to get its deserved place at level of the industrial applications. We introduce an advanced implementation which combines robust parallelization libraries, such as Zoltan, and other optimized solvers (ODEPACK), with a flexible parallelization strategy to tackle the load-imbalance barrier. We demonstrate scalability with application to a large scale, high Reynolds number combustor.

The FDF/LES Methodology

The Particle/Mesh Algorithm

The Load Balancing Problem

Irregular Decomposition Strategy & Scalability Analysis

Milestone -1: Implementation and State

Premixed Bunsen Burner

Future Milestones

Compressible Reacting Flow Equations

LES Filtering

UnclosedTerms

Direct Numerical Simulation

(DNS)

Large EddySimulation

(LES)

Reynolds AveragedSimulation

(RAS)

Filtered Density Function (FDF)

Modeling via Stochastic Diffusion Process

The following formal definition, satisfies certain normalization conditions

Marginal FDF for scalars

Scalar FDF Transport Equation

Modeled FDF Equation: Chemistry Terms are exact!

Initialization

SDE Coefficients

Interpolate from FV mesh to particles

Construct Moments on FV mesh points byEnsemble Averaging

Finite Volume Solver

Monte CarloSolver

Solution

FV Mesh points

Monte Carlo Particle

Region of High Activity

Region of Low Activity

Stiff ODE

• Particle solver fully parallel• One process runs FV solver, and only that. Scalability bottleneck!• Manual load redistribution

• Full parallelization of the FV solver (using irregular domain boundaries for structured grid). Removing the obvious scalability bottleneck

• Implement automatic load redistribution with Zoltan • Objective is 10Ks of cores!

Special Thanks!• Peyman Givi, Professor, Mechanical Engineering and

Materials Science ,University of Pittsburgh• Peter A. Strakey, Staff Scientist, National Energy

Technology Laboratory

CH4/Air φ=1Re = 24,000

Predictions

The load imbalance is due mainly to expensive stiff chemistry computations. Indeed, this is common problem with reactive flow simulations, and not specific to LES/FDF.

Data from the simulations verify the imbalance:

Conventional Uniform Decomposition (esp. with structured grid codes)

Irregular Load Balancing Decomposition

… poor load balancing

… perfect load balance at the time of Load redistribution

Good balanceuntil next

redistribution stage

Scalability

Speedup

A representative scalar fieldCPU Time spent at each “cell”