PowerPoint Presentation

Scalable Algorithms for Structured Adaptive Mesh RefinementAkhil Langer, Jonathan Lifflander, Phil Miller, Laxmikant KaleParallel Programming LaboratoryUniversity of Illinois at Urbana-Champaign

Kuo-Chuan Pan, Paul RickerDepartment of AstronomyUniversity of Illinois at Urbana-Champaign

1OutlineExperimental Results/ Performance16th April, 2013Scalable Algorithms for AMR2Introduction to Adaptive Mesh Refinement (AMR)Traditional ApproachExisting FrameworksAlgorithmDisadvantagesNew Scalable ApproachDoing it the Charm++ wayThe Mesh Restructuring AlgorithmDynamic Distributed Load BalancingComparison with Traditional Approach2Experimental Results/ Performance16th April, 2013Scalable Algorithms for AMR3Introduction to Adaptive Mesh RefinementTraditional ApproachExisting FrameworksAlgorithmDisadvantagesNew Scalable ApproachDoing it the Charm++ wayThe Mesh Restructuring AlgorithmDynamic Distributed Load BalancingComparison with Traditional ApproachOutline3Adaptive Mesh RefinementIntroductionSolving Partial Differential Equations (PDEs)PDEs solved using discrete domainAlgebraic equations estimate values of unknowns at the mesh pointsResolution/Spacing of mesh points determines error16th April, 2013Scalable Algorithms for AMR44

16th April, 2013Scalable Algorithms for AMR5Uniform meshesHigh resolution required for handling difficult regions (discontinuities, steep gradients, shocks, etc.)Computationally extremely costlyAdaptive Mesh RefinementIntroductionAdaptive Mesh RefinementStart with a coarse gridIdentify regions that need finer resolutionSuperimpose finer subgrids only on those regionsAMR makes it feasible to solve problems that are intractable on uniform gridUniform MeshAdaptively Refined Mesh

16th April, 2013Scalable Algorithms for AMR6Impact of Type Ia super nova explosion on a main sequence binary companion, rendered using FLASHAdaptive Mesh RefinementApplicationsCFDAstrophysicsClimate ModelingTurbulenceMantle Convection ModelingCombustionBiophysicsand many moreExperimental Results/ Performance16th April, 2013Scalable Algorithms for AMR7Introduction to Adaptive Mesh RefinementTraditional ApproachExisting FrameworksAlgorithmDisadvantagesNew Scalable ApproachDoing it the Charm++ wayThe Mesh Restructuring AlgorithmDynamic Distributed Load BalancingComparison with Traditional ApproachOutline7PARAMESHSAMRAI FLASH p4est Enzo Chombo deal.II and many more

Our current work focusses on Oct-tree based simulations16th April, 2013Scalable Algorithms for AMR8Traditional ApproachExisting Frameworks16th April, 2013Scalable Algorithms for AMR9Refinement structure can be represented using a quad-tree (2D)/ oct-tree (3D)Little more background on AMRAn important condition in AMRRefinement levels of neighboring blocks differ by 1

16th April, 2013Scalable Algorithms for AMR10

A set of blocks assigned to a processUse space-filling curves for load balancingTraditional ApproachTypical Implementations

16th April, 2013Scalable Algorithms for AMR11Traditional ApproachDisadvantages

O(#blocks) memory per-process O(#blocks)Memory bottlenecks!!!O(d) reductionsSynchronization overheads!!!Experimental Results/ Performance16th April, 2013Scalable Algorithms for AMR12Introduction to Adaptive Mesh RefinementTraditional ApproachExisting FrameworksAlgorithmDisadvantagesNew Scalable ApproachDoing it the Charm++ wayThe Mesh Restructuring AlgorithmDynamic Distributed Load BalancingComparison with Traditional ApproachOutline12Instead of processes, promote individual blocks to first class entities chare array with custom indicesA chare corresponds to a blockA block is now the end point of communication

16th April, 2013Scalable Algorithms for AMR13Scalable ApproachDoing it the Charm++ wayBlock NamingBitvector describing path from root to blocks nodeOne bit per dimension at each levelEasy to compute parent, children, siblings using bit manipulation

Block acts as a virtual processorRun time handles communication between arbitrary blocksoverlap of computation with communication of other blocks on same physical process

Dynamic placement of blocks (chares) on processesFacilitates dynamic load balancing

Block is a unit of algorithm expressionSimplifies implementation complexity

16th April, 2013Scalable Algorithms for AMR14Scalable ApproachDoing it the Charm++ way0100100111p0computingWaiting for boundary layers14Based on local error estimate, each block makes one of the following decisions: refine, coarse or stayrefine and stay decisions are communicated to neighborscoarse decision is implied if either refine, stay message is not received16th April, 2013Scalable Algorithms for AMR15Scalable ApproachThe Mesh Restructuring (Remeshing) Algorithm15refine, stay decisions will propagate16th April, 2013Scalable Algorithms for AMR16

Scalable ApproachThe Mesh Restructuring Algorithm

Decisions are updated based on the DFA and change in decisions are again communicatedWhen to stop?16th April, 2013Scalable Algorithms for AMR17

Scalable ApproachThe Mesh Restructuring Algorithm16th April, 2013Scalable Algorithms for AMR18Dynamic Load BalancingExperimental Results/ Performance16th April, 2013Scalable Algorithms for AMR19Introduction to Adaptive Mesh RefinementTraditional ApproachExisting FrameworksAlgorithmDisadvantagesNew Scalable ApproachDoing it the Charm++ wayThe Mesh Restructuring AlgorithmDynamic Distributed Load BalancingComparison with Traditional ApproachOutline1916th April, 2013Scalable Algorithms for AMR20ConclusionNew approach promises high performance for much more deeply refined computations than are currently practicedComparison with the Traditional Approach20Experimental Results/ Performance16th April, 2013Scalable Algorithms for AMR21Introduction to Adaptive Mesh RefinementTraditional ApproachExisting FrameworksAlgorithmDisadvantagesNew Scalable ApproachDoing it the Charm++ wayThe Mesh Restructuring AlgorithmDynamic Distributed Load BalancingComparison with Traditional ApproachOutline2116th April, 2013Scalable Algorithms for AMR22PerformanceBenchmarkAdvection benchmarkFirst-order upwind method in 2D spaceAdvection of a tracer along with the fluid

16th April, 2013Scalable Algorithms for AMR23PerformanceMesh Restructuring Latency2D Advection on BG/QTwo componentsRemeshing decisions and communicationscales wellTermination detection timeLogarithmic in #processes16th April, 2013Scalable Algorithms for AMR24

PerformanceStrong Scaling2D Advection on BG/Q with max-depth of 15Scalable Algorithms for Structured Adaptive Mesh RefinementAkhil Langer, Jonathan Lifflander, Phil MillerParallel Programming LaboratoryUniversity of Illinois at Urbana-Champaign

Kuo-Chuan Pan, Paul RickerDepartment of AstronomyUniversity of Illinois at Urbana-Champaign

Thank you!Questions??25