A Parallel Computational A Parallel Computational Framework for Discontinuous Framework for Discontinuous Galerkin MethodsGalerkin Methods

Kumar Vemaganti

Mechanical Engineering

University of Cincinnati

AcknowledgementsAcknowledgements

•ICES (formerly TICAM), UT Austin•Abani Patra, SUNY Buffalo

OutlineOutline

PI-FED in a nutshellDesign ConsiderationsImplementation IssuesSample Applications

PIFED: Parallel Infrastructure for PIFED: Parallel Infrastructure for Finite Element DiscretizationsFinite Element Discretizations

Two-dimensional Adaptive FEConforming and Non-conformingQuadrilaterals and TrianglesParallel: Partitioning through Space

Filling CurvesMultiple Linear SolversC++ and some FORTRAN

Design Considerations (for a Design Considerations (for a Research Code)Research Code)

Source code available (GPL)Flexible: Multiple Solution

ComponentsExtensible: Object OrientedPortable on *NIX: RS6000, IRIX,

Linux, Mac OS XParallel: Distributed DatastructuresAdaptive: Dynamic Datastructures

The Finite Element ProblemThe Finite Element Problem

The Finite Element ProblemThe Finite Element Problem

Discontinuous Formulations: Discontinuous Formulations: Additional TermsAdditional Terms

Additional terms on the LHS:

The Finite Element Problem is completely specified ifthe coefficients A**, B**, F*, H*, J** are specified foreach element/edge.

OrganizationOrganization

PI-FEDLibrary

Application(User-supplied)

Partitioning

Datastructures Communications

SolversDomain (Mesh)

Coefficients:A**, J**, etc.

Control Parameters

Non-overlapping Partitioning:Non-overlapping Partitioning:Hilbert Space Filling CurvesHilbert Space Filling Curves

Hilbert SFC: A one-dimensional curve that fills the unitsquare.

SFC Implementation: HSFC2D & HSFC3D (H. Carter Edwards)

PIFED: Other FeaturesPIFED: Other Features

•Solvers:•Direct: SPOOLES, SPARSE 1.3•Iterative: PCG & GMRES•AZTEC interface in the works

•Datastructures•Hashtables for elements and nodes (gNodes, pNodes)•Hash rule based on SFC key

•Communications•MPICH version of MPI•Hidden from user

Coefficient Function MCOEFF:Coefficient Function MCOEFF:Element by ElementElement by Element

#include "AFE-machine.hh"#include "AFE-3D.hh"

subroutine USER_MCOEFF ( xyzloc, xyzglob, . mat, neq, nrhs, . nreal, rval, nint, ival, . unreal, urval, unint, uival, . axx, axy, ayx, ayy, . a0x, a0y, ax0, ay0, . a00, f0, fx, fy )

Coefficient Function MCOEFFCoefficient Function MCOEFF

c-- Fill up the coefficient arrays axx(1,1) = a1111 ayy(1,1) = 0.5d0*a1212

axy(1,2) = a1122 ayx(1,2) = 0.5d0*a1212

axy(2,1) = 0.5d0*a1212 ayx(2,1) = a1122

axx(2,2) = 0.5d0*a1212 ayy(2,2) = a2222

c-- RHS coefficients f0(1,1) = -1.0d0*a1111 f0(2,1) = -1.0d0*a2222

/* Dirichlet BCs */ if ( strcmp ( Type, "Displacement" ) == 0 ) {

/* LHS terms */ BX0[0+0*(*n_c)] = a1111*unorm[0]; BX0[0+1*(*n_c)] = a1122*unorm[1]; BX0[1+0*(*n_c)] = 0.5*a1212*unorm[1]; BX0[1+1*(*n_c)] = 0.5*a1212*unorm[0];

BY0[0+0*(*n_c)] = 0.5*a1212*unorm[1]; BY0[0+1*(*n_c)] = 0.5*a1212*unorm[0]; BY0[1+0*(*n_c)] = a1122*unorm[0]; BY0[1+1*(*n_c)] = a2222*unorm[1];

Coefficient Function BCOEFF:Coefficient Function BCOEFF:Boundary EdgesBoundary Edges

Coefficient Function BCOEFFCoefficient Function BCOEFF

B0X[0+0*(*n_c)] = -1.0*a1111*unorm[0]; B0X[0+1*(*n_c)] = -0.5*a1212*unorm[1]; B0X[1+0*(*n_c)] = -1.0*a1122*unorm[1]; B0X[1+1*(*n_c)] = -0.5*a1212*unorm[0];

B0Y[0+0*(*n_c)] = -0.5*a1212*unorm[1]; B0Y[0+1*(*n_c)] = -1.0*a1122*unorm[0]; B0Y[1+0*(*n_c)] = -0.5*a1212*unorm[0]; B0Y[1+1*(*n_c)] = -1.0*a2222*unorm[1];

/* RHS terms */ hx[0] = u0*unorm[0]*a1111 + u1*unorm[1]*a1122; hx[1] = 0.5*a1212*(u0*unorm[1] + u1*unorm[0]); hy[0] = 0.5*a1212*(u0*unorm[1] + u1*unorm[0]); hy[1] = u0*unorm[0]*a1122 + u1*unorm[1]*a2222;

} /* End of Dirichlet BCs */

Coefficient Function JCOEFF: Coefficient Function JCOEFF: Interior EdgesInterior Edges

c-------------------------------------------------c-- Coefficients for this elementc-------------------------------------------------c-- JX0 JX0(1,1) = unorma(1)*a1111 JX0(1,2) = unorma(2)*a1122 JX0(2,1) = 0.5d0*unorma(2)*a1212 JX0(2,2) = 0.5d0*unorma(1)*a1212

c-- JY0 JY0(1,1) = 0.5d0*unorma(2)*a1212 JY0(1,2) = 0.5d0*unorma(1)*a1212 JY0(2,1) = unorma(1)*a1122 JY0(2,2) = unorma(2)*a2222

Coefficient Function JCOEFFCoefficient Function JCOEFF

c------------------------------------------------c-- Coefficients for neighborc------------------------------------------------c-- JX0 JX0_n(1,1) = unorma(1)*n1111 JX0_n(1,2) = unorma(2)*n1122 JX0_n(2,1) = 0.5d0*unorma(2)*n1212 JX0_n(2,2) = 0.5d0*unorma(1)*n1212

c-- JY0 JY0_n(1,1) = 0.5d0*unorma(2)*n1212 JY0_n(1,2) = 0.5d0*unorma(1)*n1212 JY0_n(2,1) = unorma(1)*n1122 JY0_n(2,2) = unorma(2)*n2222

Control ParametersControl Parameters

// Set the number of post-processing quantities int npost = 5; set_parameter ( NUM_POST_QTY, (void *)&npost );

// Set the number of RHSs int nrhs = 1; set_parameter ( NUM_RHS, (void *)&nrhs );

// Set number of solution components int nc=2; set_parameter ( NUM_COMP, (void *)&nc );

// Set the finite element type to DG int fe_type=NONCONFORMING; set_parameter ( FE_TYPE, (void *)&fe_type );

Example: Linear ElasticityExample: Linear Elasticity

• Horizontal traction force on right edge• Left edge fixed• Isotropic Material• p-level = 2

Results on Initial MeshResults on Initial Mesh

HorizontalDisplacements

VerticalDisplacements

Results on Initial MeshResults on Initial Mesh

von MisesStress

Refined MeshRefined Mesh

• Horizontal traction force on right edge• Left edge fixed• Isotropic Material• p-level = 2

Results on Refined MeshResults on Refined Mesh

HorizontalDisplacements

VerticalDisplacements

Results on Refined MeshResults on Refined Mesh

von MisesStress

Another Example:Another Example:Topology OptimizationTopology Optimization

QuickTime™ and aBMP decompressorare needed to see this picture.

Goal: Maximize thestiffness, subject tovolume constraints

Discrete problem: Determinewhich elements have materialin them, and which ones don't

Extreme MaterialsExtreme Materials

QuickTime™ and aBMP decompressorare needed to see this picture.

Extreme MaterialsExtreme Materials

Material with Poisson Ratio = -0.95