CAD DEFEATURING USING MACHINE LEARNING › _assets › documents › 2019_IMR... · CAD DEFEATURING USING MACHINE LEARNING Steven Owen1 Timothy M. Shead2 Shawn Martin3 1Sandia National

CAD DEFEATURING USING MACHINE LEARNING

Steven Owen1 Timothy M. Shead2 Shawn Martin3

1Sandia National Laboratories∗, Albuquerque, New Mexico, U.S.A. [email protected] National Laboratories, Albuquerque, New Mexico, U.S.A. [email protected] National Laboratories, Albuquerque, New Mexico, U.S.A. [email protected]

ABSTRACT

We describe new machine-learning-based methods to defeature CAD models for tetrahedral meshing. Using machinelearning predictions of mesh quality for geometric features of a CAD model prior to meshing we can identify potentialproblem areas and improve meshing outcomes by presenting a prioritized list of suggested geometric operations tousers. Our machine learning models are trained using a combination of geometric and topological features from theCAD model and local quality metrics for ground truth. We demonstrate a proof-of-concept implementation of theresulting workflow using Sandia’s Cubit Geometry and Meshing Toolkit.

Keywords: machine learning, mesh generation, tetrahedra, supervised learning, defeaturing

1. INTRODUCTION

An engineering analyst typically receives CAD mod-els and assemblies that are developed based on man-ufacturing specifications which are not directly usefulfor analysis. For example, a CAD model may con-tain many small artifacts or irrelevant details that willhave little effect on the outcome of a physics simula-tion, but dramatically slow the simulation by produc-ing needlessly-complex meshes. Some automatic sur-face meshing techniques [?, ?, ?] incorporate tolerantapproaches that can ignore small geometric featuresand artifacts in the final mesh, but without carefuluser validation, fully automatic meshing methods runthe risk of eliminating geometry that is required forsimulation.

At the opposite end of the spectrum, fully manual de-featuring of a CAD model prior to meshing requiresthorough inspection using advanced 3D CAD-basedsoftware tools such as [?, ?], after which the analyst

∗Sandia National Laboratories is a multimission labo-ratory managed and operated by National Technology &Engineering Solutions of Sandia, LLC, a wholly owned sub-sidiary of Honeywell International Inc., for the U.S. Depart-ment of Energy’s National Nuclear Security Administrationunder contract DE-NA0003525. SAND2019-6988 C

will devise a strategy for model preparation that islikely to include many complex, time-consuming ge-ometric modifications. The defeaturing processs nor-mally requires an expert user who can identify prob-lematic geometry and select the appropriate tools tomake local adjustments to the CAD model. Theseadjustments must be informed by sound engineeringjudgement based on knowledge of the physics to besimulated, along with an understanding of the meshgeneration procedure and expected mesh quality out-comes.

Thus, we seek to significantly reduce the time and ef-fort required for efficiently defeaturing CAD modelswhile maintaining the ability of users to validate re-sults and intervene in the process. Our goal is a sys-tem that permits users to graphically inspect a CADmodel, efficiently guiding them to make modificationsprior to automatic meshing that ensure quality mesh-ing outcomes. Beginning with a solid design modelcomposed of geometric entities (vertices, curves, andsurfaces), the system should predict which entities willlead to suboptimal local mesh quality, presenting themto the user in prioritized order. For each entity, aset of suggested solutions should be presented, sortedbased on their (predicted) ability to improve the lo-

cal mesh quality outcomes. The user would then havethe opportunity to preview, adjust, and perform thesuggested operations as desired.

For this work, we are using machine learning to extendthe framework described in [?], prioritizing suggestedoperations using predicted meshing outcomes to moreeffectively and efficiently guide the user.

2. PRIOR WORK

While machine learning is widely used in text, image,audio, and video analysis, there has been little researchon the application of machine learning to model prepa-ration for simulation. One notable work in this area is[?], which describes a limited environment for defea-turing CAD models where machine learning is drivenby heuristic rule-based outcomes. While proposingseveral new criteria for evaluating defeauturing resultsfrom trained models, they rely on human interactionto judge the quality of results, making scalability prob-lematic. In contrast, we use mesh quality metrics froman automatically generated FEA mesh as the trainingobjective for defeaturing. This allows for automaticgeneration of training data, relying only on an embed-ded geometry and meshing environment. Other recentwork has also demonstrated machine learning methodsuseful for shape recognition and classification of CADmodels [?, ?, ?]. While related, these methods stopshort of driving modifications to the CAD model suchas those required for mesh generation and simulation.

3. OVERVIEW

Supervised machine learning is typically character-ized as a problem where, given a training dataset{(x1,y1), ..., (xn,yn)} with vector input features xand vector output features y (typically referred to aslabels or ground-truth), it is assumed that there existsan unknown function y = f(x) that maps input fea-tures to output features. Using a learning algorithm,a model can be trained (or fit) to the data, so thatthe model approximates f . Once a model has beentrained, it can be used to evaluate new, previously-unseen input vectors to estimate (or predict) the cor-responding output vectors. To apply supervised ma-chine learning in a new problem area, the researchermust determine what the domain-specific outputs willbe, identify the available domain-specific input fea-tures that can be used to predict them, and createa training dataset containing enough examples of eachto adequately represent their distributions.

For this work, our first decision was to limit our scopeto the defeaturing of individual parts. While oper-ations correcting the interactions between parts toavoid gaps, overlaps and misalignments are of vitalimportance, we chose to save them for future work.

Next, we needed to define our machine learning modeloutputs. Since our goal, outlined in the introduction,was to rank geometric entities and solutions by theirpredicted local meshing outcomes, it followed that theoutputs of our models y would be those outcomes,represented using mesh quality metrics. Similarly, theinput features x for each model would be chosen tocharacterize the local CAD model geometry and topol-ogy that we presumed would drive those outcomes.

Given machine learning models that could predictmesh quality outcomes for a geometric entity or lo-cal region of a CAD model, we could use those pre-dicted outcomes to present users with a sorted list ofproblem areas. Then, for a given problem area, wecould use solution-specific machine learning models topresent a sorted list of suggested solutions. A key in-sight during the design phase was the recognition thatthe set of local CAD model features that might influ-ence the outcome for a given solution were themselvessolution-specific. For example, there are at least twodifferent strategies to resolve a sliver surface. One in-volves a composite operation that combines two adja-cent surfaces (see Table 1(3)), while another is to re-move the surface and extend the adjacent surfaces (seeTable 1(1)). Describing the local geometry for thesetwo distinctly different solutions requires distinctly dif-ferent feature vectors. Because the size and definitionof the input feature vectors x must be consistent for agiven machine learning model, we necessarily trainedmultiple models, one per solution type.

At evaluation time, we can use our machine-learningmodels as follows:

• Predict the mesh quality outcomes for entities ina CAD model.

• Present the user with the list of entities, sortedfrom worst-to-best quality.

• For each entity in the list:

– Select a list of candidate operations for theentity.

– Predict the mesh quality outcome for eachoperation.

– Present the user with the list of operations,sorted from best-to-worst outcome.

Thus, the user is presented with a prioritized list ofitems to fix and operations to fix them. Inexperi-enced users can quickly defeature their model usinga “greedy” approach by repeatedly choosing the firstsuggestion for every problem area, while users withgreater experience are free to follow or ignore the sug-gested operations. We note that, while this greedyapproach to defeaturing may not be optimal, it can

provide inexperienced users with a principled, data-driven starting point for their work. We discuss alter-natives to the greedy approach in Section 9.4.

4. FEATURES

To predict meshing outcomes with respect to local ge-ometric entities requires characterization of the ge-ometry and topology in the local neighborhood ofeach entity within the CAD model. For each entityGR(R = 0, 1, 2) representing vertices, curves and sur-faces respectively, a characteristic feature vector xGR

was defined. In addition, local modification operationsOn(GR) that operate on GR, were chosen. Since indi-vidual operations could involve modification of multi-ple nearby entities, a unique feature vector xOn(GR) foreach operation was also defined. While there are manypossible choices for CAD operations, for purposes ofthis study we selected nine common operations avail-able in the Cubit Meshing and Geometry Toolkit [?, ?]which are illustrated in Table 1.

Each of the nine operations On(GR) in Table 1 hasa separate machine learning model with a distinct setof associated input features. In addition, three moremodels were created to characterize the unmodifiedentities GR, making a total of twelve models used forthis study. For each model, we tested several differenttypes of input feature vectors.

Figure 1: Small curve identified in CAD model

4.1 Expert Features

Expert features characterize GR and On(GR) basedupon a fixed-length set of numerical values describ-ing an entity and its relationships with its neighbors.For example, Figure 2 illustrates expert features fora given small curve from the CAD model shown inFigure 1. Table 2 describes the attributes used forexpert features for vertices, curves and surfaces. At-tributes in Table 2 are queried from a geometry engine

Geometryoperation name

Examplebeginning state

Example endingstate

(1) removesurface

(2) tweak replacesurface

(3) compositesurfaces

(4) collapse curve

(5) virtualcollapse curve

(6) tweak removetopology curve

(7) tweak removetopology surface

(8) blunttangency

(9) remove cone

Table 1: Geometry modification operations On(GR).Example beginning and ending states of each operationare illustrated.

Figure 2: Sample expert features used for training dataat a small curve

for each entity GR and used to construct xGR . To cre-ate xOn(GR), the vectors xGR for nearby entities wereconcatenated, so that the number of features used foreach model is based upon the geometric entities in-volved in the operation. For instance, the compositeoperation O3(G2) includes attributes describing thetwo surfaces G2 involved in the operation, as well asattributes from the neighboring curves and surfacess.In contrast, the collapse curve operation O4(G1) in-cludes features describing the curve to be collapsed,G1 and its adjacent surfaces.

Since each machine learning model requires a constantsize input feature vector x and local topology arrange-ments could include varying numbers of adjacencies,we truncate or extend the size of x to ensure a con-stant size. For example, the feature vector xG2 for asurface includes attributes from surface G2 as well asattributes from up to 4 adjacent curves and surfaces,where the adjacent surfaces are chosen from the twoshortest and two longest surrounding curves. For sur-faces with less than 4 curves, zeros are used to fill theremaining indices in xG2 .

4.2 Geometric Features

We introduce surflets and curvelets as complementaryapproaches for computing feature vector, xGR andxOn(GR). Figures 3 and 4 illustrate surflet pairs devel-oped by Wahl et. al.[?]. A surflet S = (α, β, γ, δ) is afunction of distance δ and angles, α, β, γ between twopoints with oriented normals on a surface as illustratedin Figure 5. Surflet pairs can be computed for anyunique pair of points on the surface of a geometry. Toconvert an arbitrary number of surflets into a constantsize vector xGR , a four-dimensional histogram with di-mensions defined by α, β, γ and δ is constructed. Thevalues α, β, γ and δ for each surflet are computed andassigned to a discrete bucket [I(α), I(β), I(γ), I(δ)].For our application we choose five intervals in each

vertex curve surface

largest anglebetween attachedcurves

arc length surface type (pla-nar, cylindrical,parametric)

smallest anglebetween attachedcurves

distance betweenend points

number of loops

tangency type distance frommid-point tosegment

number of curves

number attachedcurves

tangent angle atstart

area

is convex tangent angle atend

perimeter

exterior angle onvolume

longestcurve/perimeterratio

angle on surface 0at start

shortestcurve/perimeterratio

angle on surface 0at end

hydraulic radius

angle on surface 1at start

u principal curva-ture at mid-point

angle on surface 1at end

v principal curva-ture at mid-point

Table 2: Expert features computed for individual geo-metric entities

dimension resulting in a total of 625 unique buckets.Our feature vector, xG is therefore a vector of 625 in-tegers that record the number of surflet pairs classifiedwithin each bucket [I(α), I(β), I(γ), I(δ)].

Figure 3: Example model showing points and normalsused for computing surflets

Our implementation of surflet-based features requiresfirst triangulating the surfaces of the CAD model toobtain a discretization. We limit the number of pointsthat influence xGR and xOn(GR) to those falling withina bounding box surrounding entity GR or On(GR).For computational efficiency, we also limit the numberof points contributing to xGR and xOn(GR) to 1000when the local triangulation is dense, sampling thepoints at random from within the bounding box.

Figure 4: Close-up of a point and normal used for surfletcalculation

Figure 5: Surflet S is a function of the distance, δ andangles α, β, γ between two point / normal pairs on asurface

We note that surflet pairs can be computed fromany point on the surface of the geometry near GR

or On(GR). While providing an accurate representa-tion of the nearby geometry, it tends to neglect anyinfluence from the local topological arrangement ofcurves and surfaces. Since our objective in defeatur-ing is to identify changes to local topology throughoperations On(GR), we also explored an alternatemethod for constructing xGR and xOn(GR) which wecall “curvelets”. In contrast to surflets, curvelets limitselection of points to those on the geometric curvesthat are topologically adjacent to GR or On(GR). In-stead of limiting selection to a bounding box, we in-clude points on all adjacent curves at GR or On(GR).In addition, rather than using the normal vector ata point, curvelets use the tangent vector on its asso-ciated curve. Surflets and curvelets can be used incombination or independently. We examine the char-acteristics and accuracy of surflets and curvelets andtheir combined effect in Section 7.

5. GROUND TRUTH

For each machine learning model associated with en-tity GR and operation On(GR), we needed to providea ground truth output vector, which we do by generat-ing a tetrahedral mesh and evaluating the local meshquality. Automatic mesh generation methods oftenprovide a variety of built-in algorithms to automati-cally improve or mitigate dirty geometry and we wishto take advantage of these capabilities. However, thismeans that the choice of meshing tool will have a sig-nificant effect on the resulting mesh quality, depend-ing on the local topology, geometric modifications, andmeshing parameters selected. Consequently, while our

proposed methods are general and could be applied tousing any automatic meshing tool, the ground truthvalues generated for this study are specific to the mesh-ing tool and would not be transferable to another toolwithout re-training. In this work, we used the toolsdescribed in [?, ?] to generate our ground truth.

5.1 Mesh Quality Metrics

While any mesh quality metric [?] could be used toevaluate a mesh, we select three specific metrics basedupon their representative characteristics. These in-clude scaled Jacobian, in-radius and deviation.

Scaled Jacobian: The scaled Jacobian, Msj is de-fined as the minimum Jacobian at any of the four ver-tices of a tetrahedron divided by the lengths of itsthree adjacent edges. Msj = 1 represents a perfectlyshaped equilateral tetrahedra, while Msj < 0 definesan inverted element. We utilize Msj as a ground truthas it is independent of mesh size and is representa-tive of the Jacobian mapping function used in finiteelement methods.

In-Radius: The in-radius, Mir is defined as the ra-dius of an inscribed sphere within a tetrahedra. Sincethis value is an absolute distance, we utilize a scaledin-radius value Msir. Msir is defined as Mir/Mir(ST ),where Mir(ST ) is the in-radius of an equilateral tetra-hedra with edge length equal to target mesh size ST .A value of Msir = 1 represents a perfectly sized ele-ment, while Msir < 1 is smaller than ST and Msir > 1is larger than ST . For training purposes we generatedata at multiple target mesh sizes. We include Msir asa ground truth to learn characteristics that will avoidsmall elements that may result in long run-times forexplicit FEA codes.

Deviation: The deviation, Md, metric is defined asthe distance from the centroid of a surface triangle toits closest point on the geometry. Unlike Msj and Msir

that describe characteristics of a tetrahedra, Md is atriangle metric that measures how closely the bound-ary of the mesh conforms to the prescribed geometry.For this metric we also compute a scaled deviationMsd = Msd/ST . A value of Msd = 0 represents a tri-angle that perfectly matches the geometry. Values ofMsd > 0 will be necessary for any geometry with cur-vature, however minimizing Msd is beneficial to ensurethe mesh adequately represents the input geometry.In this case, the maximum value for Msd defines theworst quality.

Success/Failure: We note that candidate opera-tions are identified for each small entity in a CADmodel from a generic set of options. As a result, theparticular local arrangement of curves and surfacesfor a selected operation may not be valid. In mostcases, the success or failure of an operation can only

be determined by actually performing the operationand recording the result. Whether the CAD opera-tion, On(GR) is successful and its subsequent mesh-ing is successful, is also recorded and used as a labelMsuccess. This information is useful for identifyingand eliminating solutions that would not be effectivefor defeaturing.

5.2 Locality of Mesh Quality Metrics

Assuming the mesh generation is successful followinga CAD model modification, nearby tetrahedra and tri-angles at GR can be evaluated and a controlling mini-mum Msj , Msir and maximum Msd returned as a rep-resentative ground truth for operation On(GR). Forthis study we define the locality of the mesh near GR

using two methods: bounding boxes, and local topol-ogy.

Bounding Box: We identify a set of tetrahedra,TB falling within a Cartesian aligned bounding boxB(GR) surrounding entity GR. The extent of B(GR)is computed by adding the target mesh size, ST to allsides of a tight bounding box surrounding GR. Foroperations On(GR), the set TB includes a boundingbox surrounding all entities involved in the operation.Figure 6 illustrates the set of tetrahedra, TB fallingwithin B(G1) defined by a small curve, G1. Only thosetetrahedra in TB falling within B(GR) are consideredwhen computing the controlling metrics for Msj , Msir

and Msd.

To compute TB for an operation On(GR), the entitiesinvolved in the operation are identified prior to per-forming the operation and their combined boundingbox computed. Once On(GR) is performed and a meshgenerated, the controlling metrics can be computed.While simple to implement, depending on the arrange-ment of topology, TB may encroach on other nearbyentities where the controlling metric may conflict. Wealso note that the bounding box method is sensitiveto orientation of the CAD model, as B(GR) will bealigned with the global coordinate axis. To overcomethese issues, we also introduce a method based uponthe local topology at GR.

Local Topology: We can identify the set of tetra-hedra, TT that share at least one mesh node on GR

as well as those tetrahedra immediately attached tothose at GR. The local topology method for comput-ing the controlling metrics is based only upon thosetetrahedra in TT . Figure 7 illustrates the local set oftetrahedra TT associated with a small curve. SinceTT includes only those tetrahedra in contact with GR

and those immediately adjacent, it is less likely thatTT will encroach on neighboring entities. It also hasthe advantage of being insensitive to geometry orien-tation.

Figure 6: Example set of tetrahedra TB defined bybounding box B(G1) surrounding small curve G1

We note that TT is convenient to compute forGR priorto performing geometry operations as we can easilyquery for the set of nodes associated with GR. How-ever following operation On(GR), entity GR may nolonger exist. For example, following the remove sur-face operation illustrated in Table 1, the surface G2

no longer exists, but is instead replaced by a curve de-fined by the intersection of two extended surfaces. Asa consequence, it is necessary to identify one or moresurviving entities for each operation On(GR) on whichthe set of tetrahedra TT can be discerned. For the re-move surface example, the surviving entity would bea single curve. TT in this case would be defined bythe set of nodes associated with the surviving curve.A similar set of surviving entities is also identified foreach operation On(GR).

Figure 7: Example set of tetrahedra TT defined by nodesassociated with small curve G1

6. MACHINE LEARNING MODELS

To generate training data for our study, we used asmall sampling of 94 single-part CAD parts obtained

from the open internet resource GrabCAD [?]. Fig-ure 8 illustrates a few of these CAD parts used fortraining in this study. For each part, we generatedtraining data for the twelve geometric operations de-scribed above, including 3 no op models. This in-volved identifying small entities, where small was afunction of a range of four target mesh sizes. We alsoidentified bad vertices based on tangent or sharp an-gle conditions at the vertex. Relevant CAD opera-tions selected for each small entity or bad vertex wereidentified from a pre-defined set of operations for eachentity type. Some culling of relevant operations wasinitially accomplished to reduce the number of oper-ations that needed to be trained. For example, theremove cone operation was trained only for entitieswhere the underlying CAD kernel identified it as aconic surface type. Similary blunt tangency was onlyselected for vertices with adjacent curves forming anangle less than 10 degrees. For this reason the numberof observations varied widely for each operation type.Table 3 shows the number of observations extractedfrom the CAD parts used for this study for each of the12 operations trained.

6.1 Training Data

To generate training data, we used the following pro-cedure:

1. Import CAD part

2. Compute a range of four target auto-sizes ST

based on characteristics of the part. Do steps3 to 10 for each ST

3. Identify a list of the small entities and sharp ver-tices, GR based on ST . Do steps 4 to 10 for eachGR.

4. Identify a list of relevant operations On(GR) forentity GR. Do steps 5 to 10 for each On(GR)

5. Compute fixed-length vectors of features xOn(GR)

for operation On(GR), including expert features,geometric features, and combinations of both.

6. Perform CAD operation On(GR)

7. Mesh the part with size = ST

8. Record success or failure of the geometry opera-tion and meshing as label Msuccess

9. If geometry and meshing are successful, computemetrics, Msj , Msir and Msd based on locality(TT and/or TB)

10. Write one row to a .csv file containing featuresxOn(GR), label Msuccess, and ground truth Msj ,Msir, Msd

We note that in some cases there were failures in thisprocess, either because an operation failed, or mesh-ing failed. This is indicated in step 8 of the aboveprocedure. Table 3 also lists the number of failuresfor each operation in the study. When an operationfailed, it was occasionally the result of a software de-fect. More often, the geometric kernel could not re-solve the topology for the given input. For instance, aremove surface is attempted on all surfaces identifiedas small. The local topology may not be resolvablefor all cases where a small surface is to be removed.As a result, for our specific study, over half of the re-move surface operations failed. We did not distinguishbetween those that failed due to a software defect andthose that failed due to topology. Instead, our modelswere trained to predict for any case of failure.

Failures in the subsequent meshing step happenedwhen an operation succeeded, but its modificationsaffected the local topology so badly that the mesherwas unable to proceed. In either case, we kept trackof a categorical ground truth metric Msuccess captur-ing whether the combination of operation and mesh-ing succeeded or failed. Only operations that com-pleted successfully recorded values for ground truthMsj , Msir and Msd.

Num

Obs.

Num

Failed

Num

Trained

vertex no op 1348 0 1348

curve no op 9842 0 9892

surface no op 5842 0 5842

remove surface 17,624 10,026 7598

tweak replace surface 2569 1152 1417

composite surfaces 43,551 5020 38,531

collapse curve 13,830 2113 11,717

virtual collapse curve 14,955 14,743 212

remove topology curve 7056 5175 1881

remove topology surface 3890 3102 788

blunt tangency 8059 3982 4077

remove cone 232 20 212

Totals 128,484 45,333 83,515

Table 3: Numbers of observations extracted from the 94CAD parts used in this study.

6.2 Training

We used Python [?] and the scikit-learn library [?] totrain machine learning models for each geometric op-eration. This included twelve classification models topredict whether an operation was likely to succeed orfail (see Section 7.1), and twelve regression models topredict our six per-operation mesh quality metrics (seeSection 7.2). Thus, we created a total of 24 models.

Figure 8: Examples of the 94 CAD models used for training in this study.

Due to the novelty of our problem, there was greatuncertainty over which (if any) of the features in ourtraining data would be useful for generating machinelearning models. For some model types, redundant, ir-relevant, or misleading features can negatively impactthe performance of the model. Thus, feature selec-tion - including the removal of correlated features - iscommon in these cases.

In our case, we choose to use ensembles of decisiontrees (EDTs) instead [?]. An EDT is a collection ofindividual decision trees, each of which is trained on asubset of the full training data using a technique calledbagging [?]. At evaluation time, the EDT’s predictionis a weighted sum of the predictions of each of its indi-vidual trees. Colloquially, EDTs capture the “wisdomof crowds” by allowing each tree to become an experton a subset of the data, using that localized wisdomto vote for a final result.

The beauty of EDTs is in their proven ability to exploitweak signals, even in features that are only slightly lessthan perfectly correlated. Thus, we did not perfomany feature thinning before training, finding it to beactively harmful to model accuracy.

Further, EDTs make popular general purpose machinelearning models due to their easy interpretability (eachtree contains a set of branching boolean tests that are

applied to the input features, with output predictionsstored in the leaf nodes) and their ability to computefeature importance metrics that capture how often agiven feature is useful when arriving at a decision (seeFigure 22 for sample feature importance outputs). Itis worth noting that EDT feature importance metricscapture a more nuanced view of the input featuresthan a simple correlation matrix, as they are a reflec-tion of the rich, nonlinear representation space of theindividual trees in the ensemble.

An important hyperparameter affecting the perfor-mance of an EDT is the number of trees that arecontained within it. When training EDTs, we lookto see that the performance of the ensemble convergesasymptotically as the number of trees increases. Fig-ure 9 illustrates this behavior for one of our models(note that the error is decreasing, which is synony-mous with increasing performance). Using plots likeFigure 9, we chose by inspection to use 75 trees eachfor the ensembles in the following experiments.

7. RESULTS

To evaluate the performance of a machine learningmodel, we typically split the available training datainto randomly-chosen partitions: one containing dataused to train the model, and one containing data held

0 50 100

Tree count

0.00

0.01

0.02

0.03

Mea

n a

bso

lute

err

or

topo_min_sjtopo_min_irtopo_max_devbbox_min_sjbbox_min_irbbox_max_dev

Figure 9: Performance of an EDT model versus the num-ber of trees, for our six metrics.

back to test performance after training is complete.This makes it possible to identify models that haveoverfit or memorized their training data, by evaluat-ing them solely on the unseen test data.

However, a single partitioning by itself complicatesperformance evaluation, since the partitions might,through random chance, lead to models with unre-alistically high (or low) performance. This problemis compounded when comparing models (for example:comparing two models to see whether expert or surfletfeatures produce better outcomes), since a model withan “unlucky” partitioning might be penalized unfairlywhen compared against a model with a “lucky” set ofpartitions.

To combat this effect, all of the following results werecomputed using 5× 2 cross validation, which involvesrandomly partitioning the training data into two equalsize sets; training a model on the first set and testingit on the second; training a model with the second setand testing it on the first; repeating this process fourmore times for a total of ten models. The resultingcross validated results are the averaged results fromthe individual models. Using 5 × 2 cross validationthus provides an extremely conservative estimate ofa model’s performance, and is widely used in the ma-chine learning community when comparing two modelsto see which is best.

7.1 Failure Prediction Models

First, we evaluated the performance of our twelve per-operation failure prediction models. Because the fail-ure prediction models are classification models thatproduce a single categorical “succeed” or “fail” out-put, we evaluated their performance using precisionand recall metrics. In this case precision is the per-

Figure 10: Percentage of meshing operations that suc-ceeded, by type, sorted by success rate. The first threeoperations always succeed because they are do-nothingplaceholders.

centage of operations predicted to fail that actuallyfailed, while recall is the percentage of actual failuresthat were predicted to be failures. An ideal modelshould balance high recall (avoiding false negatives)with high precision (avoiding false positives).

From Table 3 and Figure 10, we see that there weremany failures encountered during training data gener-ation, and that some operations failed more often thanthey succeeded for the local arrangement of curvesand surfaces. This suggests that the failure predictionmodels could play a significant role in avoiding prob-lems for the end user, by steering them away from op-erations that are unlikely to succeed for a given regionwithin the geometry.

Figures 11 and 12 show the precision and recall scoresrespectively for failure prediction models trained usingfour sets of features: expert, surflet, curvelet, and acombination of all three. The results are grouped byoperation, and the groups are sorted using the scoresfor models trained using only expert features. In allcases, larger values are better.

Figure 12 shows that all of our models achieve excel-lent (nearly 100%) recall, aggressively identifying all ofthe actual failures in the training data. However, Fig-ure 11 paints a more complex picture, with the modelsachieving a wide range of precision scores. The mod-els with low precision scores are too aggressive, sincelow precision in this case means that the model is pre-dicting failures for operations that actually succeededin real life. Models with a precision lower than 0.5(marked with the dashed line in Figure 11) are wrongmore often than right, and would not be useful in prac-

verte

x_no

_op

surfa

ce_n

o_op

curv

e_no

_op

com

posit

e_su

rface

s

rem

ove_

cone

colla

pse_

curv

e

blunt

_tan

genc

y

rem

ove_

surfa

ce

twea

k_re

place

_sur

face

rem

ove_

topo

logy_

curv

e

rem

ove_

topo

logy_

surfa

ce

virtu

al_co

llaps

e_cu

rve

0.0

0.5

1.0

Pre

cisi

on

Expert

Surflet

Curvelet

Combined

Figure 11: Failure prediction model precision. Larger values are better.

verte

x_no

_op

surfa

ce_n

o_op

curv

e_no

_op

com

posit

e_su

rface

s

rem

ove_

cone

colla

pse_

curv

e

blunt

_tan

genc

y

rem

ove_

surfa

ce

twea

k_re

place

_sur

face

rem

ove_

topo

logy_

curv

e

rem

ove_

topo

logy_

surfa

ce

virtu

al_co

llaps

e_cu

rve

0.0

0.5

1.0

Rec

all

Figure 12: Failure prediction model recall. Larger values are better.

tice. It is interesting to note that the performance ofthe models in Figure 11 correlates closely with the fail-ure rate in Figure 10, suggesting that unskewing thedata may improve performance. Regardless, it is clearfrom the figures that the models trained using expertfeatures have considerably higher performance thanthose trained with the other features in every case,even models trained with a combination of all threefeature types. Further, we see that curvelet featuresproduce slightly better results than surflets in mostcases.

7.2 Mesh Quality Models

Next, we evaluated the performance of our twelve per-operation mesh quality models. As in the previous sec-tion, the models were evaluated using the expert, sur-flet, curvelet, and combined features. Since these mod-els produced regression outputs predicting the scaledJacobian Msj , scaled in-radius Msir, and scaled devi-ation metrics Msd, computed using bounding-box TB

and local topology regions TT , there were a total ofsix outputs for each model. Because these metrics arecontinuous rather than categorical, the model resultsare reported using mean absolute error (MAE), the

virtu

al_co

llaps

e_cu

rve

rem

ove_

topo

logy_

curv

e

rem

ove_

topo

logy_

surfa

ce

rem

ove_

cone

colla

pse_

curv

e

blunt

_tan

genc

y

curv

e_no

_op

com

posit

e_su

rface

s

surfa

ce_n

o_op

rem

ove_

surfa

ce

verte

x_no

_op

twea

k_re

place

_sur

face

0.00

0.01

0.02

0.03

0.04

Mea

n a

bso

lute

err

or

Expert

Surflet

Curvelet

Combined

Figure 13: Mesh quality model MAE averaged across mesh quality metrics. Smaller values are better.

rem

ove_

topo

logy_

curv

e

rem

ove_

topo

logy_

surfa

ce

virtu

al_co

llaps

e_cu

rve

colla

pse_

curv

e

rem

ove_

cone

curv

e_no

_op

blunt

_tan

genc

y

surfa

ce_n

o_op

com

posit

e_su

rface

s

twea

k_re

place

_sur

face

rem

ove_

surfa

ce

verte

x_no

_op

0.00

0.05

0.10

0.15

Mea

n a

bso

lute

err

or

bbox_min_sj

topo_min_sj

Figure 14: Comparing bounding-box and local topology scaled Jacobian metrics. Smaller values are better.

average of the absolute differences between the pre-dicted and actual post-meshing quality metric values.Note that, in contrast to the failure prediction results,smaller MAE values are better.

Since the expert features performed so strongly in ourfailure prediction models, we began by looking forsimilar patterns with our mesh quality models. Asseen in Figure 13, expert features once again performsignificantly better than the geometric features, andcurvelets continued to perform better than surflets ina majority of cases.

These results lead us to assert that geometric or shapecharacteristics by themselves are insufficient to accu-rately inform a model of the effects of CAD opera-tions on a mesh. Instead, our results illustrate thatthe topological characteristics identified by our expertfeatures are needed in order to more precisely predictmeshing outcomes, and that more topological infor-mation is better than less.

Note that these results illustrate general trends amongthe different feature types, and that the MAE in Fig-ure 13 is an average of the MAE for all six mesh qual-

rem

ove_

cone

rem

ove_

topo

logy_

curv

e

rem

ove_

topo

logy_

surfa

ce

blunt

_tan

genc

y

colla

pse_

curv

e

virtu

al_co

llaps

e_cu

rve

verte

x_no

_op

com

posit

e_su

rface

s

rem

ove_

surfa

ce

twea

k_re

place

_sur

face

curv

e_no

_op

surfa

ce_n

o_op

0.00

0.02

0.04

0.06

Mea

n a

bso

lute

err

or

bbox_min_ir

topo_min_ir

Figure 15: Comparing bounding-box and local topology in-radius metrics. Smaller values are better.

rem

ove_

topo

logy_

surfa

ce

rem

ove_

topo

logy_

curv

e

com

posit

e_su

rface

s

colla

pse_

curv

e

rem

ove_

cone

verte

x_no

_op

blunt

_tan

genc

y

twea

k_re

place

_sur

face

surfa

ce_n

o_op

virtu

al_co

llaps

e_cu

rve

rem

ove_

surfa

ce

curv

e_no

_op

0.00

00.

004

0.00

80.

012

Mea

n a

bso

lute

err

or

bbox_max_dev

topo_max_dev

Figure 16: Comparing bounding-box and local topology deviation metrics. Smaller values are better.

ity metrics for each operation. Results for each of themesh quality types Msj , Msir and Msd are illustratedseparately in Figures 14, 15 and 16 respectively.

Since expert features proved more accurate in charac-terizing our CAD operations, we limit demonstrationof performance of each of our three target metrics inFigures 14 through 16 to expert features. In these fig-ures, we compare the performance of the two localitychoices, bounding box TB and topology TT . We ob-serve that for both scaled Jacobian (Figure 14) andscaled in-radius (Figure 15) most operations had anMAE in the range of 0.05 or less. That means that

predictions of Msj and Msir from the proposed ma-chine learning models can be expected to be less than0.05 on average. For Msd (Figure 16) the MAE wasless than 0.01 in most cases.

When comparing TB and TT the results were muchmore nuanced, with bounding-box locality providing asmall-but-consistent performance boost for the scaledJacobian metric, roughly identical performance for thein-radius metric, and consistently lower performancefor the deviation metric. Based on the ambiguity ofthese results, we would likely choose to use local topol-ogy based regions in production, since they select sig-

nificantly fewer tetrahedra, tend not to encroach onadjacent geometric features, and are orientation in-variant, as described in Section 5.2.

We also note that for all operations trained, we com-puted performance only for those operations predictedto succeed as indicated by our failure models (see Fig-ure 10). This tended to limit our sample size for someof the operations such as virtual collapse curve and re-move topology surface. This may account for reducedaccuracy in some of our predictions as illustrated bythe outlier virtual collapse curve performance in Fig-ure 14.

8. APPLICATION

Each model was serialized to disk for use with inter-active or automatic defeaturing. To demonstrate theproposed ML-based defeaturing environment both aninteractive graphical tool and an automatic greedy al-gorithm were implemented.

8.1 Interactive GUI

A graphical user interface panel was implemented forthe Cubit geometry and meshing toolkit [?, ?]. Fig-ure 17 illustrates the modified panel that we used tomanage and drive defeaturing. In this environment,a list of entities GR shown in Figure 17(h) predictedto cause poor quality are listed ordered from worst tobest based upon the selected metric (Figure 17(e)).Only those entities with predicted quality below auser-defined threshold shown in Figure 17(g) are dis-played.

Selecting entity GR will reveal a list of possible solu-tions On(GR) shown in Figure 17(i) prioritized basedon predicted mesh quality. Predicted quality outcomesare shown in parentheses next to each operation. Theuser can preview the operation, accept it, modify thesolution, or ignore it all-together.

A simple illustration of how this GUI might be usedis shown in Figure 18 and Table 4. In this example, atangency condition at vertex 143 is predicted to resultin nearby tetrahedra with a minimum Msj of approx-imately 0.0725. To improve the quality, the operationcomposite create surface 77 42 is suggested whichis predicted to result in Msj of approximately 0.1435.In this example, the the machine learning models pre-dict that combining or compositing the two adjacentsurfaces 77 and 42 will improve the local mesh qual-ity. The user can choose to accept this suggestion, orchoose an alternative. To evaluate the accuracy forthis one example, Figure 19 shows a mesh generatedbefore and after performing the composite operation.The local mesh quality at vertex 143, illustrated inFigure 19(a) results in Msj of about 0.1025, an error

Figure 17: A proposed graphical interface for drivingML-based defeaturing. (a) List of volumes to be defea-tured. (b) User defined size considered small. (c) Buttonto detect and populate panel with small features. (d)Option to use ML. Loads ML models. (e) User selectstarget metric for criteria (minimum Msj , Msir or maxi-mum Msd) (f) Button to execute automatic defeaturingusing greedy criteria. (g) Limit for display of predictedworst quality. (eg. entities with predicted Msj less than0.2 are listed) (h) Ordered list of entities showing pre-dicted mesh quality. (i) Ordered list of CAD operationsto correct the selected entity in (h), prioritized by pre-dicted quality.

Figure 18: Defeaturing example where tangency con-dition exists at vertex 143. Machine learning modelspredict that a composite operation between neighbor-ing surfaces 42 and 77 will improve mesh quality at thisvertex.

of about 0.03. Similarly we show the mesh followingthe operation in Figure 19(b) where the local mesh

OperationNum

tets

Global

min Msj

Num tets

Msj < 0.2

Predicted

no-op

Actual

no-op

Predicted

op

Actual

op

Initial 269957 0.1025 5

composite create surface 77 42 268528 0.1511 3 0.0725 0.1025 0.1435 0.1935

Table 4: Example of the effect on scaled Jacobian Msj from a single operation performed on CAD model shown inFigures 18 and 19. Comparison to actual mesh quality is also shown.

Figure 19: Tetrahedral meshes based on geometry inFigure 18 (a) Mesh at vertex 143 before suggested de-featuring operation. (b) following defeaturing operation.Mesh quality for this example is illustrated in Table 4.

quality resulted in Msj of 0.1935, an error of about0.05.

8.2 Greedy Algorithm

The user also has the option to accept the best pre-dicted solutions without having to manually executeeach one individually. The button shown in Fig-ure 17(f) runs a greedy algorithm as follows: the pre-dicted worst quality entities GR are successively modi-fied using the best predicted operations On(GR). Thiscontinues until all entities have a predicted quality ex-ceeding the user specified threshold at Figure 17(g).

We illustrate a simple greedy procedure in Table 5and Figures 20 and 21. In this example, seven opera-tions are automatically selected based upon minimumscaled in-radius Msir predictions. In this case, Ta-ble 5 shows the number of small elements falling belowa user defined threshold of Msir = 0.2 reduced fromover 10,000 to 1 and the minimum Msir increased from0.0078 to 0.1628. The CAD operations composite andtweak remove topology are used to defeature the modelas illustrated in Figures 20 and 21.

We note that the proposed machine learning modelsautomatically identify the conical surfaces illustratedin Figure 20(a) as those that will produce an unfa-vorable Msir metric. In this case, the surface meshgenerator used for this study [?] will characteristically

identify and refine to the apex of the conical sur-faces. Training has identified this characteristic andour models correctly predict the mesh quality out-come. Figure 20(b) shows a portion of the surfacemesh at the conical surfaces if the suggested compos-ite operation had not been applied. Figure 20(c) showsthe same surfaces once the composite is applied. Notethat the proposed models correctly identified the com-posite operation as the best method for increasing thetarget mesh quality, Msir.

Similarly, the machine learning models have predictedthat the small curve 218 illustrated in Figure 21(a)will produce mesh quality Msir less than the user-prescribed threshold of 0.2. The best choice for im-proving this condition was predicted to be the CADoperation, tweak remove topology curve, illustrated inFigure 21(a). The surface mesh without applying thisoperation is shown in Figure 21(b) and the resultingmesh, if the operation is applied, is shown in Fig-ure 21(c).

Although the principal purpose of the proposed ma-chine learning models is to predict and correct theworst quality artifacts in a CAD models without mesh-ing, to evaluate the accuracy of our methods, Table 5compares the predicted metrics to the actual meshquality from meshes produced before and after the op-eration is performed. For example, Table 5 shows thatthe predicted quality at curve 218 would change fromMsir = 0.1039 to 0.2935 as a result of performing theindicated operation. We compare that with the actualmesh quality values of 0.1070 and 0.2813 respectively.

9. CONCLUSIONS

A new application of modern machine learning tech-nologies to prepare models for simulation has been in-troduced. We have demonstrated the ability to ac-curately predict mesh quality based on local topologyof a CAD part without having to generate a mesh.We have also introduced methods for identifying CADoperations to effectively defeature a CAD model bypredicting meshing outcomes for a range of selectedoperations. A study based on a limited set of 94 open-source CAD parts was used to generate training datafor 24 separate models that predict failure and meshquality metrics. New methods for computing featuresand ground truth were introduced and their accuracy

OperationNum

tets

Global

min Msir

Num tets

Msir < 0.2

Predicted

no-op

Actual

no-op

Predicted

op

Actual

op

Initial 269957 0.0078 10000+





tweak remove topology curve 218 244754 0.1934 2 0.1039 0.1070 0.2935 0.2813



Table 5: Example of CAD operations performed automatically from greedy predictions of minimum scaled in-radius Msir.Operations and mesh illustrated in Figures 20 to 21

(a)

(b) (c)

Figure 20: Illustration of initial 4 composite operationsfrom Table 5 generated from the greedy method. Oper-ations were automatically identified based on predictionsof minimum scaled in-radius Msir. (a) 4 holes with con-ical shafts are automatically identified. (b) Mesh pro-duced on surfaces of holes without applying compositeoperations. Note that mesh generator automatically re-fines to cone apex. (c) Resulting mesh after compositeoperations applied.

assessed.

9.1 Feature Importance

We introduced new methods for defining features forour machine learning methods. We found that meshquality predictions based on expert features were moreaccurate than geometric features that used surflets andcurvelets. Expert features selected for each entity typeand operation included local attributes such as arclengths, angles, areas, curvatures and other character-istics. Although results indicated that the selected at-tributes in this implementation led to reasonably accu-rate predictions, additional study would be warrantedto identify additional features of value.

(a)

(b) (c)

Figure 21: Example of tweak remove topology curveoperation used in the greedy procedure from Table 5.(a) (Top) Curve 218 is predicted to have poor quality,Msir (Bottom) tweak remove topology curve operationapplied to curve 218. (b) Surface mesh at curve 218without applying operation. (c) Surface mesh after ap-plying operation.

Figure 22 illustrates the relative importance of someof the expert features for 8 of the 12 operations usedin this study. A higher importance for a specific fea-ture indicates its relative influence on the mesh qualitypredictions produced by the machine learning mod-els. While there is no consistent pattern in featureimportance, it is worth noting that mesh size tendsto show up among the top four features for all mod-els. The target mesh size ST is included as a featurefor all models. Intuitively, these results indicate thatmesh quality predictions are heavily dependent upon

the user prescribed mesh size.

Although a definitive optimal set of features for theprescribed operations is beyond the scope of this study,understanding which features are most useful can helpin understanding the problem domain and improvingfuture implementations. For example, our trainingmodels for this study currently incorporate a rangeof four automatically selected mesh sizes. The impor-tance that the mesh size feature plays in our resultssuggests that further training with additional meshsizes is warranted.

9.2 Ground Truth

Ground truth for this application was defined by a setof mesh quality metrics including scaled Jacobian, In-radius and deviation. These were selected based oncommon requirements for analysis codes that requirewell-shaped, isotropic elements of consistent size andgrading that conform well with the domain. For otherapplications that may require anisotropic elements orprescribed minimum elements through the thickness,additional or alternative metrics would need to be usedas ground truth.

In this study, we also proposed two different meth-ods for characterizing the locality of mesh quality met-rics. The bounding box and topology-based methodstended to yield similar predictions, but we concludedthat the use of topology-based locality was preferredsince it represented the local environment of the oper-ation better than an orientation insensitive boundingbox.

We note that this study limits ground truth to meshquality metrics that can be computed from an auto-matically generated tetrahedral mesh. Although meshquality is an important factor in preparing a simula-tion model, there are many other factors that are notas easily characterized. For example, known loads,boundary conditions and other physics-based proper-ties may influence the defeaturing performed by ananalyst or engineer. These physically-based character-istics may also need to be considered when identifyingfeatures and ground truth for future implementation.

9.3 Software Considerations

For this study we selected a set of CAD operations fortraining from the Cubit geometry and meshing toolsuite [?]. We also selected a commercial mesh gener-ation tool, Meshgems [?] that served as the basis fortraining and defining our ground truth metrics. It isworth pointing out that the methods introduced arenot specific to our implementation environment. In-deed, further work should be enlisted to identify an im-proved set of operations that can take advantage of a

more comprehensive, robust and flexible CAD model-ing environment. Additionally, alternative automaticmesh generation tools could be enlisted to train andidentify ground truth metrics.

This study limited the number of input training mod-els to a set of open-source CAD parts obtained fromGrabCAD [?]. This was done to ensure reproducibil-ity and provide a baseline for subsequent studies. Fordeployment in a practical defeaturing environment, atool for selecting and building training data based oncommon analysis use cases would be preferable. Se-rialized data that can be queried at run-time from adefeaturing tool would be updated and customized asnew CAD models are encountered.

We also recognize that in practice, depending on theCAD tool in use, an experienced analyst may identifya single CAD operation that involves multiple nearbygeometric entities (meta-entities) to accomplish a sin-gle defeaturing task. For example, defeaturing meta-entities such as a boss or blend could be accomplishedby applying a single remove operation. While this cansometimes result in a preferred outcome, for purposesof this study, because of limitations in our CAD tool,we currently limit operations to single entities, leavingmeta-entities for future work. Parametric modelingenvironments that maintain meta-entities or feature-recognition procedures could identify groupings of ge-ometric curves and surfaces from which single opera-tions could be identified and trained in a similar man-ner to that introduced in this work.

9.4 Reinforcement Learning

We have implemented a supervised machine learningapproach to assist in CAD defeaturing. The proposedautomatic greedy method suggests the best CAD al-teration at each step, given the current state of the as-sociated mesh. It may happen, however, that this ap-proach could become mired in a local minimum, suchthat multiple coordinated actions are required to re-move a particular undesired feature.

In addition to our greedy approach, we are also consid-ering the use of Reinforcment Learning (RL). RL is anapproach which can consider multiple coordinated ac-tions, even arbitrary length sequences of actions lead-ing to global minimums [?, ?]. It is, however, compu-tationally expensive and can be difficult to generalizefrom one context to another.

We will describe our work using RL in a later report,but we have so far achieved promising initial resultson a simple CAD model using Q-learning. We havediscovered global minimums using short sequences ofCAD operations. We have also enumerated statisticsshowing that the defeaturing problem can be very diffi-cult with many potential actions that can be detrimen-

tal to the overall improvement of the resulting mesh.In the future, we plan to generalize our approach toarbitrary CAD models that incorporate geometric fea-tures and potentially textual descriptions of CAD op-erations, all using a deep Q-learning framework.

Feature

mesh_size 1.65771

exterior_angle 0 1.23113

curvature2 1 0.524542

largest_angle 0.425724

short_curve_ratio 1 0.414681

shortest_curve_length 0 0.381628

long_curve_ratio 0 0.353482


max_angle 1 0.210075


Importance

(a) vertex no op

Feature

max_angle 0 2.39602

mesh_size 0.864585

face_angle11 0 0.698811

rel_mid_deviation 0 0.650515





hydraulic_radius 1 0.152276

rel_arc_length 0 0.133412

Importance

(b) curve no op

Feature

arc_length 2 2.96364

max_angle 3 1.07613


mesh_size 0.510938

min_angle 3 0.472651

num_loops 1 0.40401





Importance

(c) surface no op

Feature

mesh_size 2.51458


area 0 0.439399






min_angle 0 0.158146

perimeter 1 0.152295

Importance

(d) composite surfaces

Feature


mesh_size 0.763937

num_curves 0 0.728801

angle 0.699596






area 2 0.231626

Importance

(e) blunt tangency

Feature

mesh_size 1.81267





longest_curve_length 4 0.308023





Importance

(f) remove topology curve

Feature

mesh_size 4.07086



area 0 0.525383

perimeter 0 0.431603






Importance

(g) remove cone

Feature

mesh_size 1.58561






area 4 0.220314



rel_arc_length 3 0.159437

Importance

(h) remove surface

Figure 22: Feature importance: Top 10 expert features ranked by their importance for 8 of the 12 operations used inthis study. Feature importance was computed as a by-product of the ensemble of decision trees (EDT) method used togenerate our machine learning models. Note: values do not sum to one due to truncation and cross validation.

Documents

CAD DEFEATURING USING MACHINE LEARNING › _assets › documents › 2019_IMR... · CAD DEFEATURING USING MACHINE LEARNING Steven Owen1 Timothy M. Shead2 Shawn Martin3 1Sandia National