Local Update of Support Vector Machine Decision

Boundaries

Anirban Basudhar∗ and Samy Missoum†∗

Aerospace and Mechanical Engineering Department, The University of Arizona, Tucson, AZ, 85721, USA

This paper presents a new adaptive sampling technique for the construction of locallyrefined explicit decision functions. The decision functions can be used for both deterministicand probabilistic optimization, and may represent a constraint or a limit-state function.In particular, the focus of this paper is on reliability-based design optimization (RBDO).Instead of approximating the responses, the method is based on explicit design spacedecomposition (EDSD), in which an explicit boundary separating distinct regions in thedesign space is constructed. A statistical learning tool known as support vector machine(SVM) is used to construct the boundaries. A major advantage of using an EDSD-basedmethod lies in its ability to handle discontinuous responses. A separate adaptive samplingscheme for calculating the probability of failure is also developed, which is used within theRBDO process. The update methodology is validated through several test examples withanalytical decision functions.

I. Introduction

There are various challenges that are faced by any simulation-based design methodology. Two of the majorchallenges are - reduction of computational time and addressing problems with discontinuous responses.The necessity to consider uncertainties during the design makes the process even more tedious. Reductionin the computational time is often achieved by replacing actual models by surrogate models such as responsesurfaces1 and metamodels.2 In order to build the surrogate models, the system response is studied at se-lected design configurations given by a design of experiments (DOE).3,4 The computationally inexpensivesurrogate model built using these selected configurations is then used in place of the actual model to carryout optimization or to calculate the probabilities of failure using efficient Monte-Carlo simulations (MCS).5

However, if a DOE is used without any adaptive scheme then the number of samples required to constructan accurate model increases exponentially with the problem dimensionality.

Several adaptive methodologies have been developed for constructing surrogate models accurately with areduced number of function evaluations.6–8 However, most methods that have been developed do not con-sider the presence of discontinuities in the responses.9 Response discontinuities present a serious problem formost optimization or probabilistic techniques. In optimization, this restricts any traditional gradient-basedmethods or response approximation techniques. When considering reliability, discontinuities hamper theuse of approximation methods such as First and Second Order Reliability Methods (FORM and SORM),10

Advanced Mean Value (AMV),11 or Monte-Carlo simulations (MCS)5 with surrogates. As a result of thedifficulties in optimization and the calculation of failure probability, considering both of them together inreliability-based design optimization (RBDO)12–14 is even more challenging. Therefore, a new adaptivemethodology for performing RBDO is presented in this paper to address the above difficulties of high com-putational cost and discontinuous responses.

In order to address the problem due to discontinuous responses, the method is based on an alternate ap-proach “Explicit Design Space Decomposition” (EDSD).9 Several approaches have been used for EDSD such

∗Graduate Student, Aerospace and Mechanical Engineering Department, The University of Arizona, Tucson, AZ, 85721,USA, and AIAA Student Member.

†Assistant Professor, Aerospace and Mechanical Engineering Department, The University of Arizona, Tucson, AZ, 85721,USA, and AIAA Member.

1 of 14

American Institute of Aeronautics and Astronautics

50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference <br>17th4 - 7 May 2009, Palm Springs, California

AIAA 2009-2189

Copyright © 2009 by Anirban Basudhar and Samy Missoum. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

as the use of hyperplanes, ellipsoids, convex hulls and a pattern recognition technique called SVM.9,15–17 TheSVM-based approach, which is used in this paper, is more general compared to the others and can also defineboundaries that are non-convex or that define disjoint regions in the space. The EDSD approach, insteadof approximating the responses at selected design configurations, involves classifying them into two distinctclasses based on the response values. In the case of discontinuous responses, a clustering technique such asK-means18 is used for the classification. An explicit boundary is then constructed such that it separates thetwo classes.

The use of SVM was first proposed with a focus on problems that had discontinuous responses.17 Sev-eral adaptive schemes were subsequently developed in order to reduce the number of samples that wererequired.19,20 All these schemes focused on updating the SVM-based decision function globally over theentire design space. As a result, even the regions where high accuracy was not needed were refined. Further,the extension of a global update scheme to a high number of dimensions starts to become computationallycostly. As the global update scheme starts to reach its limit in terms of the dimensionality, there is a needfor local update schemes.

Therefore, in this paper a new technique developed to locally refine the SVM decision functions is presented.The methodology is used to locally update the SVM decision boundaries for performing RBDO. In addition,a part of the methodology can be used as a separate scheme for calculating the probability of failure at agiven point. Two and three-dimensional examples demonstrating the application of the update methodologyare presented.

II. Support Vector Machine

SVM is a powerful machine learning technique, which has the ability to define complex decision functionsthat optimally separate two classes of data samples.21,22 The data may be multidimensional and the twoclasses may form several disjoint regions in the space. These features make SVM a very useful tool in prob-abilistic design and optimization.

In order to construct the SVM classifier, a set of N training points xi in a d dimensional space is chosen.Each point is associated with one of two classes characterized by a value yi= ±1. The SVM algorithm findsthe boundary (decision function) that optimally separates the training data into the two classes. For thesake of conciseness, the reader is referred to21,23,24 for the detail of the construction of SVM.

The classification of any arbitrary point x is given by the sign of s defined as:

s = b +N∑

i=1

λiyiK(xi,x) (1)

where b is a scalar referred to as the bias, λi are Lagrange Multipliers obtained from the quadratic pro-gramming optimization problem used to construct the SVM. The training samples for which the LagrangeMultipliers are non-zero are called the (support vectors). Typically, the number of supports vectors (NSV )is much smaller than the total number of samples N . K is referred to as the kernel. In this paper, we willuse a polynomial kernel:

K(xi,x) = (1 + 〈xi,x〉)p (2)

where p is the degree of the polynomial kernel. An example of classification using a polynomial kernel isprovided in Figure 1.

III. Methodology for the identification of locally updated explicit decisionfunctions

The methodology used for the identification of locally refined explicit boundaries with SVM is presentedin this section. The use of SVM for the approximation of decision boundaries was first proposed in order

2 of 14

American Institute of Aeronautics and Astronautics

−4 −3 −2 −1 0 1 2 3 4−4

−3

−2

−1

0

1

2

3

4

x1

x2

margin

support vectors

class −1

class +1

explicit decision function

Figure 1. Two-dimensional polynomial kernel separating the blue and magenta classes. The zero value iso-contour represents the optimal decision function and the support vectors are shown with circles.

to handle problems with discontinuous responses.17 Several update schemes were subsequently derived toconstruct accurate decision functions with reduced number of samples.19,20 All these methods were howeverglobal methods, which updated the decision functions over the entire design space. However, for the purposeof RBDO, high accuracy is required only in certain regions of the space. Therefore, a methodology to con-struct locally updated decision functions is presented in this section. The proposed methodology updatesthe decision functions only at regions that are important in affecting the optimum solution. First, a schemeto accurately calculate the probability of failure at a given point in the space is presented. It is then usedwithin an RBDO process, for which the decision functions are refined locally.

The first step consists of performing an initial DOE4 to sample the design space. In this paper, a DOEgenerated using CVT25 is used. The responses of the system for the DOE samples (design configurations)are then evaluated and classified into two classes of system behaviors (e.g., that correspond to feasible orinfeasible). For details on the selection and response classification of the initial DOE, the reader may refer.19

The classified design configurations are used to train the initial SVM decision function using a polynomialkernel. The degree of the polynomial is automatically selected; the lowest degree (or the simplest SVM)without any training misclassification is selected. The size of the initial DOE is maintained rather small.Therefore, the initial prediction of the decision function may be inaccurate. It is then refined in the impor-tant regions identified by the initial estimate, as explained in the following sections. As new samples getevaluated, the degree of the polynomial is also updated.

1. Application to probability of failure calculation. Update algorithm.

In order to calculate the probability of failure accurately at a given point, the region to be updated orthe “update region” is selected such that it accounts for the probability density functions of the randomvariables. The space to be updated is chosen around the mean (point at which the probability of failure isneeded) upto a few standard deviations. The number of standard deviations is selected based on the order ofthe probability of failure. The initial SVM decision function constructed with the CVT DOE gives the firstestimate of the probability of failure. The steps of the update algorithm to select the subsequent samples,shown in Figure 2, are explained below:

Step 1: New sample on decision function with maximum minimum distance from existing training samples- In the first step of the algorithm, a new sample is selected on the decision function while maximizing thedistance to the nearest training sample (Figure 3). Such a sample lying on the SVM=0 decision function

3 of 14

American Institute of Aeronautics and Astronautics

Initial DOE (CVT) and

response classification

Initial SVM decision

function and probability of

failure calculation (MCS)

Sample on the boundary with

maximum minimum distance to

training samples. SVM update.

Sample near the support vector

with maximum minimum

distance to the opposite class.

SVM update.

Sample on the boundary with

minimum distance to the mean

position (MPP). SVM update.

Stopping

criterion

response

class +1

response

class -1

safe (-1)failure (+1)

initial SVM

boundary

new sample (s)

new sample

updated SVM

boundary

updated SVM

boundary

SV

new sample

updated SVM

boundary

MPP

x1

x2

x1

x2

x1

x2

x1

x2

x1

x2

response

mean point

Probability of

failure calculation

(MCS)

Figure 2. Methodology for the calculation of probability of failure using SVM decision function.

has the highest probability of being misclassified and is bound to change the decision function as no pointcan lie within the margin SVM=±1. The decision function is reconstructed after adding the new sample tothe training set.

The optimization problem is:

maxx

||x− xnearest||

s.t. b +N∑

i=1

λiyiK(xi,x) = 0 (3)

where xnearest is the training sample closest to the new sample. This is a maxmin problem for which theobjective function is non-differentiable. The problem is made differentiable by reformulating it as:

maxx,z

z

s.t. ||x− xi|| ≥ z i = 1, 2, ..., N

b +N∑

i=1

λiyiK(xi,x) = 0 (4)

The problem can be solved by using a global optimizer (a stochastic method (Genetic Algorithm) or a de-terministic tool (Baron26)). However, in our work, the global optimization problem was solved by a local

4 of 14

American Institute of Aeronautics and Astronautics

0 2 4 6 8 100

2

4

6

8

10

0

0

0

0

x1

x2

new sample on decision function

distance to closest sample (maximized)

SVM decision function

0 2 4 6 8 100

2

4

6

8

10

0

0

0 0

x1

x2

new sample

updated SVM decision function

Figure 3. Selection of a new training sample on the decision function while maximizing the distance to theclosest sample. The right figure shows the updated SVM decision function.

optimizer (sequential quadratic programming) starting from N starting points given by the training samples.This approach was chosen because of the difficulties encountered with the global optimizers in solving themaxmin distance problem.

However, the selected sample is evaluated only if the maximum possible change due to its addition leads toa relative probability of failure change of at least δ1 with respect to the previous estimate. For this purpose,the sample is assigned ±1 class label and both the possibilities are analyzed. Further, if the possible relativeprobability of failure change is more than δ1, then another sample is selected around the selected sample.The optimization problem is:

minx

ymm

(b +

N∑

i=1

λiyiK(xi,x)

)

s.t. ||x− xmm|| ≤ R

ymm

(b +

N∑

i=1

λiyiK(xi,x)

)≤ 0 (5)

where xmm is the solution to Equation 4, ymm is the class label for xmm and R is the radius of a hyperspherearound xmm.

R =12||xmm − xopp|| (6)

where xopp is the closest sample belonging to the opposite class. On the contrary, if the possible relativeprobability of failure change is less than δ1, then the sample found with Equation 4 is not evaluated. Insteadtwo samples xoption1 and xoption2 are found with Equation 5, assuming values of +1 and −1 for ymm. Inorder to select the better sample out of of the two, the following criterion is used:

• The training misclassification εt1 upon adding xoption1 to the +1 class is calculated. Similarly, thetraining misclassification εt2 upon adding xoption2 to the −1 class is calculated.

• If εt1 is zero and εt2 is non-zero then the sample xoption1 is evaluated. Similarly, if εt2 is zero and εt1 isnon-zero then the sample xoption2 is evaluated. If both εt1 and εt2 are zero then from among xoption1

and xoption2, the sample which is expected to produce a higher change in the failure probability isevaluated.

• If the selected sample is found to belong to the opposite class after evaluation (i.e. xoption1 is foundto belong to −1 class or xoption2 is found to belong to +1 class) then the alternate sample is alsoevaluated. If this sample is also found to belong to the opposite class after its evaluation then xmm isevaluated.

5 of 14

American Institute of Aeronautics and Astronautics

The SVM decision function is reconstructed after evaluating of the new sample(s).

Step 2: New sample around a support vector with maximum distance to a sample belonging to the oppositeclass - Selection of new samples on the decision function only may not always be sufficient as the samplesnear the decision function might belong to a single class in some regions. For example, the SVM may attimes undergo a small change upon adding a sample, which may not be sufficient to remove the discrepancywith respect to the actual decision function. However, since a sample exists in that region now, a new samplewill not be added using the step 1. An example is shown in Figure 4.

−4 −3 −2 −1 0 1 2 3 4−4

−3

−2

−1

0

1

2

3

4

x1

x2

sample on the decision function while maximizing the distance to nearest sample

−4 −3 −2 −1 0 1 2 3 4−4

−3

−2

−1

0

1

2

3

4

x1x

2

new sample

updated decision function (small change)

Figure 4. Small change in the SVM decision function due to the addition of a new sample on the decisionfunction.

In order to remove this limitation, samples are added slightly away from the decision function in step 2.The motive behind this step is to search for regions around the boundary which are populated by samplesbelonging to a single class. Selection of a sample in such regions is a two step process.

• The support vector farthest from existing samples belonging to the opposite class is located. The newsample must lie within a hypersphere around the selected support vector. Radius of the hypersphereis given by half of the distance to the closest sample belonging to the opposite class.

R =12||x∗sv − xopp|| (7)

where x∗sv is the selected support vector and xopp is the closest sample belonging to the opposite class.

• A new sample is selected within the hypersphere such that it belongs to the opposite class accordingto the current SVM prediction (Figure 5). The optimization problem is given by:

minx

y∗sv

(b +

N∑

i=1

λiyiK(xi,x)

)

s.t. ||x− x∗sv|| ≤ R

y∗sv

(b +

N∑

i=1

λiyiK(xi,x)

)≤ 0 (8)

where y∗sv is the class label (±1) of the selected support vector. It is likely that the above problem will havea feasible solution.

Step 3: New sample at the most probable point (MPP) - A new sample is selected on the decision functionwith smallest distance to the mean position around which the probability of failure is required. However,

6 of 14

American Institute of Aeronautics and Astronautics

−4 −3 −2 −1 0 1 2 3 4−4

−3

−2

−1

0

1

2

3

4

x1

x2

distance to the closest samplebelonging to the opposite class

selected support vectorR

new sample

−4 −3 −2 −1 0 1 2 3 4−4

−3

−2

−1

0

1

2

3

4

x1

x2

updated decision function

new sample

Figure 5. Update of the SVM decision function due to the addition of a new sample based on the step 2 ofthe algorithm.

the sample is evaluated only if the distance changes by at least 1% compared to the previous iteration. Theoptimization problem is:

minx

||x− x0||

s.t. b +N∑

i=1

λiyiK(xi,x) = 0 (9)

where x0 is the point at which probability of failure is calculated.

Step 4: Stopping criterion - Steps 1, 2 and 3 are repeated until a stopping criterion is met. The criterionis based on the probabilities of failure calculated using the SVM decision function with a large number ofMonte-Carlo samples. The update is terminated if the change in the probability of failure is less than acritical value for d + 1 consecutive iterations. The stopping criterion is:

∣∣∣P k+1f − P k

f

∣∣∣P k

f

≤ δ1 (10)

2. Application to RBDO. Update algorithm.

An adaptive sampling methodology for performing RBDO is presented in this section. In general, an RBDOproblem is defined as:

minx

f(x)

s.t. P (g(x) > 0) ≤ Pt (11)

where f(x) is the objective function, P (g(x) > 0) is the probability of failure and Pt is the target failureprobability. A salient feature of the proposed RBDO methodology is the use of two SVM decision functionsinstead of just one. One of the decision functions (Sd) approximates the limit state g (x) = 0 similar toSection 1. The other SVM decision function Sp defines a boundary with the probability of failure equal toPt. While defining the boundary Sd requires function evaluations at the training samples, Sp is trained withprobabilities of failure calculated based on the predicted boundary Sd. The failure probabilities are calcu-lated using MCS. The initial predictions of both Sp and Sd are built using CVT DOEs and the deterministicand probabilistic optima (xd and xp) are obtained using Sd and Sp. The boundaries are then refined basedon the update scheme. The steps of the RBDO methodology are shown in Figure 6 and explained below.The update is performed in two phases.

7 of 14

American Institute of Aeronautics and Astronautics

Initial SVM decision

functions and optima

Evaluation of xd, xp and

the MPP of xp. Update of

the decision function.

Evaluation of a sample on the

boundary with maximum

minimum distance to training

samples. SVM update.

Evaluation of a sample near

the support vector with

maximum minimum distance

to the opposite class.

objective function

increasing

feasible (-1)infeasible (+1)

new sample on

Sd = 0

new sample near

new xd

new update region

update region

updated Sd

updated Sd

updated Sd

x1

x2

x1

x2

x1

x2

x1

x2

xd

xp

initial SVMs

Sd and Sp

MPP

*

svx

Update Pf

calculated at xp

(Figure 2 )

1

1xx

kk

dd

Update xp 1

1xx

kk

pp

updated Sp

updated Sp

updated Sp

No

Yes

No

Figure 6. Methodology for RBDO using SVM decision function.

Phase 1:

• Initial SVM boundaries Sd and Sp: The ranges of the random variables for RBDO are extended upto a few standard deviations outside the side constraints. This enables the calculation of Pf at thedesign space boundaries. The number of standard deviations is selected based on the value of Pt. Theconstruction of the initial estimate of the decision function Sd is similar to the previous section. Thedifference with the failure probability method lies in the construction of Sp. Since Sp is constructedbased on the estimated Sd, it does not require additional expensive function evaluations. Therefore,an additional larger set of CVT samples is used to construct Sp. The probabilities of failure at eachof these samples is calculated using MCS with Sd as the limit-state function. These samples are thenclassified into +1 and −1 classes based on whether the probability is greater or less than Pt. The SVMboundary Sp is then constructed such that it separates these classes. The initial estimate for xd isobtained as:

minx

f(x)

s.t. S0d ≤ 0 (12)

where S0d is the initial SVM prediction of the Sd. Similarly, the initial xp is obtained by replacing S0

d

with S0p .

• Local refinement of the decision function Sd with the addition of new samples: The purpose of thesamples added in phase 1 is to find an accurate xd and to refine the SVM decision functions within

8 of 14

American Institute of Aeronautics and Astronautics

a certain region around it that includes the probabilistic optimum xp. Thus, the samples are selectedsuch that they lie within an “update region” (hypersphere) around the previous estimate of xd. Forthe first iteration, the entire space is considered as the update region. After the first iteration, theupdate region radius is defined as:

Ru = max

(∣∣∣∣xk+1

d − xkd

∣∣∣∣ ,∣∣∣∣xk+1

d − x+1

∣∣∣∣ ,∣∣∣∣xk+1

d − x−1

∣∣∣∣ , 0.5(

V

N

) 1d

,∣∣∣∣xk+1

d − xk+1p

∣∣∣∣)

(13)

where k represents the iteration number. The steps for the update of Sd are given below. d + 1 newsamples are selected within the update region around xd at every iteration. In addition two moresamples to be evaluated are determined based on xp.

– Step 1: Evaluation of the deterministic optimum - The first sample to be added in every iterationis the deterministic optimum xd obtained during the previous iteration. The SVM is updatedafter evaluating the sample.

– Steps 2 and 3: Evaluation of the probabilistic optimum and its MPP - In addition to d+1 samplesselected within the update region based on Sd, two more samples are evaluated - the probabilisticoptimum xp and its MPP (closest point on Sd, Equation 9).

– Step 4: New sample with maximum minimum distance from existing training samples lying onthe decision function Sd within the update region - A new sample is selected within the updateregion by maximizing the distance to the closest training sample such that it lies on the decisionfunction Sd (Equation 14). This is similar to the step 1 used in the scheme for probability of failurecalculation, except for an additional constraint of a spherical update region. The optimizationproblem is:

maxx,z

z

s.t. ||x− xi|| ≥ z i = 1, 2, ..., N∣∣∣∣x− xk+1d

∣∣∣∣ ≤ Ru

b +N∑

i=1

λiyiK(xi,x) = 0 (14)

– Step 5: New sample around a support vector within the update region with maximum distance toa sample belonging to the opposite class - Selection of a sample in this step is carried out in thesame manner as step 2 of the algorithm for probability of failure calculation. The only differenceis the additional spherical update region constraint. The support vector and the new sample tobe selected must lie within the update region.The corresponding optimization problem is:

minx

y∗sv

(b +

N∑

i=1

λiyiK(xi,x)

)

s.t. ||x− x∗sv|| ≤ R∣∣∣∣x− xk+1d

∣∣∣∣ ≤ Ru

y∗sv

(b +

N∑

i=1

λiyiK(xi,x)

)≤ 0 (15)

The radius R is given by:

R =12||x∗sv − xopp|| (16)

For a two-dimensional example, each of the above steps is repeated at every iteration. For eachadditional dimension, one more sample is selected based on step 4.

9 of 14

American Institute of Aeronautics and Astronautics

• Local refinement of the decision function Sp: Since the position of xp is also involved in the selection ofsamples for the update of Sd, it is important to update it at every iteration. Therefore, all the samplesused for updating Sd are added to the training set of Sp also. In addition samples are added withinan update region centered around xp using Equations 14 and 15. It should be noted that assigning aclass to these samples is inexpensive and does not require actual function evaluations. The radius ofthe region is:

Ru = max

(∣∣∣∣xk+1

p − xkp

∣∣∣∣ ,∣∣∣∣xk+1

p − x+1

∣∣∣∣ ,∣∣∣∣xk+1

p − x−1

∣∣∣∣ , 0.5(

V

N

) 1d

,∣∣∣∣xk+1

p − xk+1d

∣∣∣∣)

(17)

• Stopping criterion: At every iteration, the new positions of xd and xp are obtained. The stoppingcriterion is based on the position of xd:

∣∣∣∣xk+1d − xk

d

∣∣∣∣ ≤ ε1 (18)

where ε1 is a small positive number.

Phase 2:

• Local refinement of the decision functions with the addition of new samples: Once the final estimatesof xd and xp are obtained in phase 1, the probability of failure is calculated accurately at xp with theupdate methodology presented in Section 1. This leads to further refinement of the decision functionSd. However, number of standard deviations chosen for the update region is twice of that used for thecalculation of failure probability at a given point. This is done because the probabilities of failure atthe points used to construct Sp also need to be accurate in the proximity of xp.

• Stopping criterion: The refinement of Sd for the accurate calculation of the failure probability at xp

may change the distribution of failure probabilities in the space. As a result, the position of xp mightchange. The stopping criterion for the phase 2 and the entire RBDO methodology is based on theposition of the probabilistic optimum xp.

∣∣∣∣xk+1p − xk

p

∣∣∣∣ ≤ ε1 (19)

IV. Examples

Several test examples demonstrating the ability of the update methodology to locally refine the decisionfunctions are presented. The functions that are to be approximated are known analytical functions, forwhich the actual probability of failure or the optimum solution are already known. The analytical decisionfunctions chosen have two and three variables. The first two examples show the calculation of failure prob-ability at a given point. The third example presents an RBDO problem.

In all the problems, a polynomial kernel is used. The degree of the polynomial is automatically selected asexplained in Section III. For the probability of failure update, 106 MCS samples are generated in order tocalculate Pf with the predicted SVM decision functions during the update. The value of δ1 for the stoppingcriterion is 0.5% of the predicted probability of failure for two-dimensional problems. For three-dimensionalproblems it is 0.1% of the predicted probability of failure. For the RBDO methodology, the value of ε1 is0.5% of the maximum variable range. The final results obtained with the update schemes are compared tothose using the actual decision functions for all the problems.

The following notation will be used in the results section:

• Ninitial is the initial training set size.

• Ntotal is the total number of samples required at the end of the update.

• fopt is the optimum objective function value.

10 of 14

American Institute of Aeronautics and Astronautics

• PSV Mf is the probability of failure using the SVM limit-state function.

• P actualf is the probability of failure using the actual function.

The analytical decision functions are written in the form g(x) = 0. In order to perform the SVM classifica-tion, the samples corresponding to g(x) > 0 and g(x) < 0 are labeled +1 and −1 respectively. The +1 classrepresents failure region and the −1 class is the safe region.

A. Probability of failure - example 1

The actual decision function in this example is a function of two random variables x1 and x2. Both thevariables have a standard normal distribution. The range of the variables is taken as four standard deviations([−4, 4]). The equation of the limit-state function is:

g(x1, x2) = −(5x1 + 10)3 − (5x2 + 9.9)3 + 18 (20)

Figure 7 shows the updated SVM limit-state function and the actual function. A total of 77 samples arerequired, starting from 10. The final predicted Pf is compared to the actual Pf calculated with the analyticaldecision function. 107 MCS samples are used for calculating the probabilities. The predicted and actualprobabilities of failure are 0.0059 and 0.0057. Thus, there is an error of 3.74%.

−4 −3 −2 −1 0 1 2 3 4−4

−3

−2

−1

0

1

2

3

4

x1

x2

Figure 7. Probability of failure calculation. Two-dimensional non-convex example. The green and red curvesare the SVM and the actual limit-state functions. The red diamond is the point at which probability iscalculated.

B. Probability of failure - example 2

This example presents the probability of failure calculation with a limit-state function with three randomvariables x1, x2 and x3. All the variables have a standard normal distribution. The range of the variables istaken as [−4, 4]. The equation of the limit-state function is:

g(x1, x2, x3) = −(5x1 + 10)3 − (5x2 + 9.9)3 − (5x3 + 5)3 + 18 (21)

The updated SVM limit-state function and the actual function are shown in Figure 8. The initial DOEconsists of 20 samples and the final limit-state function is constructed with 201 samples. The predicted

11 of 14

American Institute of Aeronautics and Astronautics

and actual probabilities of failure are 0.0040 and 0.0039, which corresponds to an error of 1.62%. For thisproblem, the comparison is made with 5× 106 MCS samples due to limited computer memory.

Figure 8. Probability of failure calculation. Three-dimensional non-convex example. The green and redsurfaces are the SVM and the actual limit-state functions. The red diamond is the point at which probabilityis calculated.

C. RBDO - example 3

The actual decision function in this example is a function of two random variables x1 and x2. The optimiza-tion problem is:

minx

2x1 + x2

s.t. Pf = P (g(x1, x2) > 0) ≤ Pt = 0.001 (22)

where the limit-state function g(x1, x2) consists of two equations given as:

g1(x1, x2) = x1 − 2x2 − 2 ≤ 0

g2(x1, x2) = x21 − 6x1 − x2 + 8 ≤ 0 (23)

The range of both design variables is [0, 10] and both the variables have a normal distribution with standarddeviation of 0.2. The range of the space for the update is taken as [−0.8, 10.8] in order to facilitate thecalculation of failure probabilities at the design space boundaries. Figure 9 shows the decision function andthe final optimum solution. The initial DOE for Sd consists of 10 CVT samples. The training set for Sp

consists of additional 40 CVT samples. The results obtained after the end of phase 1 and phase 2 are listedin Table 1.

Update stage Ninitial Ntotal xp fopt PSV Mf P actual

f

phase 1 10 105 (2.285, 0.901) 5.47 10−3 5.6× 10−3

phase 2 105 141 (2.200, 1.092) 5.49 10−3 9.5× 10−4

Table 1. Results for example 3. Optimum solutions obtained after phase 1 and phase 2 of the algorithm.

12 of 14

American Institute of Aeronautics and Astronautics

0 2 4 6 8 10

0

2

4

6

8

10

x1

x2

xp

f(x) decreasing

1 1.5 2 2.5 3 3.5−0.5

0

0.5

1

1.5

2

2.5

x1

x2

xp

Figure 9. Two-dimensional RBDO example. The green and red curves represent the SVM Sd and the actualdecision function. The black curve is the SVM decision function Sd. The red diamond shows the final solutionxp. The left figure shows the entire design space and the right figure shows the magnified region around xp,which is the final update region for the phase 2 of the algorithm.

V. Conclusion

A methodology for failure probability calculation and RBDO has been developed and successfully applied totwo and three-dimensional decision functions. The decision boundaries for RBDO are updated locally, whichavoids function evaluations in the unimportant regions of the design space. The probability of failure updateis successfully implemented for accurately predicting the probabilities of failure of the order 10−3. The RBDOscheme, which also uses the failure probability update scheme, is able to predict the probabilistic optimumaccurately with a relatively small number of function evaluations. The application of the methodology isdemonstrated for a target probability of the order 10−3. As the methodology is based on the EDSD approach,it has the advantage of being applicable to problems with discontinuous responses. In future, the efficiencyof the update will also be compared to other existing methods for continuous responses.

VI. Acknowledgement

The support of the National Science Foundation (award CMMI-0800117) is gratefully acknowledged.

References

1Myers, R.H. and Montgomery, D.C. Response Surface Methodology. Wiley, 2nd edition, 2002.2G. Gary Wang and S. Shan. Review of metamodeling techniques in support of engineering design optimization. Journal of

Mechanical Design, 129(4):370–380, 2007.3Montgomery, D.C. Design and Analysis of Experiments. Wiley and Sons, 2005.4Giunta, A.A., Wojtkiewicz, S.F., and Eldred, M.S. Overview of modern design of experiments methods for computational

simulations. In 41st AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV, 2003.5Melchers, R. Structural Reliability Analysis and Prediction. John Wiley & Sons, 1999.6Huang, D., Allen, T.T., Notz, W.I., and Miller, R.A. Sequential kriging optimization using multiple-fidelity evaluations.

Structural and Multidisciplinary Optimization, 32:369–382, 2006.7Bichon, B.J., Eldred, M.S., Swiler, L.P., Mahadevan, S., and McFarland, J.M. Multimodal reliability assess-

ment for complex engineering applications using efficient global optimization. In Proceedings of the 48th conferenceAIAA/ASME/ASCE/AHS/ASC on Structures, Dynamics and Materials. Paper AIAA-2007-1946., Honolulu, Hawaii, April2007.8Wang, G.G., Wang, L., and Shan, S. Reliability assessment using discriminative sampling and metamodeling. SAE Trans-

actions, Journal of Passenger Cars - Mechanical Systems, 114:291–300, 2005.9Missoum, S., Benchaabane, S., and Sudret, B. Handling birfurcations for the optimal design of transient dynamics problems.

In Proceedings of the 45th conference AIAA/ASME/ASCE/AHS/ASC on Structures, Dynamics and Materials, Palm Springs,California, USA, April 2004.

13 of 14

American Institute of Aeronautics and Astronautics

10Haldar, A. and Mahadevan, S. Probability, Reliability, and Statistical Methods in Engineering Design. Wiley and Sons,New-York, 2000.11Youn, B.D., Choi, K.K., and Du, L. Adaptive probability analysis using an enhanced hybrid mean value method. Structuraland Multidisciplinary Optimization, 29(2):134–148, 2005.12Youn, B.D. and Choi, K.K. Selecting probabilistic approaches for reliability-based design optimization. AIAA Journal,42(1):124–131, January 2004.13Kharmanda, G., Mohamed A., and Lemaire M. Efficient reliability-based design optimization using a hybrid space withapplication to finite element analysis. Structural and Multidisciplinary Optimization, 24:233–245, 2002.14Youn, B. D., Choi K.K., Yang, R.J., and Gu,L. Reliability-based design optimization for crashworthiness of vehicle sideimpact. Structural and Multidisciplinary Optimization, 26:272–283, 2004.15Hurtado, Jorge E. An examination of methods for approximating implicit limit state functions from the viewpoint of statisticallearning theory. Structural Safety, 26:271–293, July 2004.16Missoum, S., Ramu, P., and Haftka, R.T. A convex hull approach for the reliability-based design of nonlinear transientdynamic problems. Computer Methods in Applied Mechanics and Engineering, 196(29):2895–2906, 2007.17A. Basudhar, S. Missoum, and A. Harrison Sanchez. Limit state function identification using Support Vector Machines fordiscontinuous responses and disjoint failure domains. Probabilistic Engineering Mechanics, 23(1):1–11, 2007.18Hartigan, J.A. and Wong, M.A. A k-means clustering algorithm. Applied Statistics, 28:100–108, 1979.19Basudhar, A. and Missoum, S. Adaptive explicit decision functions for probabilistic design and optimization using supportvector machines. Computers & Structures, 86(19–20):1904–1917, 2008.20Basudhar, A. and Missoum, S. Parallel update of failure domain boundaries constructed using support vector machines. In7th World Congress on Structural and Multidisciplinary Optimization, Seoul, Korea, May 2007.21Shawe-Taylor, J. and Cristianini, N. Kernel Methods For Pattern Analysis. Cambridge University Press, 2004.22Tou, J.T. and Gonzalez, R.C. Pattern Recognition Principles. Addison-Wesley, 1974.23Vapnik, V.N. Statistical Learning Theory. John Wiley & Sons, 1998.24Gunn, S.R. Support vector machines for classification and regression. Technical Report ISIS-1-98, Department of Electronicsand Computer Science, University of Southampton, 1998.25Romero, Vincente J., Burkardt, John V., Gunzburger, Max D., and Peterson, Janet S. Comparison of pure and “latinized”centroidal voronoi tesselation against various other statistical sampling methods. Journal of Reliability Engineering and SystemSafety, 91(1266–1280), 2006.26Tawarmalani, M. and Sahinidis, N.V. Global optimization of mixed-integer nonlinear programs: A theoretical and computa-tional study. Mathematical Programming, 99:563–591, 2004.

14 of 14

American Institute of Aeronautics and Astronautics