A Genetic Algorithm Hybrid for Constructing Optimal Response Surface Designs

QUALITY AND RELIABILITY ENGINEERING INTERNATIONAL

Qual. Reliab. Engng. Int. 2004; 20:637–650

Published online 10 June 2004 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/qre.573

Research A Genetic Algorithm Hybrid forConstructing Optimal ResponseSurface DesignsDavid Drain1,∗,†, W. Matthew Carlyle2, Douglas C. Montgomery3, Connie Borror4 andChristine Anderson-Cook5

1Department of Mathematics and Statistics, University of Missouri, Rolla, 1870 Miner Circle, Rolla, MO 65409-0020, U.S.A.2Operations Research Department, Naval Postgraduate School, 1411 Cunningham Road, Monterey, CA 94943-5219, U.S.A.3Industrial Engineering Department, Arizona State University, P.O. Box 875906, Tempe, AZ 85287-5906, U.S.A.4LeBow College of Business, Drexel University, Philadelphia, PA 19104, U.S.A.5Department of Statistics, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061-0439, U.S.A.

Hybrid heuristic optimization methods can discover efficient experiment designsin situations where traditional designs cannot be applied, exchange methods areineffective, and simple heuristics like simulated annealing fail to find good solutions.One such heuristic hybrid is GASA (genetic algorithm–simulated annealing),developed to take advantage of the exploratory power of the genetic algorithm,while utilizing the local optimum exploitive properties of simulated annealing.The successful application of this method is demonstrated in a difficult designproblem with multiple optimization criteria in an irregularly shaped designregion. Copyright c© 2004 John Wiley & Sons, Ltd.

KEY WORDS: design of experiments; genetic algorithm; heuristic optimization

INTRODUCTION

Experiment design is an activity that seeks to select experiments that are optimal in some sense.Some experiments are designed to enhance parameter estimation (D-optimality), some to improveprediction over a region of interest (G-optimality). See Myers and Montgomery1 for explanations of

these and other ‘alphabetically optimal’ experimentation goals. Actual experiment goals are often too complexto express in terms of one of these single-number optimality criteria. Experiments may have to be performedunder unfavorable conditions, control variable settings may be confined to irregularly shaped regions becauseof equipment limitations, restrictions on randomization may require blocking, and the need for experimentalcontrols may force the use of center points. Experiments often must balance several competing objectives—prediction ability and slope estimation, for example. Cost considerations also drive experiments to be as smallas possible. All these requirements generate difficult and complex design problems for which there is nodeterministic path to a solution.

Heuristic optimization has been used in design of experiments to solve a variety of problems. Welch2 usedbranch-and-bound search to find D-optimal designs, Haines3 used simulated annealing to find exact optimaldesigns for linear regression models, and numerous authors4–7 have used the genetic algorithm to find optimal

∗Correspondence to: David Drain, Department of Mathematics and Statistics, University of Missouri, Rolla, 1870 Miner Circle, Rolla,MO 65409-0020, U.S.A.†E-mail: [email protected]

Copyright c© 2004 John Wiley & Sons, Ltd.Received 7 January 2003

Revised 25 April 2003

638 D. DRAIN ET AL.

or near-optimal designs. Researchers in other fields have recognized that hybrid heuristics—combinations ofsimple heuristics into more complex algorithms—can solve some problems more effectively than individualheuristic algorithms (Li et al.8 and Kragelund9, for example), but applications to date in design of experimentshave not taken full advantage of this research. The discussion following is intended to demonstrate theimportance of hybrid heuristics in design of experiments, and also to give some guidance on their use.The emphasis is on effective application of heuristics to solve practical problems in design of experiments, andnot on evaluation of the heuristic techniques themselves. Examples were done on a popular computing platformwith readily available software‡; neither exotic software nor parallel processing supercomputers were required.

CLASSICAL OPTIMIZATION METHODS IN DESIGN OF EXPERIMENTS

Consider a typical experiment design problem.

• An experiment involving two design variables (x1 and x2) is to be conducted.• A full quadratic model including interaction is assumed.• A total of six experimental runs will be done.• A D-optimal experiment is desired.

So in this example, the model is

y = β0 + β1x1 + β2x2 + β12x1x2 + β11x21 + β22x

22 + ε

where ε represents an independent and identically normally distributed error term.The design matrix expanded to model form X(m) is found by computing a column for each non-random term

of the model in terms of the experimental runs:

X(m) =

1 x11 x21 x211 x2

21 x11x21

1 x12 x22 x212 x2

22 x12x22

1 x13 x23 x213 x2

23 x13x23

1 x14 x24 x214 x2

24 x14x24

1 x15 x25 x215 x2

25 x15x25

1 x16 x26 x216 x2

26 x16x26

and the desired experiment maximizes |X(m)′X(m)| among all experiments meeting the conditions of this designproblem.

A candidate experiment is one that meets the basic requirements of the design problem, namely it has theability to estimate all of the model parameters. However, it may not be optimal. For example,

D =

1 11 −1

−1 1−1 −1

0.033 −0.6670.667 0.033

is a candidate experiment for this problem.

‡The computing platform was a Dell� Dimension 4100 running Microsoft� Windows ME on a 1 GHz Intel� Pentium III.Most programming was done in Microsoft� Visual Basic for Applications 6.0 from within Microsoft� Excel 2000 workbooks.Some additional tasks were accomplished using SAS� Release 8.01, Minitab� Release 14 beta 2, and Mathcad� 2001 Professional.

Copyright c© 2004 John Wiley & Sons, Ltd. Qual. Reliab. Engng. Int. 2004; 20:637–650

A GENETIC ALGORITHM HYBRID 639

Some of these types of problems can be solved directly and definitively, either with matrix algebra orcalculus, but this problem is not amenable to such solutions. Exchange algorithms have had some successwith optimization problems such as this one. DETMAX and other such algorithms work by selecting a setof runs from a large candidate set; see Cook and Nachtsheim10 for a review of these algorithms. Exchangealgorithms are not guaranteed to find a truly optimal experiment, although they will generally find the bestexperiment from among the set of candidate runs. If the truly optimal set of runs is not among the candidate set,an exchange algorithm cannot find the optimal experiment. For this problem, repeated attempts were made withDETMAX to find the optimal experiment from among 300 000 random candidate runs, and the optimum wasnever achieved.

HEURISTIC OPTIMIZATION METHODS

Heuristic optimization methods have emerged as an effective alternative to classical methods, largely as theresult of the recent advances in computing power. Heuristic methods are computationally intensive algorithms,often based on some physical or natural model for effective problem solving. See Mitchell11 for generaldescriptions of many heuristic algorithms. Heuristic algorithms do not guarantee that a true optimum is found,but they often find very good solutions when no other solution method can be applied. Some heuristic algorithmswith applications in design of experiments are described below.

Complete enumeration is one of the simplest heuristics. It can only be applied when the population of possiblesolutions is finite because it consists of finding every possible solution and evaluating it, then selecting the bestsolution found. Complete enumeration is preferable to other heuristics when the population of possible solutionsis relatively small because it guarantees that the solution found is indeed the optimum.

Scatter search consists of the repeated generation and evaluation of random solutions. It is often employed tofind a starting population for other heuristic methods.

Hillclimbing refers to a variety of algorithms that seek to improve from a starting point, either by examiningnearby points, or by using calculus to find a direction likely to lead to improvement. Hillclimbing techniquesare likely to find only local (rather than global) optima, so they are often repeated from many different startingpoints.

Simulated annealing is based on physical annealing processes used in metallurgy and glass making. In thosephysical processes, movement or rearrangement of molecules within the material is related to temperature—higher temperature means more movement. By starting at a high temperature, then slowly cooling to the eventualtemperature for use, the materials achieve a higher degree of some desirable property like strength or shock-resistance than they would have if cooled quickly. Simulated annealing is similar to hillclimbing, except thatinferior solutions are accepted with a probability related to the ‘temperature’ at a particular time. This meansthat simulated annealing is somewhat less likely to become trapped near local optima because it can move to lessoptimal solutions early in the algorithm’s progress, when the temperature is high. A typical simulated annealingalgorithm for minimization of an objective function, O(solution), works as shown in Figure 1. Starting at agiven design, the algorithm tries a series of candidate designs in a neighborhood of nearby designs, where theneighborhood might be determined by the Euclidean distance between the designs. Designs with a superior(smaller) objective function are always selected, but inferior designs might also be selected with a probabilityinversely related to temperature. The greater the temperature, the greater the probability that an inferior solutionwill be selected. Temperature is decreased according to some user-determined rule during the course of thealgorithm. Note that the user has many choices with this algorithm: the starting and ending temperatures, thetemperature reduction regime, the number of annealing steps, and the definition of ‘neighborhood’.

The genetic algorithm is modeled on the process of biological evolution. See Figure 2 for a descriptionof a simple form of the algorithm. The genetic algorithm starts with a population of candidate designs, thenuses some rule to decide which designs will breed and produce the next generation of designs. The rulecan be based on chance (roulette breeding), or on merit (performance with regard to experiment objectives),or on other criteria. Breeding pairs produce offspring according to two successive processes: crossover andmutation.


640 D. DRAIN ET AL.

Read starting solutionSet starting temperature, TFor I = 1 to N (number of annealing steps)

Randomly select a candidate solution in a neighborhood of the current solutionIf that candidate solution is better than the current solution

Then set the current solution to this candidate solutionElse compute probability of acceptance of the candidate as follows:

PAccept = e−[O(Candidate)−O(Current)]/T

If a random number from [0, 1] is less than this probability of acceptanceThen set the current solution to this candidate solutionElse the current solution is unchanged

Decrease temperature according to scheduleReport current solution

Figure 1. Simulated annealing

Read starting populationFor I = 1 to N (maximum number of generations)

Select breeding population on the basis of chance and meritWhile new population is less than desired population size

Randomly select a breeding pair from the breeding populationCrossover and mutate to create two new population members

Save final population

Figure 2. The genetic algorithm

Through crossover, offspring receive some runs from one parent experiment, and some from the other.For example, if the crossover rule were to split after the third row, the parents

1 −11 1

−1 0.33−1 −0.33

0.50 10.75 −1

0.33 1−0.33 −1

1 10.50 −0.501 −1

−1 1

would produce the offspring

1 −11 1

−1 0.330.50 −0.501 −1

−1 1

0.33 1−0.33 −1

1 1−1 −0.33

0.50 10.75 −1

Many different forms of mutation have been used with the genetic algorithm. One form of mutation is to selecta random run from a random member of the population to replace an existing run in the present experiment.Heredia-Langner et al.12 use Gaussian perturbation of runs randomly selected for mutation, and Borkowski5

uses a linear combination of two runs called ‘blending’.



Table I. GASA parameters

Parameter name Example value

Candidate sheet K2Evaluation sheet KEval2Output sheet K4Summary sheet KS4Winner sheet KW4Input vectors Vec01Output vectors

GASA iterations 36Maximum population size 500GA initial roulette rate 0.25GA final roulette rate 0.02GA initial mutation rate 0.04GA final mutation rate 0.005GA elite reservation 30SA iterations 6SA initial temperature at first GA 13SA initial temperature at last GA 26SA final temperature 50Starting SA T2 (radius) 0.1Ending SA T2 (radius) 0.01Starting SA T3 (probability) 0.07Ending SA T3 (probability) 0.0054CrossBreed proportion 0.75CrossBreed search distance 20

HYBRID HEURISTIC OPTIMIZATION

Hybrid heuristic optimization algorithms combine features of two or more simple heuristics to create morepowerful algorithms. Hybrids are a natural extension of simple heuristics, and in fact, simple forms of them arealready used in design of experiments. For example, using random starting points as the starting population for agenetic algorithm is a hybrid: scatter search followed by the genetic algorithm. See Talbi13 for a comprehensivetaxonomy of hybrid heuristics.

One particularly useful hybrid for design of experiments is the genetic algorithm–simulated annealing hybrid(GASA). The genetic algorithm has the ability to explore a space of possible solutions without being trapped inlocal optima, but it does not efficiently exploit good solutions it finds. Simulated annealing exploits good startingpoints very effectively, but if initiated from a poor starting point, it will become trapped at a local optimumrather than finding the global optimum. GASA combines the exploration ability of the genetic algorithm withthe exploitation ability of simulated annealing. Simulated annealing also allows less precise evaluation of theobjective function at higher temperatures (during the early iterations of the algorithm), and this same rulecan be applied to the genetic algorithm part of GASA for objective functions that are particularly difficultto evaluate—average prediction variance over the region of experimentation, for example. This provides GASAwith a significant computational efficiency advantage when compared with a simple genetic algorithm.

Figure 3 is a description of an implementation of GASA that was written for evaluating prediction errorvariance and slope estimation variance for designs with noise variables. Table I lists some of the user-adjustableparameters that guide the algorithm’s operations, and gives an example of their actual values. More details aboutthe meaning of individual algorithm parameters are given in subsequent paragraphs.

The algorithm starts by reading all the parameters, many of which have both starting and ending values.Each of these iteration-dependent parameters starts at the initial value, and is changed in steps until the finalvalue is reached at the last iteration. For example, the initial roulette rate is 0.25, meaning that 25% of thebreeding population is selected purely by chance during the first iteration of the algorithm. The final roulette


642 D. DRAIN ET AL.

Read parameters from control sheetDefine rules for parameter modification by iterationInitialize population with candidate listCheck candidates to assure they meet experiment constraintsRead predetermined test vectors if presentGenerate random test vectors if necessaryEvaluate population member fitness by evaluating performance at the current set of test vectorsFor the chosen number of iterations

Adjust GASA parameters toward terminal valuesSelect a predetermined number of elitesSelect an (iteration-dependent) number of breeders on the basis of fitnessSelect an (iteration-dependent) number of breeders by chanceWhile new population is less than desired population size

If cross-breeding is required, thenRandomly select a breederRandomly select a number of potential co-breedersSelect the most different potential co-breeder (greatest Euclidean distance) for breedingElse randomly select a pair for breeding (self-breeding is not allowed)

Produce two offspringCrossover at the design midpointMutate (at the current mutation rate) by replacing a random run in an offspring witha random run from a population member

Simulated annealing sub-algorithmFor each (new) population member except elites

For the given number of SA iterationsCalculate temperature parameters for this iterationEvaluate design fitness at the (SA iteration dependent) set of test vectorsPerturb the present design (within temperature constraints)Evaluate the perturbed designIf perturbed design is an improvement,

Then make it the current designElse select the perturbed design with a (temperature-influenced) small probability

Destroy all old population members except elitesReport results

Figure 3. The GASA algorithm

rate is 0.02, and this will be used on the final (36th) iteration. From the second iteration to the last, the rouletterate is multiplied by a constant that assures the final roulette rate will be realized on the 36th iteration.

The algorithm then proceeds to input a set of candidate designs—usually at least a hundred. Candidatesare checked to assure they meet any constraints on variable magnitude, and rescaled if necessary. A set of 13particularly challenging (likely to have high variances) test vectors was specified for this problem on the Vec01sheet as shown in Table I. These are supplemented by random test vectors to get the total of 50 vectors at whichdesigns will eventually be evaluated. Population member fitness is evaluated, and the algorithm is ready to beginits first iteration.

During each iteration, some number of elite members (30 in this case) will be selected purely on the basis offitness to live to the next generation. Elites are not guaranteed to be in the breeding population for any givengeneration, but their high fitness does make this likely. The breeding population is selected on the basis of fitnessand chance, as directed by the current value of the roulette parameter. Breeding pairs can either be chosen at



random, on the basis of genetic diversity, or some combination of the two. In this example, 75% of breedingswill require that 20 potential co-breeders are examined for their difference from the chosen breeder, and thenthe most different co-breeder will be selected. Nearness is defined by elementwise Euclidian distance betweendesigns. For example, the distance between the following two designs

[−1 01 1

] [1 1

−1 0.5

]

is given by

3.0414 =√

(−1 − 1)2 + (0 − 1)2 + (1 − (−1))2 + (1 − 0.5)2

Each breeding pair produces two offspring through crossover and mutation. Crossover is at the midpoint of thedesign. The mutation rate depends on the iteration, and it determines the percent of experiment runs (a part of apopulation member) that are mutated. Mutation consists of substituting a random run from a random populationmember for one run of an offspring. Breeding proceeds until the desired population size is achieved.

Every member of the new population is now subjected to modification by simulated annealing.Three ‘temperature’ parameters are specified for this part of the algorithm, in addition to a fixed numberof simulated annealing iterations. The first temperature specifies the number of test vectors at which eachpopulation member is evaluated at each iteration of simulated annealing. The ‘SA initial temperature at firstGA’ gives the starting temperature for the SA at the first GASA iteration, and the ‘SA initial temperature atlast GA’ gives the starting temperature for the SA at the last GASA iteration. The final temperature for theSA is the same regardless of GASA iteration. The second temperature parameters control the definition of‘neighborhood’ for the SA, which is stated in terms of elementwise Euclidian distance as for crossbreeding.The third SA temperature parameter defines the probability of acceptance in the event that a candidate designhas a worse objective function than the original design. This is defined according to the standard SA definitionof temperature, T:

PAccept = e−[O(Candidate)−O(Current)]/T

Once every new population member has been through the simulated annealing process, the old population isallowed to die, and GASA proceeds to the next generation.

Output from the program consists of the entire population at the last generation, a summary sheet givingonly the objective function value for each design at every iteration, and the winning design from each iteration.Execution time, and starting and ending best fitness are reported on the control sheet.

EXAMPLE 1–ALGORITHM VALIDATION

The algorithm was validated for a problem with a known solution from Borkowski5.

• Two control variables are to be varied in the range [−1, 1].• A full quadratic model with interaction is assumed.• Six experiment runs are allowed.• A D-optimal design is desired.

The optimal design shown in Table II is known to achieve a determinant of 267.7372.Fifteen starting designs were chosen using the D-optimal design capability of Minitab14, each from among

100 000 uniform random variates on [−1, 1]. This candidate set was supplemented with 110 random candidates.The GASA algorithm was run for 60 generations with the parameters shown in the ‘step 1’ column of Table III,during which time the greatest determinant in the population increased from 251.6 to 255.8. The resultingpopulation was cleaned of clones (unintentional duplicated designs), then winning designs from all the runs andthe original D-optimal designs from Minitab were taken as the starting point for the next step. In step 2, GASA


644 D. DRAIN ET AL.

Table II. D-Optimal design fortwo variables in six runs

X1 X2

1 1−1 1−1 −1

1 −0.394 4490.394 449 −1

−0.131 483 0.131 483

Table III. GASA parameters for Example 1

Parameter Step 1 Step 2 Step 3

Input vectors Vec01 Vec01 Vec01GASA iterations 60 220 220Maximum population size 500 500 500GA initial roulette rate 0.25 0.1 0.1GA final roulette rate 0.05 0.02 0.02GA initial mutation rate 0.35 0.01 0.01GA final mutation rate 0.1 0.005 0.005GA elite reservation 20 30 30SA iterations 12 6 6SA initial temperature at first GA 2 2 2SA initial temperature at last GA 2 2 2SA final temperature 2 2 2Starting SA T2 (radius) 0.1 0.03 0.03Ending SA T2 (radius) 0.005 0.0008 0.0008Starting SA T3 (probability) 18.6 18.6 0.62Ending SA T3 (probability) 0.27 0.27 0.03CrossBreed proportion 0.75 0.75 0.75CrossBreed search distance 10 20 20Starting best fitness −251.6063 −255.8312 −264.3928Ending best fitness −255.8312 −266.4706 −267.7372

Table IV. D-Optimal designproduced using the GASA

algorithm

X1 X2

−1.000 000 −1.000 0001.000 000 1.000 000

−1.000 000 1.000 0000.394 266 −1.000 0001.000 000 −0.394 536

−0.131 616 0.131 481

ran for 220 generations, and increased the best determinant from 255.8 to 266.47. The resulting population wascleaned of clones, then added to 15 D-optimal designs from Minitab (each based on 300 000 random variates)and 98 random designs to form the starting population for step 3. In step 3, the published optimal determinatewas achieved in the 194th generation. The resulting design shown in Table IV differs insignificantly from thepublished optimal design.



EXAMPLE 2–A MORE CHALLENGING DESIGN PROBLEM

This is a problem in robust parameter design involving both control variables and noise variables.Control variables are manipulated during routine manufacturing so they take values very close to their targets.Noise variables can be deliberately varied only during an experiment; at all other times they are assumed tovary according to a known distribution that may involve correlations among noise variables. Low predictionerror variance is one goal of the experiment (similar to G-Optimality), and slope estimation variance (varianceof estimates of control–noise variable interaction terms) is another. Good estimation of slope is important inprocess robustness studies because process targets should be located not only where mean performance is good,but also where interactions between control and noise factors are small. Details of the problem follow.

• Two control variables and two noise variables are involved.• Control variables are bound by the following set of constraints:

−1 ≤ x1 ≤ 1 x1 + x2 ≤ 1.5−1 ≤ x2 ≤ 1 −1 ≤ x1 + x2

• Noise variables z1 and z2 are confined within a circle of radius 1.00.• The model is quadratic in the control variables, first-order linear in the noise variables, and includes

control–noise variable interactions:

y(x, z) = β0 + x′β + x′Bx + z′γ + x′�z + ε

where β0 is a constant, β is a 2 × 1 vector of coefficients representing the effect of each control variable,and B is a 2 × 2 matrix of coefficients for squared and interaction terms in the control variables.The 2 × 1 vector γ quantifies the first-order effect of the noise variables, and the 2 × 2 matrix �

quantifies interactions between noise and control variables. The error term ε is assumed to be identicallyand independently normally distributed with mean zero and standard deviation σ . See Myers andMontgomery1 or Myers et al.15 for further details regarding this model.During the experiment, the noise variables are assigned fixed values, but during routine manufacturingoperations the noise variables are elements of random vectors identically and independently normallydistributed with zero mean and covariance matrix VZ . The following parameter values are to be assumedin evaluations:

VZ =[

1 −0.55−0.55 1

]γ =

[0.125

−0.495

]� =

[0.0 0.175

−0.75 0.125

]

• Both prediction error variance and slope estimation variance are of interest.

Prediction error variance (vm) and slope estimation variance (vs) will be computed according to formulae inBorror et al.16; details of that computation are not given here. A (smaller is better) desirability function is usedto balance the competing needs to reduce each of these variances:

d(vm, vs) = 1 − √f (vm)g(vs)

where

f (vm) =

(vm − 120

120

)2

for vm ≤ 120

0 for vm > 120

and

g(vs ) =

(vs − 30

30

)2

for vs ≤ 30

0 for vs > 30


646 D. DRAIN ET AL.

Table V. Benchmark design for design problem 2

X1 X2 Z1 Z2

1 −1 0.8391 −0.5441−1 1 0.8433 −0.5375

1 −1 −0.8416 0.5402−1 1 −0.8405 0.5419

1 0.5 0.5422 0.84030 −1 0.5392 0.84220 −1 −0.542 −0.8404

−1 0.5 −0.5442 −0.8391 0.5 −0.9609 0.27691 0.5 0.3478 −0.93760.0679 0.0679 0.5304 0.8478

−0.25 1 −0.594 −0.8045

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

Figure 4. GASA progress in Example 2

A benchmark design shown in Table V was chosen using the D-optimal design feature in Minitab from acandidate set of 10 000 runs consisting of vertices, edge midpoints, and the area centroid in the control variables,and random variates exactly meeting the radius constraint in the noise variables.

This design achieved a median prediction error variance of 25.63, and 90th percentile of 31.89. The medianslope estimation variance was 5.04, and the 90th percentile was 8.41.

The GASA algorithm was initiated from a candidate population of two designs similar to the benchmarkdesign supplemented by 200 random designs. Table VI shows the parameters used for this and following steps.The best desirability function in the population improved from 0.6326 to 0.3145 after 50 generations usinga population of 200 designs. The population diversity was then enhanced by eliminating some unintentionalclones, and adding 300 random designs in addition to the original benchmark design. After an additional 50generations with a population size of 500, the best desirability function in the population was improved from0.3217 to 0.2821. The progress of the algorithm is apparent in Figure 4, where the minimum desirability functionby generation is the lower line, and the average for the population is the upper line. The x-axis on this graphis the generation number. Note the spike at the 51st generation when new (and clearly inferior) populationmembers were introduced.

The best design found is shown in Table VII. This design achieved a median prediction error variance of 22.09,and 90th percentile of 31.82. This represents a 14% reduction in median prediction error variance. Figure 5 is afraction of design space graph (FDS) showing the relative performance of the benchmark design (solid curve)



Table VI. GASA parameter settings for Example 2

Parameter Step1 Step2

Input vectors Vec01 Vec01GASA iterations 50 50Maximum population size 200 500GA initial roulette rate 0.25 0.25GA final roulette rate 0.02 0.02GA initial mutation rate 0.04 0.04GA final mutation rate 0.005 0.005GA elite reservation 30 30SA iterations 9 9SA initial temperature at first GA 13 13SA initial temperature at last GA 26 26SA final temperature 50 50Starting SA T2 (radius) 0.1 0.1Ending SA T2 (radius) 0.01 0.01Starting SA T3 (probability) 0.07 0.07Ending SA T3 (probability) 0.0054 0.0054CrossBreed proportion 0.75 0.75CrossBreed search distance 20 20Starting best fitness 0.5601 0.3352Ending best fitness 0.3292 0.2930

Table VII. Winning design for design problem 2

X1 X2 Z1 Z2

−0.2221 −0.7504 −0.5520 −0.80480.8033 −0.0831 −0.8673 −0.4086

−0.1455 0.0036 −0.8740 −0.14080.7890 −0.9914 −0.5053 −0.6250

−0.9762 0.8692 −0.6313 −0.7492−0.9257 0.5072 0.7118 0.6841

0.2013 −0.6931 0.8748 0.38300.7102 0.7473 −0.6410 −0.74960.9741 0.3816 0.7378 0.6733

−0.0246 0.3288 0.5722 −0.79340.0920 0.8781 0.3490 0.47980.2808 −0.9561 0.2979 0.9155

and the design found by GASA (dashed curve). The x-axis of the FDS is the fraction of the design space overwhich the predication error variance is less than or equal to the corresponding value on the y-axis. See Zahranet al.17 for a more complete description of the FDS. The design’s median slope estimation variance was 3.42, andthe 90th percentile was 5.12. This represents a 32% improvement in median slope estimation variance. Figure 6is the FDS for slope estimation variance comparing the two designs, with the benchmark design performanceshown as a solid curve, and the GASA-produced design shown with a dashed curve.

MAKING EFFECTIVE USE OF HYBRID HEURISTICS

Artificial intelligence theorists have studied the performance of heuristic optimization methods extensively, andone of the important results they have discovered is the ‘no free lunch’ (NFL) theorem. One form of this theoremis given by Wolpert and Macready18:


648 D. DRAIN ET AL.

10

20

30

40

Fraction

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Figure 5. Fraction of design space graphs of prediction error variance

1

2

3

4

5

6

7

8

9

10

11

12

Fraction

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Figure 6. Fraction of design space graphs of slope estimation variance

Roughly speaking, we show that for both static and time-dependent optimization problems, theaverage performance of any pair of algorithms across all possible problems is identical.

This means, for example, that scattersearch is just as good as an efficient hillclimbing algorithm, if theirperformance is averaged over all possible problems. The important lesson in the NFL theorem is that noheuristic can be applied successfully to all problems; rather, the heuristic must be chosen to match the problembeing solved. With that lesson in mind, this section gives some suggestions on making effective use of hybridheuristics within the context of experiment design problems.

Good starting points are essential, especially if computational efficiency is a concern. A heuristic commencingfrom a badly chosen set of starting points may never find the global optimum, or even a good local optimum.



Starting from known good designs is not cheating—it is taking fair advantage of knowledge about the problem.If few good designs are known, then utilization of the entire palette of heuristics is required to enhance thepopulation of good starting points. One strategy that seems to work well is to use scattersearch to get a populationof (probably rather poor, but diverse) starting points, then to use simulated annealing to improve these, andfinally to ‘polish’ these solutions with hillclimbing to assure that at least a local optimum has been achieved.Another strategy demonstrated in the examples above is to use some criterion that generally produces gooddesigns (D-optimality) to generate starting points for designs measured by other criteria.

Evaluate the designs in a way that can clearly distinguish good designs from bad. This is trivial whenusing a single-number criterion like D-optimality, but more difficult when a complete evaluation requiresintegration over an area, such as with average prediction error variance. One shortcut used in Example 2 wasto evaluate designs at the most challenging points of the space (the vertices) early in the algorithm to identifysuperior designs. When balancing several evaluation criteria desirability functions seem to work well, but theircoefficients must be carefully chosen so that few designs are ranked as totally undesirable, and no designs arefound to be perfectly desirable.

All but the simplest heuristics have parameters that must be ‘tuned’ to achieve the best results: startingtemperatures for simulated annealing, or crossbreeding ratios for the genetic algorithm, for example.Advantageous parameter settings can be found by executing designed experiments with the heuristic parametersas factors and heuristic performance as the response. Parameter settings may also have to be changed during thecourse of the optimization process, so close attention should be paid to heuristic performance and adjustmentsmade as necessary.

CONCLUSION

Hybrid heuristic optimization methods are useful for solving experiment design problems in cases where othermethods cannot be applied, or are ineffective. They can accommodate multiple optimization criteria, constraintson design regions, and complex responses requiring evaluation over the entire design region. The GASAalgorithm was demonstrated to be effective in each of these situations.

REFERENCES

1. Myers RH, Montgomery DC. Response Surface Methodology: Process and Product Optimization Using DesignedExperiments (2nd edn). Wiley: New York, 2002.

2. Welch WJ. Branch-and-bound search for experimental designs based on D optimality and other criteria. Technometrics1982; 24:41–48.

3. Haines LM. The application of the annealing algorithm to the construction of exact optimal designs for linear-regressionmodels. Technometrics 1987; 29:439–447.

4. Hamada M, Martz HF, Reese CS, Wilson AG. Finding near-optimal Bayesian experimental designs via geneticalgorithms. The American Statistician 2001; 55:175–181.

5. Borkowski JJ. Using a genetic algorithm to generate small exact response surface designs. Journal of Probability andStatistical Science 2003; 1:65–88.

6. Parkinson DB. Robust design employing a genetic algorithm. Quality and Reliability Engineering International 2000;16:201–208.

7. Forouraghi B. A genetic algorithm for multiobjective robust design. Applied Intelligence 2000; 12:151–161.8. Li WD, Ong SK, Nee YC. Hybrid genetic algorithm and simulated annealing approach for the optimization of process

plans for prismatic parts. International Journal of Production Research 2002; 40:1899–1922.9. Kragelund LV. Solving a timetabling problem using hybrid genetic algorithms. Software—Practice and Experience

1997; 27:1121–1134.10. Cook RD, Nachtsheim CJ. A comparison of algorithms for constructing exact D-optimal designs. Technometrics 1980;

22:315–324.11. Mitchell TM. Machine Learning. McGraw-Hill: New York, 1997.


650 D. DRAIN ET AL.

12. Heredia-Langner A, Carlyle WM, Montgomery DC, Borror CM, Runger GC. Genetic algorithms for the constructionof D-optimal designs. Journal of Quality Technology 2003; 35:28–46.

13. Talbi EG. A taxonomy of hybrid metaheuristics. Journal of Heuristics 2002; 8:541–564.14. Minitab Inc. Release 14 for Windows. State College Pennsylvania, 2003.15. Myers RH, Khuri AI, Vining G. Response surface alternatives to the Taguchi robust parameter design approach.

American Statistician 1992; 46:131–139.16. Borror CM, Montgomery DC. Myers RH. Evaluation of statistical designs for experiments involving noise variables.

Journal of Quality Technology 2002; 34:54–70.17. Zahran AR, Anderson-Cook CM, Myers RH. Fraction of design space to assess prediction capability of response

surface designs. Journal of Quality Technology 2003; 34:377–386.18. Wolpert DH, Macready WG. No free lunch theorems for optimization. IEEE Transactions on Evolutionary

Computation 1997; 1:67–82.

Authors’ biographies

David Drain is an Assistant Professor in the Department of Mathematics and Statistics at the University ofMissouri, Rolla. He received his PhD in Industrial Engineering from Arizona State University in 2003; andhis MS in Applied Statistics in 1980, MA in Mathematics in 1976, and BS in Mathematics in 1974 fromBowling Green State University. He has over 23 years of industry experience as an applied statistician, mostrecently at Intel Corporation where he utilized the design of experiments and statistical process control insemiconductor manufacturing operations. His present research interests include heuristic optimization, responsesurface methods in the presence of correlated noise variables, and applications of spatial statistics in thesemiconductor industry.

W. Matthew Carlyle is an Associate Professor in the Operations Research Department at the NavalPostgraduate School. He joined the faculty in 2002 after working for five years as an Assistant Professorin the Department of Industrial Engineering at Arizona State University. He received his PhD in OperationsResearch from Stanford University in 1997 and his BS in Information and Computer Science from GeorgiaTech in 1992. His research interests include effective models and solution procedures for large combinatorialoptimization problems. Applications of this research have included modeling and analysis of Navy combatlogistics force size and structure, sensor mix and deployment, communications network diversion, workforceplanning, underground mining, and semiconductor manufacturing.

Douglas C. Montgomery is Professor of Engineering and Statistics at Arizona State University. He is an authorof 13 books and over 150 technical papers. He has been a recipient of the Shewhart Medal, the BrumbaughAward, the Hunter Award, and the Shewell Award (twice) from the American Society for Quality Control.He has also been a recipient of the Ellis R. Ott Award. He is one of the Chief Editors of Quality and ReliabilityEngineering International, a former Editor of the Journal of Quality Technology, and a member of several othereditorial boards. Professor Montgomery is a Fellow of the American Statistical Association, a Fellow of theAmerican Society for Quality Control, a Fellow of the Royal Statistical Society, a Fellow of the Institute ofIndustrial Engineers, and an Elected Member of the International Statistical Institute. He also serves on theTechnical Advisory Board of the United States Golf Association.

Connie Borror is an Assistant Professor in the Department of Decision Sciences. Her research interests includestatistical process control, experimental design, response surface methodology, and measurement systemsanalysis.

Christine M. Anderson-Cook is an Associate Professor in the Department of Statistics at Virginia Tech.Her areas of research include response surface methodology, design of experiments, graphical assessmentmethods and industrial statistics. She is a member of ASQ and ASA.


Documents

A Genetic Algorithm Hybrid for Constructing Optimal Response Surface Designs