1

OQGRG: a Scatter Search Approach for Solving Constrained Non-Linear Global

Optimization Problems

Zsolt Ugray, The University of Texas at Austin, MSIS Dept.Leon Lasdon, The University of Texas at Austin , MSIS Dept.

John Plummer, The University of Texas at Austin , MSIS Dept.

2

Heuristic Search

• Problem: min f(x) subject to x in X• A heuristic defines a neighborhood N(x) for each x

– Example:x=permutation of n jobs– For a pairwise exchange heuristic, N(x) = set of all permutations

reachable from x by interchanging 2 components of x,n(n-1)/2 elements

• Descent method:– choose x0 in X– make a move: find x1 in N(x0) such that f(x1)<f(x0)– if no such x1 exists, stop– else, replace x0 by x1 and repeat

3

Meta-heuristic Search• Meta-heuristic: a heuristic which guides another heuristic, prevents

it from stopping at a local solution• Examples: genetic algorithms, tabu and scatter search, simulated

annealingADVANTAGES• Better chance of finding global optimum• Can handle continuous and discrete variables, nonsmooth functions• Can handle non-algebraic problem forms

– routing (traveling salesman)– scheduling

DISADVANTAGES• Sometimes no guarantee of success, no bounds• Often slow unless well-designed• Few off-the shelf codes available

4

Simulated Annealing Choose an initial solution, x, an initial temperature, T, a lower

limit on temperature, TLOW, and an inner iteration limit, L. While (T>TLOW), do { For k = 1,2,…,L, do{• Make a random choice of an element, xnew in N(x) .• move_value=f(xnew)-f(x)• If move_value<= 0, (downhill move) set x = xnew• If move_value > 0 (uphill move) set x = with probability

exp(-move_value/T). end inner loop} Reduce temperature according to an annealing schedule. An

example is new T = c*T, where 0<c<1.} End temperature loop

5

Genetic Algorithms• .Choose an initial population and evaluate the fitness of each

individual• .While termination condition not satisfied do{

– 2.1 If crossover condition satisfied then» 2.1.1 {Select parent individuals» 2.1.2 Choose crossover

parameters» 2.1.3 Perform crossover}

– 2.2 If mutation condition satisfied then» 2.2.1 {choose mutation points» 2.2.2 perform mutation}

– 2.3 evaluate fitness of offspring– 2.4 update population}

6

OQGRG-A Multi-start Search Method

• Intended for nonlinearly constrained, smooth, non-convex NLP’s and MINLP’s

• Combines the OptQuest Callable Library (a scatter search code) of Laguna, Kelly,Glover with a local NLP solver

• Currently uses LSGRG2 , a sparse gradient-based GRG code, as the local NLP solver

• Written in C

7

OptQuest (OQ) and OQ Callable Library• OQ is the Glover-Laguna-Kelley implementation of scatter

search• Option within the Crystal Ball Excel add-in for Monte-

Carlo simulation• Handles MINLP’s• OQ callable library:

– set up problem– define variables, bounds, constraints, requirements– create initial population– for N iterations:

• get trial solution• evaluate objective• put trial solution, objective back to database• regenerate population if necessary

8

Initializing and Updating the Population

• Initial Population (size = psize)– include origin,user point,upper and lower bounds– generate stratified random points within bounds– choose a subset whose distance from previous

points is maximized• Use population to generate new trial points,

evaluate objective at each one• After all evaluated, population is the best psize

solutions found so far

9

Old Solutions

New Solutions

Scatter Search: trial solution generation

10

ex8_1_5 initial population

-11

-6

-1

4

9

-11 -6 -1 4 9

x

y

11

ex8_1_5 initial population

-11

-6

-1

4

9

-11 -6 -1 4 9

x

y

12

OptQuest Difficulties

• If feasible region is narrow (e.g. equality constraints) then very difficult for OQ to find a feasible solution

• Often finds good solution fast, but requires many additional function evaluations to get high accuracy

• Gradient-based NLP solvers (GRG,SQP,SLP) are much better at getting feasible and attaining high accuracy, but find “nearest” local optimum.

13

When to Start the Local Solver• Local solver used is GRG (sparse version)• A GRG call solver is expensive-many function

evaluations• Ideally: start once in the basin of attraction of

each local solution• Don’t start from a trial point if

– too close to a previously found local solution(distance filter)

– exact penalty function value is too large(merit filter)

14

Distance Filter

• Store max distance traveled to each local optimum

• If the distance from a candidate starting point to any local opt is < 0.75*maxdist, don’t start GRG

• This assumes regions of attraction are spherical, and maxdist is a good estimate of the radius

15

Merit Filter• Use L1 exact penalty , P(x,w), as merit function

– P(x,w)=obj(x)+sum(i,wi*infeasi(x))– wi > max abs Lagrange multiplier for constraint i over

all local solutions• Don’t start from candidate points which have

value for P > threshold• Initially threshold=best P value over all candidate

points so far• If P > threshold more than 20 consecutive times,

threshold <-- threshold +0.2*abs(threshold)

16

OPQUEST/GRG ALGORITHM-11/1/00

INITIALIZATION

Read_Problem_Parameters (size, variables, bounds, constraints, starting point);Setup_OptQuest_Parameters (size, iteration limits, population, accuracy, variables,

bounds, constraints);Initialize_OptQuest_Population;

INITIAL OPTQUEST ITERATIONS

For (I=1; I<(initial iteration limit); I+1) DOGet (Trial Solution from OptQuest);Evaluate (Objective + Constraint Values at Trial Solution,);Put (Trial Solution , Objective + Constraint Values to OptQuestdatabase);ENDDO

Get_Best_Point_from_OptQuest_database (Starting Point);Call_Local_Solver_GRG (Trial Solution, Local Solution);

MAIN ITERATIVE LOOP

While (Stopping Criteria not met) DO {1. Get (Trial Solution from OptQuest);

Evaluate (Objective + Constraint Values at Trial Solution,);Put (Trial Solution, Objective + Constraint values to OptQuest database);Calculate_ Penalty_ Function (Trial Solution);DISTANCE FILTER:If (Trial Solution is within a neighborhood of any local solutions found) {

GOTO 1.;}MERIT FILTER:If (Penalty Function > Best Trial Solution found) {

If (above condition true WaitCycle consecutive times) {Relax_Best_Trial_Solution (RelaxFactor);}

GOTO 1.;}Call_Local_Solver_GRG (Trial Solution, Local Solution);Analyze_Solution (GRG Terminating Condition);Update_Local_Solutions_Found;Update_Largest_Lagrange_Multipliers_Found;

ENDDO

Write_Problem_Solution_Statistics (number of iterations, local solutions found, gapsfrom known global solution, function calls, elapsed time);

17

Handling Discrete Variables

• Current strategy:– GRG treats discrete variables as fixed, at values provided by OQ– OQ varies both discrete and continuous variables– GRG solutions not returned to OQ– Apply distance and merit filters only if discrete variables are the same

• Another strategy we will try– as above, but OQ varies only the discrete variables– equivalent to OQ solving the discrete problem, where GRG optimizes

over the continuous ones

18

GAMS Interface• Motivation: Large Set of Test Problems Coded in

GAMS by Floudas et.al., best known sol available– downloadable at http://titan.princeton.edu/TestProblems/– Most from Chemical process design or operation– 144 problems– See book: Handbook of Test Problems in Local and Global

Optimization, by Floudas et. al., Kluwer Academic Publishers, ISBN 0-7923-5801-5

• Uses GAMS C Language Library Routines• most small but some with over 100

variables, a few with over 1000.• Most arise from chemical engineering

19

max discrete max linear max nonlinearSeries problems max vars vars constraints constraints Problem Type

EX2_1_x 14 24 0 10 0 concave QP (min)EX3_1_x 4 8 0 4 6 quad obj and constEX4_1_x 9 2 0 0 2 obj or cons polynomialEX5_2_x 2 32 0 8 11 bilinear-poolingEX5_3_x 2 62 0 19 34 distillation col sequencingEX5_4_x 3 27 0 13 6 heat exch networkEX6_1_x 4 12 0 3 6 gibbs free energy minEX6_2_x 10 9 0 3 0 gibbs free energy minEX7_2_x 4 8 0 3 12 gen geometric progEX7_3_x 6 17 0 10 11 robust stability ana.EX8_1_x 8 6 0 0 5 small unconstrained,nl constrainedEX8_2_x 5 55 0 6 75 batch plant design-uncertaintyEX8_3_x 14 141 0 43 65 CSTR network synthesisEX8_4_x 8 62 0 0 40 constrained least squaresEX8_5_x 6 6 0 2 2 min tan plane distanceEX8_6_1 N from 4 to 147 3N 0 0 0 Lenard-Jones energy minEX8_6_2 N from 5 to 80 3N 0 0 0 Morse energy minEX9_1_x 10 29 0 27 5 bilevel LPEX9_2_x 9 16 0 11 6 bilevel QP

EX12_2_x 6 11 8 9 4 MINLPEX14_1_x 9 10 0 4 17 infinity norm solution of equationsEX14_2_x 9 7 0 1 10 infinity norm solution of equations

Total 144Floudas Problem Summary

20

Parameters of base case

• Sort by max abs x value of best known solution

• Solve all probs with max x between 10**k and 10**(k+1) in a group. Set bounds of + and - 10**(k+1) on all unbounded variables

• Use 200 initial OQ iterations and 1000 total, but try again with (1000,5000) if fall short of best known sol.

• OQ knows linear constraints, generates trial points which satisfy them

21

Floudas problems-results

• Best known solution found or improved on in 118 of 121 problems

• Best OQGRG solution found on first GRG call (itn 201) in 65 problems

• Average GRG calls to find best OQGRG solution: bnd GRG calls vars

– 1 2.22 7.5– 10 2.13 9.2– 100 3.83 11.5– 10,000 1.4 52.6– 1000 7.8 69.1

22

Floudas results-group averages

• Bnd total GRG fcn to best vars• 1 16.4 414 7.5• 10 11.9 592 9.2• 100 19.9 985 11.5• 10,000 15.3 1356 52.6• 1000 23.7 2538 69.1

23

Advantages and Disadvantages

• Advantages– Finds good or optimal solutions rapidly– Can handle problems of general form with hundreds or

thousands of continuous variables and/or constraints– Can handle MINLP’s with many discrete variables– strong ability to find a feasible solution (due to GRG)– works very differently from DICOPT or branch and bound

• Disadvantages– No guarantee of finding a global solution– No way to tell if you have found a global solution

24

Future Work

• Determine effects of varying important OQGRG options and parameters– basin and merit filter parameters– OQ treats continuous vars as discrete– OQ strategies to intensify or diversify– fractional change criterion for overall method

• Comparison with LGO,MLSL of Fylstra, Random starts, DICOPT

• Improvements to OQGRG algorithm motivated by test results