1 © 2011 The MathWorks, Inc.

Global Optimization with MATLAB®

Products

Account Manager

이 장 원 차장

Application Engineer

엄 준 상

2

Agenda

Introduction to Global Optimization

Survey of Solvers with Examples

– MultiStart

– Global Search

– Pattern Search

– Simulated Annealing

– Genetic Algorithm / Multiobjective

Genetic Algorithm

Additional Resources

Question & Answer

-2

0

2

-3-2

-10

12

3

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Local minima

Global minima

3

Optimization – Finding answers to problems automatically

Objectives

Achieved?

NO

Optimal

Design

YES Model or

Prototype

Modify Design

Parameters

Initial

Design

Parameters

OPTIMIZATION PROCESS

Finding better (optimal) designs

Faster design evaluations

Useful for trade-off analysis (N dimensions)

Non-intuitive designs may be found

Optimization benefits include: Design process can be performed:

Antenna Design Using Genetic Algorithm http://ic.arc.nasa.gov/projects/esg/research/antenna.htm

Manually

(trial-and-error or iteratively)

Automatically

(using optimization techniques)

4 0 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

t

c

c=b1e-b

4t+b

2e-b

5t+b

3e-b

6t

Example Global Optimization Problems

Why does fmincon have a hard time finding the

function minimum?

Why didn’t nonlinear regression find a good fit?

Why didn’t fminunc find the maximum volumetric

efficiency?

0 5 10

-10

-5

0

5

10

x

Starting at 10

0 5 10

-10

-5

0

5

10

x

Starting at 8

0 5 10

-10

-5

0

5

10

x

Starting at 6

x s

in(x

) +

x c

os(2

x)

0 5 10

-10

-5

0

5

10

x

Starting at 3

0 5 10

-10

-5

0

5

10

x

Starting at 1

0 5 10

-10

-5

0

5

10

x

Starting at 0

x s

in(x

) +

x c

os(2

x)

Revolutions Per Minute, RPM

Manifold

Pre

ssure

Ratio

Peak VE Value = 0.96144

Start

End

1000 2000 3000 4000 5000 60000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

5

Global Optimization

Goal:

Want to find the lowest/largest value of

the nonlinear function that has many local

minima/maxima

Problem:

Traditional solvers often return one of the

local minima (not the global)

Solution:

A solver that locates globally optimal

solutions Rastrigin’s Function

6

Global Optimization Toolbox – For problems that

contain multiple maxima/minima or are non-smooth

Optimization T

oolbox

Global Optimization

Toolbox

Faster/fewer function eval

uations Larger problems (higher d

imensions) Finds local minima/maxim

a

Finds global minima/maxi

ma (most of the time) Better on

non-smooth

stochastic

discontinuous

undefined gradients

Custom data types

(in GA and SA solvers)

7

MULTISTART

8

What is MultiStart?

Run a local solver from

each set of start points

Option to filter starting

points based feasibility

Supports parallel

computing

11

GLOBAL SEARCH

12

What is GlobalSearch?

Multistart heuristic algorithm

Calls fmincon from multiple

start points to try and find a

global minimum

Filters/removes non-promising

start points

13

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview Schematic Problem

Peaks function

Three minima

Green, z = -0.065

Red, z= -3.05

Blue, z = -6.55

x

y

14

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 0 Run from specified x0

x

y

15

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1 Generate stage 1 start points via Scatter Search

3

6

0

0 0

4

0

-2

x

y

16

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1 Find stage 1 start point with lowest penalty value

3

6

0

0 0

4

0

-2

x

y

17

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1 Run from best stage 1 point

x

y

18

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Generate stage 2 start points using Scatter Search

x

y

19

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Analyse each stage 2 point in turn.

x

y

20

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Don’t run points that are in basins of existing

minimum

x

y

21

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Analyse each stage 2 point in turn.

x

y

22

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Don’t run points whose penalty value exceeds

threshold

6

Current penalty

threshold value : 4

x

y

23

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Analyse each stage 2 point in turn

x

y

24

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Run start point if it satisfies distance & merit criteria

Current penalty

threshold value : 4

-3

x

y

25

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Expand basin of attraction if minimum already found

Current penalty threshold value : 2

-0.1

x

y Basins can overlap

28

SIMULATED ANNEALING

29

What is Simulated Annealing?

A probabilistic metaheuristic approach based upon the

physical process of annealing in metallurgy.

Controlled cooling of a metal allows atoms to realign from a

random higher energy state to an ordered crystalline

(globally) lower energy state

By analogy, simulated annealing replaces a current solution

by randomly choosing a nearby solution

A “nearby” solution is determined by the solution

temperature

30

Simulated Annealing Overview – Iteration 1 Run from specified x0

x

y

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

0.9

31

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1 Randomly generate a new point according to probability distribution and

current temperature

3

x

y 0.9

Possible New Points:

Standard Normal N(0,1) * Temperature

Temperature = 1

32

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1 If lower, accept the point, if higher, accept based upon acceptance

probability

3

x

y 0.9

Temperature = 1

11.01

1/)9.03(

Taccepte

P

33

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1 Randomly generate a new point according to probability distribution and

current temperature

3

x

y 0.9

Temperature = 1

0.3

34

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1 Randomly generate a new point according to probability distribution and

current temperature, accept new point if lower value

3

x

y 0.9

Temperature = 1

0.3

35

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 2 Lower temperature according to temperature schedule

3

x

y 0.9

Temperature = 1

0.3

36

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 2 Lower temperature according to temperature schedule and generate new

point

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

37

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N-1 After several iterations, the search radius becomes small and we narrow

in on a local solution

3

x

y 0.9

Temperature = 0.1

0.3

-1.2

-3

38

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N Reset temperature and start the process again (reanneling)

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

39

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N Reset temperature and start the process again (reannealing)

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

27.01

1/))3(2(

Taccepte

P

40

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N Reset temperature and start the process again (reannealing)

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

41

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1 Reduce temperature and continue…

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

42

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1 Reduce temperature and continue…

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

43

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration … Reduce temperature and continue…

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

-6.5

47

PATTERN SEARCH

(DIRECT SEARCH)

48

What is a Pattern Search?

An approach that uses a pattern of search directions around

the existing points, the mesh

Polls the mesh for a better solution and moves to that point

Expands/contracts the mesh around the current point when a

solution is not found

Does not rely on gradient information

49

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1 Run from specified x0

x

y

3

50

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1 Apply pattern vector, poll new points for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)

51

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 2 Increase mesh size and repeat…

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

1.6

0.4

4.6

2.8

-4

0.3 -2.8

Complete Poll

52

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 3 Mesh expansion: increase mesh size and repeat…

x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

1.6

0.4

4.6

2.8

-4

0.3 -2.8

53

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 4 Refine mesh: decrease mesh size and repeat…

x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

1.6

0.4

4.6

2.8

-4

0.3 -2.8

54

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration N Continue expansion/contraction until convergence…

x

y

3 1.6

0.4

4.6

2.8

-4

0.3 -2.8

-6.5

57

GENETIC ALGORITHM

58

What is a Genetic Algorithm?

Genetic Algorithms use concepts from evolutionary biology to

find exact or approximate solutions to optimization problems

Start with an initial generation of candidate solutions that are

tested against the objective function

Subsequent generations evolve from the 1st

through selection, crossover and mutation

The individual that best minimizes the given

objective is returned as the ideal solution

59

How Evolution Works – Binary Case

Selection

– Retain the best performing bit strings from one generation to the

next. Favor these for reproduction

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

– child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

– parent = [ 1 0 1 0 0 1 1 0 0 0 ]

– child = [ 0 1 0 1 0 1 0 0 0 1 ]

60

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1 Evaluate initial population

x

y

61

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1 Select a few good solutions for reproduction

x

y

62

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2 Generate new population and evaluate

x

y

63

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2 Select a few good solutions for reproduction

x

y

64

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3 Generate new population and evaluate

x

y

65

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3 Select a few good solutions for reproduction

x

y

66

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration N Continue process until stopping criteria are met

x

y

Solution found

69

Comparison of Solver (Default) Performance

x

y

rf([x,y])

-6 -4 -2 0 2 4 6-6

-4

-2

0

2

4

6

FuncValue

Min

StartPt

x

y

rf([x,y])

-60 -40 -20 0 20 40 60-60

-40

-20

0

20

40

60

FuncValue

Min

StartPt

x

y

rf([x,y])

-40 -20 0 20 40 60

-40

-20

0

20

40

60FuncValue

Min

StartPt

70

71

Additional Resources

Upcoming Webinars

Speeding Up Optimization with Parallel Computing

(August 2010)

On-demand Webinars

Genetic Algorithm in Financial Applications

Tips & Tricks: Getting Started with Optimization

Introduction to Optimization

72

Contact Information

North America

– Phone: 508-647-7000

– E-mail: support@mathworks.com

Outside North America

– Contact your local MathWorks office or reseller:

www.mathworks.com/contact

73

Questions?

74

MATLAB Provides the Foundation for

Optimization

The leading environment for technical computing

– Customizable

– Numeric computation

– Data analysis and visualization

– The de facto industry-standard, high-level programming language for algorithm development

– Toolboxes for statistics, optimization, symbolic math, signal and image processing, and other areas

– Foundation of the MathWorks product family

75

Optimization Toolbox Solve standard and large-scale optimization problems

Graphical user interface and command line functions for: Linear and nonlinear programming

Quadratic programming

Nonlinear least squares and nonlinear equations

Multi-objective optimization

Binary integer programming

Additional Capabilities:

Parallel computing support in selected solvers

Customizable algorithm options

Choose between standard and large-scale algorithms

Output diagnostics

76

Global Optimization Toolbox Solve multiple maxima, multiple minima, and nonsmooth optimization problems

Graphical user interface and command line functions for: Global Search solver

Multistart solver

Genetic algorithm solver

– Single objective

– Multiobjective with Pareto front

Direct search solver

Simulated annealing solver

Useful for problems not easily addressed with Optimization Toolbox: Discontinuous

Highly nonlinear

Stochastic

Discrete or custom data types

Undefined derivatives

Multiple maxima/minima

77

Anatomy of an Optimization Problem

General Form Accepted by MATLAB Solvers

Objective Function

Subject to Constraints (i.e. such that)

uxl

xcbxA

xcbAx

eqeqeq

0)(,

0)(,

)(min xfx

Typically a linear or nonlinear function

Linear constraints

• inequalities

• equalities

• bounds

Nonlinear constraints

• inequalities

• equalities

Decision variables

78

MATLAB Optimization Products and Example

Applications

MATLAB Statistics Toolbox

Curve Fitting Toolbox

Optimization Toolbox Genetic Algorithm and

Direct Search Toolbox

Solving Equations

f(x) = 0

Root finding

Systems of equations

Real roots finding (1D):

Linear Systems:

F(x) = 0

Ax-b = 0 (i.e. Ax=b)

>> x = A \ b

Real roots finding (N-D):

Nonlinear Systems

F(X) = 0

Noisy, discontinuous root

finding (N-D)

Noisy, discontinuous systems:

F(X) = 0

Curve/Modeling

Fitting

curve fitting

parameter estimation

(model fitting)

Basic (linear) curve fitting Advanced (nonlinear) curve

fitting

Model Fitting (least squares)

Constrained curve fitting

Parameter Estimation

Noisy, Discontinuous

parameter estimation

Trade-Off Studies

Maximization

Minimization

Goal seeking

Multiobjective

Unconstrained nonlinear

minimization

Constrained nonlinear

minimization

Noisy, Discontinuous, ill-

defined mimization

-6 -4 -2 0 2 4 6-100

-50

0

50

100

x

x3-2 x-5

0 0.5 1 1.5 2 2.50.5

0.6

0.7

0.8

0.9