Parallel Algorithms for Black-Box Optimization...Black-box optimization problems The MADS algorithm Parallel versions of MADS Codes Discussion PP10 Ecole Polytechnique de Montr eal

Black-box optimization problems The MADS algorithm Parallel versions of MADS Codes Discussion

PP10

Ecole Polytechnique de Montreal

Parallel Algorithms for Black-BoxOptimization

Sebastien Le DigabelCharles AudetJohn Dennis

2010–02–24

PP10: black-box optimization 1/18


Presentation outline

Black-box optimization problems

The MADS algorithm

Parallel versions of MADS

Codes

Discussion





The MADS algorithm


Codes

Discussion





The MADS algorithm


Codes

Discussion





The MADS algorithm


Codes

Discussion





The MADS algorithm


Codes

Discussion




The MADS algorithm


Codes

Discussion




We consider the nonsmooth optimization problem:

Minimize f(x)subject to x ∈ Ω,

where evaluation of the functions are usually the result of acomputer code (a black-box).



Black-boxes as illustrated by J. Simonis [ISMP 2009]




The MADS algorithm


Codes

Discussion



MADS and direct search principle

Algorithm

f(black box)

x1,x2,... f(x1),f(x2),...

x0 x*



Mesh Adaptive Direct Search (MADS)

I Audet and Dennis [SIOPT, 2006]

I Iterative algorithm that evaluates the black-box functions atsome trial points.

I Trial points are generated on a spatial discretization: themesh.

I One iteration consists in generating a list of trial pointsconstructed from poll directions. These directions grow dense.

I At the end of the iteration, the mesh size is reduced if no newiterate is found.

I Algorithm is backed by a convergence analysis based on theClarke Calculus for nonsmooth functions.



Poll illustration (successive fails and mesh shrink)

∆k = 1

rxk

rp1

rp2

r

p3

trial points=p1, p2, p3




∆k = 1

rxk

rp1

rp2

r

p3


∆k+1 = 1/4

rxkr

p4 rp5

r

p6

= p4, p5, p6




∆k = 1

rxk

rp1

rp2

r

p3


∆k+1 = 1/4

rxkr

p4 rp5

r

p6

= p4, p5, p6

∆k+2 = 1/16

rxk

rp7

r p8rSSp9

= p7, p8, p9




The MADS algorithm


Codes

Discussion



pMADS

I Idea: simply evaluate the trial points in parallel.I Synchronous version:

I The iteration is ended only when all the evaluations in progressare terminated.

I Processes can be idle between two evaluations.I The algorithm is identical to the scalar version.

I Asynchronous version:I If a new best point is found, the iteration is terminated even if

there are evaluations in progress. New trial points are thengenerated.

I Processes never wait between two evaluations.I ’Old’ evaluations are considered when they are finished.I The algorithm is slightly reorganized.



PSD-MADS

I PSD: Parallel Space Decomposition.

I Idea: each process executes a MADS algorithm on asubproblem and has responsibility of small groups of variables.

I Based on the block-Jacobi method [Bertsekas, Tsitsiklis 1989]and on the Parallel Variable Distribution [Ferris, Mangasarian1994].

I Objective: solve larger problems (' 50− 500 instead of' 10− 20).

I Asynchronous method.

I Convergence analysis.

I Audet, Dennis, Le Digabel [SIOPT 2008].



PSD-MADS: processesI Master

I receives all slave’s signalsI updates current solution and meshI decides subproblem variablesI sends subproblem data

I SlavesI receive subproblem dataI optimize subproblemI send optimization data

I Cache serverI memorizes all black-box evaluationsI allows the “cache search” in slave processes

Master+ Cache Server

pollstern variables1 direction

subproblem data

optimiza

tion result

slave 1ns variables

2ns directions

slave 2ns variables

2ns directions

slave 3ns variables

2ns directions

slave 4ns variables

2ns directions

slave 5ns variables

2ns directions



! P =1x0=x*=[10 10 10 10]f(x0)=10

Master

time



! P =1x0=x*=[10 10 10 10]f(x0)=10

Slave s2

Pollster

Slave s3

!0=1 f(x0)=10

f(x0)=10

! min=1 N2=3,4

!0=1! min=1

f(x0)=10N3=2,3

f(y1)=14

! P =1

stop (1 it.) it. fail

!=1

time

Master

x0=[ 10 10 10 10 ]

]y1=[ 11 10 10 10

x0=[ 10 10 10 10 ]

10 10 ]y1=[ 11 10

x0=[ 10 10 10 10 ]

10 10y1=[ 11 10 ]



Master!P =1x0=x*=[10 10 10 10]f(x0)=10

Pollster

Slave s3

!0=1 f(x0)=10

f(x0)=10

!min=1 N2=3,4

!0=1!min=1

f(x0)=10N3=2,3

f(y1)=14

!P =1

f(x0)=10!=1


f(y1)=12

x*=[10 10 9 10]f(x*)=9

Slave s2

f(y1)=16

it. success

f(y2)=11

time

!=1/4

]

y3=[

10 ]

]y1=[11 10 10 10

x0=[10 10 10 10 ] x0=[10 10 10 10 ]

y1=[10 10 9.75 10 ]

x0=[10 1010 10]

10 10 ]11 10y1=[ 10 10 ]y2=[ 9 10

10 10

10

10 11y2=[10 10 ]11 10y1=[

x0=[1010 10

y3=[

f(y2)=9



!P =1

!min=1

!min=1

!P =1x0=x*=[10 10 10 10]f(x0)=10

Slave s2

Pollster

Slave s3

!0=1 f(x0)=10

f(x0)=10

min=1 N2=3,4

!0=1!min=1

f(x0)=10N3=2,3

f(y1)=14

f(x0)=10!=1


f(y1)=12

x*=[10 10 9 10]f(x*)=9

f(y2)=9

f(y1)=16 f(y2)=11

it. success

!P =1

!=1/4 !=1 f(x0)=9

f(y3)=10

stop (3 ev.)

stop (3 ev.)

!0=1

!0=1

!


N2=1,3

Master

time

N3=2,410

]

f(y3)=15

9

10 10 10 10 ] ]10 10 10 10 ]

10 10 10 10x0=[ ]

10 10 9 10

10 10 ]11 10 10 10 ]9 10 10 10 ]10 11y1=[ y2=[ y3=[

10

y1=[

x0=[

10 9 9 10

10 10

11 10 10 10

10

x0=[

x0=[10 10 9 10 ]

x0=[ x0=[

10 ]10y1=[ 11 10 10 ]10y2=[ 10 11 10 ]10y3=[ 9 10

10 9y1=[11 10 ]

x0=[10 ]

y1=[ ]10 10 9.75 10y1=[]

10 ]910y1=[ 11




The MADS algorithm


Codes

Discussion



Free C++ codes (1/2)

I NOMAD: Nonsmooth Optimization with MADS:I Polytechnique Montreal.I MPI.I MADS, pMADS and PSD-MADS (March).I http://www.gerad.ca/nomad.

I APPSPACK: Asynchronous Parallel Pattern Search:I Sandia National Laboratories.I MPI.I Generalized Pattern Search.I http://software.sandia.gov/appspack.


http://www.gerad.ca/nomad

http://software.sandia.gov/appspack/


Free C++ codes (2/2)

I HOPSPACK: Hybrid Optimization Parallel Search PACKage:I Sandia National Laboratories.I MPI and multithreading.I Multiple solvers.I http://software.sandia.gov/trac/hopspack.

I DAKOTA: Design Analysis Kit for Optimization and TerascaleApplications:

I Sandia National Laboratories.I MPI.I Multiple solvers.I http://www.cs.sandia.gov/DAKOTA.


http://software.sandia.gov/trac/hopspack/

http://www.cs.sandia.gov/DAKOTA



The MADS algorithm


Codes

Discussion


Documents

Parallel Algorithms for Black-Box Optimization...Black-box optimization problems The MADS algorithm Parallel versions of MADS Codes Discussion PP10 Ecole Polytechnique de Montr eal