..........

.....

.....................................................................

.....

......

.....

.....

.

.

......

Covariance Matrix Adaptation EvolutionStrategies(CMA-ES)

Hossein Abedi

Evolutionary Computation

Autumn 2014

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 1 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Overview

...1 Introduction

...2 Selection and Recombination

...3 Adaptation of covariance matrix

...4 Step size control

...5 Experiments

...6 Conclusion

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 2 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Introduction

Idea

Introduced by Hansen and Ostermeier in 2001

The idea:

Figure : Movement toward a minimum through 3 generations

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 3 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Selection and Recombination

Generating the children

New points are sampled normally distributed:Xi ∼ M + σNi (0,C ), for i=1,...,λ

Figure : Different shapes of C as a hyperelipsoid in 2D

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 4 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Selection and Recombination

Selection and Recombination

The mean vector M ∈ ℜn is calculated as the weighted average of thebest candidate solutions: M=

∑µi=1 wiXi :λ

Where:∑µi=1 wi = 1

w1 ⩾ w2 ⩾ ... ⩾ wµ > 0f (X1:λ) ⩽ f (X2:λ) ⩽ ... ⩽ f (Xµ:λ)

µeff = ( ||w ||1||w ||2 )

2 = 1∑µi=1 w

2i

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 5 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Estimating the covariance matrix from scratch

For the sake of simplicity set σ(g) = 1

Estimating distribution within the population:

C(g+1)emp = 1

λ−1

∑λi=1(X

(g+1)i − 1

λ

∑λj=1 Xj)(X

(g+1)i − 1

λ

∑λj=1 Xj)

T

Estimating distribution of sampled steps:

C(g+1)λ = 1

λ

∑λi=1(X

(g+1)i −M(g))(X

(g+1)i −M(g))T

Where:

The sampled steps are:

X(g+1)i −M(g)

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 6 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Estimating the covariance matrix

Estimating distribution of the most successful steps:

C(g+1)µ = 1

µ

∑µi=1 wi (X

(g+1)i :λ −M(g))(X

(g+1)i :λ −M(g))T

Estimation of Multivariate Normal Algorithm(ENMA):

C(g+1)µ = 1

µ

∑µi=1(X

(g+1)i :λ −M

(g+1)enma )(X

(g+1)i :λ −M

(g+1)enma )T

Where:

M(g+1)enma = 1

µ

∑µj=1 X

(g+1)j :λ

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 7 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Estimating the covariance matrix

Comparison:

Figure : Covariance matrix estimation on f (x1, x2) = −∑2

i=1 xi

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 8 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Rank µ update

Smaller λ means faster but less global search

To give recent generations a higher weight, consider a leraning rate cµand the equation below:

C (g+1) = (1− cµ)C(g) + cµ

1(σ(g))2

C(g+1)µ

Where:1cµ

is called the time back horizon

Figure : Example of exponential smoothing

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 9 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Rank µ update

C (g+1) = (1− cµ)C(g) + cµ

1µ

∑µi=1 wiOP(

X(g+1)i :λ −M(g)

σ(g) )

Where:

OP(y) = yyT

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 10 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Rank one update

Evolution Path (Pc ∈ ℜn): sum of consecutive steps:M(g+1)−M(g)

σ(g) + M(g)−M(g−1)

σ(g−1) + ...

Figure : Evolution path

N(0, I )y1 + N(0, I )y2 + ...+ N(0, I )yg ∼ N(0,∑g

i=1 yiyTi )

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 11 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Rank one update

Using exponential smoothing:

P(g+1)c = (1− cc)P

(g)c +

√cc(2− cc)µeff

M(g+1)−M(g)

σ(g)

Where:√cc(2− cc)µeff is a scaling factor such that :P

(g+1)c ∼ N(0,C )

So rank one update with sign is :

C (g+1) = (1− c1)C(g) + c1OP(P

(g+1)c )

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 12 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Adaptation of covariance matrix

Cumulation

C (g+1) = (1− c1 − cµ)C(g) + c1(y

(g+1)c )(P

(g+1)c )T +

...cµ1µ

∑µi=1 wiOP(

X(g+1)i :λ −M(g)

σ(g) )

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 13 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Step size control

Step size control

Using the evolution path for adapting the stepsize σ

Figure : Different evolution path senarios for 6 consecutive mean vectors

σ(g+1) = σ(g) exp ( cσdσ (||p(g+1)

σ ||E ||N(0,I )|| − 1))

Where:

p(g+1)σ = (1− cσ)p

(g)σ +

√cσ(2− cσ)µeff (C

(g))−12M(g+1)−M(g)

σ(g)

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 14 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Experiments

Test on seperable and non rotated

0 200 400 600 800 1000 1200 1400 1600 1800 20000

2

4

6

8

10

12

14

0.01*function evauations

f min

CLPSOCMA−ES

Figure : Results on Ackley test function

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 15 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Experiments

Test on CEC2015(shifted,rotated and non-seperable)

0 10 20 30 40 50 60 70 80 90 10010

12

14

16

18

20

22

24

% of function evaluation

log(

f min

)

CLPSOCMA−ES

Figure : Results on function 2 CEC2015

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 16 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Experiments

Test on CEC2015(shifted,rotated and non-seperable)

0 10 20 30 40 50 60 70 80 90 100500

502

504

506

508

510

512

514

516

518

% of function evaluations

f min

CMA−ESCLPSO

Figure : Results on function 5 CEC2015

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 17 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Experiments

Test on CEC2015(shifted,rotated and non-seperable)

0 10 20 30 40 50 60 70 80 90 100603

604

605

606

607

608

609

610

611

612

% of function evaluation

f min

CLPSOCMA−ES

Figure : Results on function 6 CEC2015

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 18 / 19

..........

.....

.....................................................................

.....

......

.....

.....

.

Conclusion

Conclusion

Applicable to problems in which many variables are correlated

Good local search

Hossein Abedi (Evolutionary Computation) CMA-ES Autumn 2014 19 / 19