A Short Introduction to COCO

Tea Tušar

Computational Intelligence GroupDepartment of Intelligent SystemsJožef Stefan InstituteLjubljana, Slovenia

July 15, 2018

Workshop on Game-Benchmark for Evolutionary AlgorithmsGenetic and Evolutionary Computation Conference, GECCO 2018Kyoto, Japan

1

Why benchmark optimization algorithms?

No free lunch theorem ⇒ No algorithm works best for alloptimization problems

Purpose of benchmarking: To be able to select the best algorithmfor the given real-world optimization problem

Preconditions

• The real-world problem with some known properties• Test problems with similar properties to those of the real-worldproblem

• Results of several optimization algorithms on these testproblems for any number of evaluations

2

How to benchmark optimization algorithms?

The COCO platform

• COCO (Comparing Continuous Optimizers)• https://github.com/numbbo/coco• Automatized benchmarking of optimization algorithms

• Test problems with known properties• Data of previously run algorithms available for comparison• Provides interfaces to C/C++, Python, Java, Matlab/Octave

• Being developed at Inria Saclay, France, since 2007

3

Benchmarking with COCO

COCOexperiments

C

Results of youralgorithm

Log �les

COCOpostprocessing

Python

Results of otheralgorithms

Log �les

Data pro�les

0 2 4 6log10(# f-evals / dimension)

0.0

0.2

0.4

0.6

0.8

1.0

Frac

tion

of fu

nctio

n,ta

rget

pai

rs

RANDOMSEARF1-CMAESRL-SHADE-GP1-CMAESCGA-ring1BFGS-P-raMCS huyer1plus2mirsimplex pEDA-PSO eL-BFGS-B-CMAES posoPOEMS kuCMAES-APOALPS hornIP-500 liVNS garciNIPOPaCMADE posik best 2009bbob f1-f24, 5-D

51 targets: 100..1e-0815 instances

v2.2.1.417

Tables

C/C++interface

C

Pythoninterface

Python

Javainterface

Java

Matlab/Octaveinterface

Matlab

Your favoritealgorithm

Matlab

Requirements: C compiler and Python (other languages are optional)

4

The fixed-target approach

Interested in the runtime (number of function evaluations) neededto achieve a target value

Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Fixe

d bu

dget

Fixed target

Convergence graph

5

Data profile

The data profile is the empirical cumulative distribution function(ECDF) of the recorded runtimes

0 20 40 60 80 100Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Convergence graph

6

Data profile

The data profile is the empirical cumulative distribution function(ECDF) of the recorded runtimes

0 20 40 60 80 100Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Convergence graph

6

Data profile

The data profile is the empirical cumulative distribution function(ECDF) of the recorded runtimes

0 20 40 60 80 100Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Convergence graph

0 20 40 60 80 100Runtime (number of function evaluations)

0.0

0.2

0.4

0.6

0.8

1.0

Frac

tion

of ta

rget

s

Data profile

6

Data profile

Data profiles can aggregate performance over multiple runs

0 20 40 60 80 100Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Convergence graph

7

Data profile

Data profiles can aggregate performance over multiple runs

0 20 40 60 80 100Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Convergence graph

7

Data profile

Data profiles can aggregate performance over multiple runs

0 20 40 60 80 100Runtime (number of function evaluations)

Qual

ity in

dica

tor (

to b

e m

inim

ized)

Convergence graph

0 20 40 60 80 100Runtime (number of function evaluations)

0.0

0.2

0.4

0.6

0.8

1.0

Frac

tion

of ta

rget

s

Data profile

7

COCO test suites

Test suites and algorithm results

• bbob test suite with 24 functions (173 algorithms)• bbob-noisy test suite with 30 functions (45 algorithms)• bbob-biobj test suite with 55 functions (16 algorithms)

Algorithm results collected at 9 BBOB Workshops (since 2009, mostlyat GECCO conferences)

Under development

• Suite with constrained problems• Suite with large-scale problems• Suites with real-world problems

8