Introduction Metrics BiMADS Other methods References
Multiobjective Optimization
MTH8418
S. Le Digabel, Polytechnique Montreal
Winter 2019(v4)
MTH8418: Multiobjective 1/37
Introduction Metrics BiMADS Other methods References
Plan
Introduction
Metrics
BiMADS
Other methods
References
MTH8418: Multiobjective 2/37
Introduction Metrics BiMADS Other methods References
Introduction
Metrics
BiMADS
Other methods
References
MTH8418: Multiobjective 3/37
Introduction Metrics BiMADS Other methods References
Multiobjective optimization problemI The multiobjective optimization problem (MOP) can be
formally stated asminx∈Ω
F (x)
I whereF : Ω→ R ∪ +∞p
andF (x) =
(f (1)(x), f (2)(x), . . . , f (p)(x)
)I p is the number of objective functions
I Case p = 2: Biobjective optimization problem (BOP)
I The feasible set Ω remains unchanged
I Typically, the different objectives are contradictory: A decreasein one objective causes an increase in the other objectives
MTH8418: Multiobjective 4/37
Introduction Metrics BiMADS Other methods References
Difficulty
I Single-objective optimization: Particular case where p = 1.An optimal solution typically consists of a single vector x ∈ Ω
I Multiobjective optimization: There is usually no such vectorthat simultaneously minimizes all of the p ≥ 2 objectivefunctions
I The solution consists of a set of trade-off solutions in Ω, thePareto solutions
I The methods presented in this lesson construct anapproximation to this set
MTH8418: Multiobjective 5/37
Introduction Metrics BiMADS Other methods References
Pareto notionI Single-objective: u, v ∈ Ω can be trivially ranked by
comparing f(u) and f(v)
I Generalization with p > 1:I u dominates v, denoted u ≺ v, if and only if F (u) ≤ F (v) and
f (q)(u) < f (q)(v) for at least one index q in 1, 2, . . . , pI u is indifferent to v, denoted u ∼ v, if and only if u does not
dominate v, and v does not dominate u
I A point u ∈ Ω is Pareto optimal if and only if there is now ∈ Ω such that w ≺ u
I The set of Pareto optimal solutions is the Pareto set ΩP
I The image of ΩP under the mapping F defines the solution tothe problem and is called the Pareto front FP ⊆ Rp
MTH8418: Multiobjective 6/37
Introduction Metrics BiMADS Other methods References
Pareto front example
Feasible region : Ω ⊂ R3
6f (2)
-f (1)
cx1∈Ω.
...........................................
...........................................
...........................................
..........................................
..........................................
..........................................
..................................................................................
......................................... ........................................ .......................................... ..............................................................................
............ cF (x1)c
x2∈Ω.
....................................................
...................................................
..................................................
..................................................
.................................................
.................................................
................................................................................................
............................................... .............................................. ............................................... ................................................................................................
...
................................................... c
F (x2)
sx3∈Ω.
......................................
..................................... ................................... .................................. .................................. ......................................................................
....................................
.....................................
.....................................
......................................
sF (x3)
sx4∈Ω.
...............................................................................
............................................................................ ......................................................................... ...................................................................... .........................................................................................................................................
..................................................................
.................................................................
................................................................
...............................................................
..............................................................
sF (x4)
F (Ω)
.
..........................
.......................
......................
..........................................
.................... .................... .................... .............................................
.
.........................
...........................
........................
....
......................
........
................................
.......................
....................
..................
.................
........
.......
........
......
..............
..............
...............
..................
.....................
..............................................................................
...........................................................................
.....................
...................
..................
..................
..............................................................................................
...........................................................................................
...........................................................................................
.................
..........
.......................
.....................
................... ................. ................ ..............
.................
..........
.......................
.....................
................... ................. ............... ..............
.................
..........
.......................
.....................
................... ................. ............... ..............
Image of Ω in objective space R2
Dominance zonefor F (x1)
MTH8418: Multiobjective 7/37
Introduction Metrics BiMADS Other methods References
Individual minima
The individual minima of F are the solutions to thesingle-objective optimization problems
minx∈Ω
f (q)(x) , for q ∈ 1, 2, . . . p
MTH8418: Multiobjective 8/37
Introduction Metrics BiMADS Other methods References
How to choose one solution?
I Can be done visually with p = 2 and some knowledge of theproblem. Large and small slopes should be identified.
I For p ≥ 2, engineers use carpet plots.
I More generally, this is the subject of multicriteria optimization.
MTH8418: Multiobjective 9/37
Introduction Metrics BiMADS Other methods References
The ε-constraint method
I The most commonly used method.
I It transforms objectives into constraints: The original problemwith p objectives becomes a problem with one objective andp− 1 constraints.
I Then, change the bounds on the constraints in order to graspthe Pareto front.
MTH8418: Multiobjective 10/37
Introduction Metrics BiMADS Other methods References
Weighted sums of objectives for BOP (p = 2)Natural single-objective reformulation: Solve
minx∈Ω
αf (1)(x) + (1− α)f (2)(x) (1)
Inconvenient: Some regions of the Pareto front are never optimalfor (1), regardless of α
MTH8418: Multiobjective 11/37
Introduction Metrics BiMADS Other methods References
Application: Constraint sensitivity analysis
I Biobjective optimization can be used in order to conductsensitivity analyses relative to constraints
I The constraint of interest is transformed as an objectivefunction
I The analysis of the approximated Pareto front allows tointerpret the impact of this constraint on the original objective
I Two different tools are available within NOMAD:I A post-optimization analysis. Cheap and rough approximation
of the sensitivities
I An additional biobjective execution. More expensive, but givesa good approximation of the sensitivities
MTH8418: Multiobjective 12/37
Introduction Metrics BiMADS Other methods References
Sensitivity analysis: Example
−0.1 −0.05 0 0.05 0.125.8
25.9
26
26.1
26.2
26.3
26.4
26.5
Sensitivity to x1−2 ) 0
constraint value
obje
ctiv
e fu
nctio
n va
lue
MTH8418: Multiobjective 13/37
Introduction Metrics BiMADS Other methods References
Introduction
Metrics
BiMADS
Other methods
References
MTH8418: Multiobjective 14/37
Introduction Metrics BiMADS Other methods References
Metrics
I [Audet et al., 2018]
I How to compare the approximations to the Pareto frontobtained by different solvers?
I S: Set of solvers; P: Set of problems
I To draw performance and data profiles, we need aperformance measure tp,s > 0 for each p ∈ P and s ∈ S
I Fp,s: Approximated Pareto front determined by the solvers ∈ S for problem p ∈ P
I Fp: Approximated Pareto front for problem p. Obtained by∪s∈SFp,s and by removing the dominated points
MTH8418: Multiobjective 15/37
Introduction Metrics BiMADS Other methods References
Purity metric
I Purity metric:
purityp,s =|Fp,s ∩ Fp||Fp,s|
∈ [0; 1]
I The higher the better
I Take tp,s = 1/purityp,s if purityp,s 6= 0, +∞ otherwise
I Problem: The purity is equal to one (i.e. perfect) for a solverthat gives only one non-dominated solution
MTH8418: Multiobjective 16/37
Introduction Metrics BiMADS Other methods References
Largest holeI These measures compute the spread of an approximated
Pareto front with the maximum size of the “holes” in thefront. We need |Fp,s| > 1
I tp,s = Γp,s = maxq∈1,2,...,p
(max
i∈1,2,...,|Fp,s|
δ
(q)i
)where δ
(q)i
represents the distance between the ith point of Fp,s and itsclosest neighbor, in terms of f (q)
I HRS (Hole Relative Size): tp,s = maxi∈1,2,...,|Fp,s|
di/d
where
di represents the distance between the ith point of Fp,s and
its closest neighbor, and d =∑|Fp,s|
i=1 di/|Fp,s|
I Standard deviation: tp,s =
√√√√ |Fp,s|∑i=1
(di−d)2
|Fp,s|−1
MTH8418: Multiobjective 17/37
Introduction Metrics BiMADS Other methods References
Progress measures (1/2)
I These measures are focused on the convergence of themethods. Useful for plotting simplified data profiles
I Progress for objective q ∈ 1, 2, . . . , p at evaluation k:
prog(q)k = log
√f(q)1
f(q)k
where f(q)k represents the best value
obtained after the kth evaluation, in terms of f (q)
I We need feasible starting solutions, and all objective valuesneed to be > 0
I We could consider tp,s = maxq∈1,2,...,p
prog(q)k for s and p
MTH8418: Multiobjective 18/37
Introduction Metrics BiMADS Other methods References
Progress measures (2/2)
I Number of non-dominated points at each evaluation: For kthe number of evaluation or a group of n+ 1 evaluations,consider |Fp,s|
I Or consider the number of new non-dominated pointsbetween two values of k
I Number of waves: Consider all the solutions produced bysolver s on problem p. Recursively remove the non-dominatedpoints, and W is the number of times that this operation isnecessary to consider all the points. The more W is close to1, the better is s
MTH8418: Multiobjective 19/37
Introduction Metrics BiMADS Other methods References
Generational Distance (GD)
I Measures a distance between Fp,s and Fp
I GDp,s =
√∑|Fp,s|i=1 d2i,p|Fp,s|
I di,p represents the distance between the ith point in Fp,s andthe closest point of Fp
I The standard deviation of the GD measures the deformationof the front obtained by s ∈ S compared to the global
approximation: STDGDp,s =∑|Fp,s|
i=1 (di,p−GDp,s)2
|Fp,s|−1
I Maximum Pareto Front Error: MEp,s = maxi∈1,2,...,|Fp,s|
di,p
MTH8418: Multiobjective 20/37
Introduction Metrics BiMADS Other methods References
Hypersurface
I Consider tp,s = HSp,s =Sp,s
Sp
I Sp,s represents the surface under the plot of Fp,s and Sp thesurface under the plot of Fp
I Not easy to generalize/compute for p > 2
MTH8418: Multiobjective 21/37
Introduction Metrics BiMADS Other methods References
Introduction
Metrics
BiMADS
Other methods
References
MTH8418: Multiobjective 22/37
Introduction Metrics BiMADS Other methods References
BiMADS: Series of single-optimization executionsI [Audet et al., 2008]
I Based on a single-objective optimization algorithm: MADS
I MADS is launched on a series of subproblems
I Constraints are handled by MADS with EB/PB/PEBtechniques
I Each subproblem is obtained by a single-objectivereformulation that is not based on weights
I The solutions of each of these subproblems produces a localapproximation of the Pareto set
I The set of undominated solutions produces an approximationof the entire Pareto set
I BiMADS is implemented in NOMAD
MTH8418: Multiobjective 23/37
Introduction Metrics BiMADS Other methods References
I Reference point in the objective space: r ∈ R2
I Reformulated objective:
φr(F (x)) :=
−
p∏q=1
(rq − f (q)(x))2 if F (x) ≤ r,p∑q=1
((f (q)(x)− rq)+
)2otherwise
I When minimized on x ∈ Ω, starting from “F−1(r)”, itpotentially generates a solution that dominates r
6f (2)
-f (1)
...........................................
..............
φr<0
φr<0
φr=0
φr>0
r. .......................... ........................ ...................... ..................... ...................
.................
...............
...............
................
.................
........
........
..
. ........................ ...................... .................... .................. ...........................
..............
..............
........
......
6f (2)
-f (1)
......................
..................
...................................
................. ................ .................. .....................................
....
.......................
........................
.
....................
......
....................
.................
...............
.........
....
............
............
.............
...............
...............................................................
..................................................
...................
.................
................
.............................................................
.....................
..................
................
.............. ............ ...........
.............................................................
.............................................................
.................
.....................
................. .............. ............. ...........
.................
.....................
................. .............. ............ ...........
. ............................. .......................... ........................ ..................... ................... ................ ............. .........................................
r
I Every Pareto solution can be obtained with some r.
MTH8418: Multiobjective 24/37
Introduction Metrics BiMADS Other methods References
I Reference point in the objective space: r ∈ R2
I Reformulated objective:
φr(F (x)) :=
−
p∏q=1
(rq − f (q)(x))2 if F (x) ≤ r,p∑q=1
((f (q)(x)− rq)+
)2otherwise
I When minimized on x ∈ Ω, starting from “F−1(r)”, itpotentially generates a solution that dominates r
6f (2)
-f (1)
...........................................
..............
φr<0
φr<0
φr=0
φr>0
r. .......................... ........................ ...................... ..................... ...................
.................
...............
...............
................
.................
........
........
..
. ........................ ...................... .................... .................. ...........................
..............
..............
........
......
6f (2)
-f (1)
......................
..................
...................................
................. ................ .................. .....................................
....
.......................
........................
.
....................
......
....................
.................
...............
.........
....
............
............
.............
...............
...............................................................
..................................................
...................
.................
................
.............................................................
.....................
..................
................
.............. ............ ...........
.............................................................
.............................................................
.................
.....................
................. .............. ............. ...........
.................
.....................
................. .............. ............ ...........
. ............................. .......................... ........................ ..................... ................... ................ ............. .........................................
r
I Every Pareto solution can be obtained with some r.
MTH8418: Multiobjective 24/37
Introduction Metrics BiMADS Other methods References
I Reference point in the objective space: r ∈ R2
I Reformulated objective:
φr(F (x)) :=
−
p∏q=1
(rq − f (q)(x))2 if F (x) ≤ r,p∑q=1
((f (q)(x)− rq)+
)2otherwise
I When minimized on x ∈ Ω, starting from “F−1(r)”, itpotentially generates a solution that dominates r
6f (2)
-f (1)
...........................................
..............
φr<0
φr<0
φr=0
φr>0
r. .......................... ........................ ...................... ..................... ...................
.................
...............
...............
................
.................
........
........
..
. ........................ ...................... .................... .................. ...........................
..............
..............
........
......
6f (2)
-f (1)
......................
..................
...................................
................. ................ .................. .....................................
....
.......................
........................
.
....................
......
....................
.................
...............
.........
....
............
............
.............
...............
...............................................................
..................................................
...................
.................
................
.............................................................
.....................
..................
................
.............. ............ ...........
.............................................................
.............................................................
.................
.....................
................. .............. ............. ...........
.................
.....................
................. .............. ............ ...........
. ............................. .......................... ........................ ..................... ................... ................ ............. .........................................
r
I Every Pareto solution can be obtained with some r.
MTH8418: Multiobjective 24/37
Introduction Metrics BiMADS Other methods References
Reference point selection
I Use the ordering propertyinherent to p = 2 to computegaps between 3 successiveundominated solutions in theobjective space
I Choose r with the largest gap
I Associate a weight to r todecrease the probability ofchoosing it again and preventstalling when the Pareto front isdiscontinuous
MTH8418: Multiobjective 25/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs
6f (2)
-
f (1)
c
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs
6f (2)
-
f (1)
cs cc c c
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs
6f (2)
-
f (1)
cs cc c ccc
c cccs
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs
6f (2)
-
f (1)
cs cc c ccc
c cccs
s s s ss s s
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
I Main iterations:I Reference point determination:
Use the set of feasible ordered undominated points generatedso far to generate a reference point r
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs6f (2)
-
f (1)
cs cc c ccc
c cccs
s s s ss s s
s s s ss s s
r. ............................... ............................. ........................... ......................... ....................... ..................... ...................
.................
..................
...................
....................
.............
........
. ............................. .......................... ........................ ..................... ................... ................. ..............................................................
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
I Main iterations:I Reference point determination:
Use the set of feasible ordered undominated points generatedso far to generate a reference point r
I Single-objective minimization:Solve the subproblem min
x∈Ωφr(F (x))
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs6f (2)
-
f (1)
cs s s ss s s
s s s ss s s
r. ............................... ............................. ........................... ......................... ....................... ..................... ...................
.................
..................
...................
....................
.............
........
. ............................. .......................... ........................ ..................... ................... ................. ..............................................................
c csc ccc c cI Initialization:
Solve minx∈Ω
f (q)(x) for q ∈ 1, 2I Main iterations:
I Reference point determination:Use the set of feasible ordered undominated points generatedso far to generate a reference point r
I Single-objective minimization:Solve the subproblem min
x∈Ωφr(F (x))
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs6f (2)
-
f (1)
cs cs cs ssccccc c cc
cc
s cscs
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
I Main iterations:I Reference point determination:
Use the set of feasible ordered undominated points generatedso far to generate a reference point r
I Single-objective minimization:Solve the subproblem min
x∈Ωφr(F (x))
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
BiMADS: successive MADS runs6f (2)
-
f (1)
cs cs cs ssccccc c cc
cc
s cscsr. .................................... ................................. ............................... ............................. .......................... ........................ ..................... ................... ................ .............. ............. .............
.............. ................................. .............................. ............................ ......................... ...................... ................... ................. .............. ........... ..............
I Initialization:Solve min
x∈Ωf (q)(x) for q ∈ 1, 2
I Main iterations:I Reference point determination:
Use the set of feasible ordered undominated points generatedso far to generate a reference point r
I Single-objective minimization:Solve the subproblem min
x∈Ωφr(F (x))
MTH8418: Multiobjective 26/37
Introduction Metrics BiMADS Other methods References
MultiMADSI [Audet et al., 2010]I Based on the natural boundary intersection (NBI) framework,
and the convex hull of individual minima
I Consider the simplex g(1), g(2), g(3) obtained from theindividual minima
MTH8418: Multiobjective 27/37
Introduction Metrics BiMADS Other methods References
MultiMADS: Example of solution
MTH8418: Multiobjective 28/37
Introduction Metrics BiMADS Other methods References
Convergence analysisI MADS solves the single objective subproblems
I These solutions x are such that if φr(F (.)) is Lipschitz near x,then φr(F (x); d) ≥ 0 for every direction d in the hypertangentcone THΩ (x) to the domain Ω at x
I Every Pareto point is the optimal solution of a reformulation
I Let x ∈ Ω be a refining point produced by MADS on asingle-objective subproblem for some r ∈ Rp. If F is Lipschitznear x, then for any direction d ∈ THΩ (x), there exists a
q ∈ 1, 2, . . . , p such that(f (q)
)(x; d) ≥ 0
I When the functions are regular, it means that moving in afeasible direction deteriorates at least one objective: This is atradeoff solution (Pareto)
MTH8418: Multiobjective 29/37
Introduction Metrics BiMADS Other methods References
Introduction
Metrics
BiMADS
Other methods
References
MTH8418: Multiobjective 30/37
Introduction Metrics BiMADS Other methods References
Direct Multisearch (DMS)I [Custodio et al., 2011]
I Native adaptation of GPS to the unconstrained multiobjectivecase (p ≥ 2 and use of the EB)
I Intensification with a poll step in which the acceptationcriteria are based on the Pareto dominance
I Diversification with a search step
I Convergence based on the Clarke derivatives
I Differences with biMADS and multiMADS:I BiMADS is a framework using MADS in a subproblem while
DMS is a native multiobjective methodI At each step, DMS tries to improve the entire front, while
biMADS focuses on a specific part of it
MTH8418: Multiobjective 31/37
Introduction Metrics BiMADS Other methods References
NSGA-III NSGA-II: Non-dominated Sorting Genetic Algorithm, for BOP
(p ≥ 2) [Deb et al., 2002]
I Constraints are treated with the inclusion of the violation inthe dominance relation
I Each objective parameter is treated separately
I Mutation and crossover are performed on the population
I Selection based on “non-dominated sorting” (intensification),and “crowded-distance sorting” (diversification)
I Heuristic: No guarantee on the quality of the approximatedPareto front
I From the same team: Archive-based Micro Genetic Algorithm(AMGA) [Tiwari et al., 2008]
MTH8418: Multiobjective 32/37
Introduction Metrics BiMADS Other methods References
Multiobjective solversI NOMAD (p = 2)
I NSGA-II: Several implementations can be found:I MATLAB version
I C versions
I AMGA2 [Tiwari et al., 2011]
I DMS: MATLAB version (by email)
I DFL toolbox:I DFMO: Linesearch, constraints, FORTRANI MODIR: DIRECT, constraints, FORTRANI MOIF: Implicit filtering, bounds, MATLAB
MTH8418: Multiobjective 33/37
Introduction Metrics BiMADS Other methods References
Introduction
Metrics
BiMADS
Other methods
References
MTH8418: Multiobjective 34/37
Introduction Metrics BiMADS Other methods References
References I
Audet, C., Bigeon, J., Cartier, D., Le Digabel, S., and Salomon, L. (2018).
Performance indicators in multiobjective optimization.Technical Report G-2018-90, Les cahiers du GERAD.
Audet, C., Dennis, Jr., J., and Le Digabel, S. (2012).
Trade-off studies in blackbox optimization.Optimization Methods and Software, 27(4–5):613–624.(Constraint Sensitivity Analysis).
Audet, C., Savard, G., and Zghal, W. (2008).
Multiobjective Optimization Through a Series of Single-Objective Formulations.SIAM Journal on Optimization, 19(1):188–210.(biMADS).
Audet, C., Savard, G., and Zghal, W. (2010).
A mesh adaptive direct search algorithm for multiobjective optimization.European Journal of Operational Research, 204(3):545–556.(multiMADS).
Collette, Y. and Siarry, P. (2002).
Optimisation multiobjectif.Eyrolles.(metrics).
MTH8418: Multiobjective 35/37
Introduction Metrics BiMADS Other methods References
References II
Custodio, A., Madeira, J., Vaz, A., and Vicente, L. (2011).
Direct multisearch for multiobjective optimization.SIAM Journal on Optimization, 21(3):1109–1140.(DMS, metrics, profiles).
Das, I. and Dennis, Jr., J. (1998).
Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear MulticriteriaOptimization Problems.SIAM Journal on Optimization, 8(3):631–657.(NBI).
Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. (2002).
A fast and elitist multiobjective genetic algorithm: NSGA-II.IEEE Transactions on Evolutionary Computation, 6(2):182–197.
E.F.Campana, M.Diez, G.Liuzzi, S.Lucidi, R.Pellegrini, V.Piccialli, F.Rinaldi, and A.Serani (2018).
A Multi-objective DIRECT algorithm for ship hull optimization.Computational Optimization and Applications, 71(1):53–72.(MODIR).
G.Cocchi, G.Liuzzi, A.Papini, and M.Sciandrone (2018).
An implicit filtering algorithm for derivative-free multiobjective optimization with box constraints.Computational Optimization and Applications, 69(2):267–296.(MOIF).
MTH8418: Multiobjective 36/37
Introduction Metrics BiMADS Other methods References
References III
Laumanns, M., Zitzler, E., and Thiele, L. (2000).
A Unified Model for Multi-Objective Evolutionary Algorithms with Elitism.In Congress on Evolutionary Computation, volume 1, pages 46–53, Piscataway, New Jersey, USA.(metrics).
Liuzzi, G., Lucidi, S., and Rinaldi, F. (2016).
A Derivative-Free Approach to Constrained Multiobjective Nonsmooth Optimization.SIAM Journal on Optimization, 26(4):2744–2774.(DFMO).
Tiwari, S., Fadel, G., and Deb, K. (2011).
AMGA2: improving the performance of the archive-based micro-genetic algorithm for multi-objectiveoptimization.Engineering Optimization, 43(4):377–401.
Tiwari, S., Koch, P., Fadel, G., and Deb, K. (2008).
AMGA: An Archive-based Micro Genetic Algorithm for Multi-objective Optimization.In Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, GECCO ’08,pages 729–736, New York, NY, USA. ACM.
Zitzler, E., Deb, K., and Thiele, L. (2000).
Comparison of Multiobjective Evolutionary Algorithms: Empirical Results.Evolutionary Computation, 8(2):173–195.(metrics).
MTH8418: Multiobjective 37/37