Definition: Run-Time Distribution (2)

Given OLVA A′ for optimisation problem Π′:

I The success probability Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q)is the probability that A′ finds a solution for a solubleinstance π′ ∈ Π′ of quality ≤ q in time ≤ t.

I The run-time distribution (RTD) of A′ on π′ is theprobability distribution of the bivariate random variable(RTA′,π′ ,SQA′,π′).

I The run-time distribution function rtd : R+ × R+ 7→ [0, 1],defined as rtd(t, q) = Ps(RTA,π ≤ t,SQA′,π′ ≤ q),completely characterises the RTD of A′ on π′.

Stochastic Local Search: Foundations and Applications 21

Definition: Run-Time Distribution (2)

Given OLVA A′ for optimisation problem Π′:

I The success probability Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q)is the probability that A′ finds a solution for a solubleinstance π′ ∈ Π′ of quality ≤ q in time ≤ t.

I The run-time distribution (RTD) of A′ on π′ is theprobability distribution of the bivariate random variable(RTA′,π′ ,SQA′,π′).

I The run-time distribution function rtd : R+ × R+ 7→ [0, 1],defined as rtd(t, q) = Ps(RTA,π ≤ t,SQA′,π′ ≤ q),completely characterises the RTD of A′ on π′.

Stochastic Local Search: Foundations and Applications 21

Definition: Run-Time Distribution (2)

Given OLVA A′ for optimisation problem Π′:

I The success probability Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q)is the probability that A′ finds a solution for a solubleinstance π′ ∈ Π′ of quality ≤ q in time ≤ t.

I The run-time distribution (RTD) of A′ on π′ is theprobability distribution of the bivariate random variable(RTA′,π′ ,SQA′,π′).

I The run-time distribution function rtd : R+ × R+ 7→ [0, 1],defined as rtd(t, q) = Ps(RTA,π ≤ t,SQA′,π′ ≤ q),completely characterises the RTD of A′ on π′.

Stochastic Local Search: Foundations and Applications 21

Typical run-time distribution for SLS algorithm applied tohard instance of combinatorial optimisation problem:

P(solve)

rel. soln.quality [%]

run-time [CPU sec]

10.80.60.40.2

0

2.52

1.51

0.50

0.11

10100

Stochastic Local Search: Foundations and Applications 22

Typical run-time distribution for SLS algorithm applied tohard instance of combinatorial optimisation problem:

P(solve)

rel. soln.quality [%]

run-time [CPU sec]

10.80.60.40.2

0

2.52

1.51

0.50

0.11

10100

Stochastic Local Search: Foundations and Applications 22

Typical run-time distribution for SLS algorithm applied tohard instance of combinatorial optimisation problem:

P(solve)

rel. soln.quality [%]

run-time [CPU sec]

10.80.60.40.2

0

2.52

1.51

0.50

0.11

10100

Stochastic Local Search: Foundations and Applications 22

Qualified RTDs for various solution qualities:

relative solution quality [%]

00

0.10.20.30.40.50.60.70.80.9

1

0.5 1 1.5 2 2.5

10s3.2s 1s0.3s0.1s

P(s

olve

)

run-time [CPU sec]

0.010

0.10.20.30.40.50.60.70.80.9

1

0.1 1 10 100 1 000

0.8%0.6%0.4%0.2%

opt

P(s

olve

)

Stochastic Local Search: Foundations and Applications 23

Qualified run-time distributions (QRTDs)

I A qualified run-time distribution (QRTD) of an OLVA A′

applied to a given problem instance π′ for solution quality q’is a marginal distribution of the bivariate RTD rtd(t, q)defined by:

qrtdq′(t) := rtd(t, q′) = Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q′).

I QRTDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I QRTDs characterise the ability of a given SLS algorithm fora combinatorial optimisation problem to solve the associateddecision problems.

Note: Solution qualities q are often expressed as relative solutionqualities q/q∗ − 1, where q∗ = optimal solution quality for givenproblem instance.

Stochastic Local Search: Foundations and Applications 24

Qualified run-time distributions (QRTDs)

I A qualified run-time distribution (QRTD) of an OLVA A′

applied to a given problem instance π′ for solution quality q’is a marginal distribution of the bivariate RTD rtd(t, q)defined by:

qrtdq′(t) := rtd(t, q′) = Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q′).

I QRTDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I QRTDs characterise the ability of a given SLS algorithm fora combinatorial optimisation problem to solve the associateddecision problems.

Note: Solution qualities q are often expressed as relative solutionqualities q/q∗ − 1, where q∗ = optimal solution quality for givenproblem instance.

Stochastic Local Search: Foundations and Applications 24

Qualified run-time distributions (QRTDs)

I A qualified run-time distribution (QRTD) of an OLVA A′

applied to a given problem instance π′ for solution quality q’is a marginal distribution of the bivariate RTD rtd(t, q)defined by:

qrtdq′(t) := rtd(t, q′) = Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q′).

I QRTDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I QRTDs characterise the ability of a given SLS algorithm fora combinatorial optimisation problem to solve the associateddecision problems.

Note: Solution qualities q are often expressed as relative solutionqualities q/q∗ − 1, where q∗ = optimal solution quality for givenproblem instance.

Stochastic Local Search: Foundations and Applications 24

Qualified run-time distributions (QRTDs)

I A qualified run-time distribution (QRTD) of an OLVA A′

applied to a given problem instance π′ for solution quality q’is a marginal distribution of the bivariate RTD rtd(t, q)defined by:

qrtdq′(t) := rtd(t, q′) = Ps(RTA′,π′ ≤ t,SQA′,π′ ≤ q′).

I QRTDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I QRTDs characterise the ability of a given SLS algorithm fora combinatorial optimisation problem to solve the associateddecision problems.

Note: Solution qualities q are often expressed as relative solutionqualities q/q∗ − 1, where q∗ = optimal solution quality for givenproblem instance.

Stochastic Local Search: Foundations and Applications 24

Typical solution quality distributions for SLS algorithm appliedto hard instance of combinatorial optimisation problem:

P(solve)

rel. soln.quality [%]

run-time [CPU sec]

10.80.60.40.2

0

2.52

1.51

0.50

0.11

10100

Stochastic Local Search: Foundations and Applications 25

Solution quality distributions for various run-times:

relative solution quality [%]

00

0.10.20.30.40.50.60.70.80.9

1

0.5 1 1.5 2 2.5

10s3.2s 1s0.3s0.1s

P(s

olve

)

run-time [CPU sec]

0.010

0.10.20.30.40.50.60.70.80.9

1

0.1 1 10 100 1 000

0.8%0.6%0.4%0.2%

opt

P(s

olve

)

Stochastic Local Search: Foundations and Applications 26

Solution quality distributions (SQDs)

I A solution quality distribution (SQD) of an OLVA A′ appliedto a given problem instance π′ for run-time t’ is a marginaldistribution of the bivariate RTD rtd(t, q) defined by:

sqdt′(q) := rtd(t ′, q) = Ps(RTA′,π′ ≤ t ′,SQA′,π′ ≤ q).

I SQDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I SQDs characterise the solution qualities achieved by agiven SLS algorithm for a combinatorial optimisation problemwithin a given run-time bound (useful for type 2 applicationscenarios).

Stochastic Local Search: Foundations and Applications 27

Solution quality distributions (SQDs)

I A solution quality distribution (SQD) of an OLVA A′ appliedto a given problem instance π′ for run-time t’ is a marginaldistribution of the bivariate RTD rtd(t, q) defined by:

sqdt′(q) := rtd(t ′, q) = Ps(RTA′,π′ ≤ t ′,SQA′,π′ ≤ q).

I SQDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I SQDs characterise the solution qualities achieved by agiven SLS algorithm for a combinatorial optimisation problemwithin a given run-time bound (useful for type 2 applicationscenarios).

Stochastic Local Search: Foundations and Applications 27

Solution quality distributions (SQDs)

I A solution quality distribution (SQD) of an OLVA A′ appliedto a given problem instance π′ for run-time t’ is a marginaldistribution of the bivariate RTD rtd(t, q) defined by:

sqdt′(q) := rtd(t ′, q) = Ps(RTA′,π′ ≤ t ′,SQA′,π′ ≤ q).

I SQDs correspond to cross-sections of the two-dimensionalbivariarate RTD graph.

I SQDs characterise the solution qualities achieved by agiven SLS algorithm for a combinatorial optimisation problemwithin a given run-time bound (useful for type 2 applicationscenarios).

Stochastic Local Search: Foundations and Applications 27

Note:

I For sufficiently long run-times, increase in mean solutionquality is often accompanied by decrease in solution qualityvariability.

I For PAC algorithms, the SQDs for very large time-limits t ′

approach degenerate distributions that concentrate allprobability on the optimal solution quality.

I For any essentially incomplete algorithm A′ (such as IterativeImprovement) applied to a problem instance π′, the SQDs forsufficiently large time-limits t ′ approach a non-degeneratedistribution called the asymptotic SQD of A′ on π′.

Stochastic Local Search: Foundations and Applications 28

Note:

I For sufficiently long run-times, increase in mean solutionquality is often accompanied by decrease in solution qualityvariability.

I For PAC algorithms, the SQDs for very large time-limits t ′

approach degenerate distributions that concentrate allprobability on the optimal solution quality.

I For any essentially incomplete algorithm A′ (such as IterativeImprovement) applied to a problem instance π′, the SQDs forsufficiently large time-limits t ′ approach a non-degeneratedistribution called the asymptotic SQD of A′ on π′.

Stochastic Local Search: Foundations and Applications 28

Note:

I For sufficiently long run-times, increase in mean solutionquality is often accompanied by decrease in solution qualityvariability.

I For PAC algorithms, the SQDs for very large time-limits t ′

approach degenerate distributions that concentrate allprobability on the optimal solution quality.

I For any essentially incomplete algorithm A′ (such as IterativeImprovement) applied to a problem instance π′, the SQDs forsufficiently large time-limits t ′ approach a non-degeneratedistribution called the asymptotic SQD of A′ on π′.

Stochastic Local Search: Foundations and Applications 28

Solution quality statistic over time (SQTs)

I The development of solution quality over the run-time ofa given OLVA is reflected in time-dependent SQD statistics(solution quality over time (SQT) curves).

I SQT curves based on SQD quantiles (such as median solutionquality) correspond to contour lines of the two-dimensionalbivariarate RTD graph.

I SQT curves are widely used to illustrate the trade-off betweenrun-time and solution quality for a given OLVA.

I But: Important aspects of an algorithm’s run-time behaviourmay be easily missed when basing an analysis solely ona single SQT curve.

Stochastic Local Search: Foundations and Applications 29

Solution quality statistic over time (SQTs)

I The development of solution quality over the run-time ofa given OLVA is reflected in time-dependent SQD statistics(solution quality over time (SQT) curves).

I SQT curves based on SQD quantiles (such as median solutionquality) correspond to contour lines of the two-dimensionalbivariarate RTD graph.

I SQT curves are widely used to illustrate the trade-off betweenrun-time and solution quality for a given OLVA.

I But: Important aspects of an algorithm’s run-time behaviourmay be easily missed when basing an analysis solely ona single SQT curve.

Stochastic Local Search: Foundations and Applications 29

Solution quality statistic over time (SQTs)

I The development of solution quality over the run-time ofa given OLVA is reflected in time-dependent SQD statistics(solution quality over time (SQT) curves).

I SQT curves based on SQD quantiles (such as median solutionquality) correspond to contour lines of the two-dimensionalbivariarate RTD graph.

I SQT curves are widely used to illustrate the trade-off betweenrun-time and solution quality for a given OLVA.

I But: Important aspects of an algorithm’s run-time behaviourmay be easily missed when basing an analysis solely ona single SQT curve.

Stochastic Local Search: Foundations and Applications 29

Solution quality statistic over time (SQTs)

I The development of solution quality over the run-time ofa given OLVA is reflected in time-dependent SQD statistics(solution quality over time (SQT) curves).

I SQT curves based on SQD quantiles (such as median solutionquality) correspond to contour lines of the two-dimensionalbivariarate RTD graph.

I SQT curves are widely used to illustrate the trade-off betweenrun-time and solution quality for a given OLVA.

I But: Important aspects of an algorithm’s run-time behaviourmay be easily missed when basing an analysis solely ona single SQT curve.

Stochastic Local Search: Foundations and Applications 29

Typical SQT curves for SLS optimisation algorithms applied toinstance of hard combinatorial optimisation problem:

P(solve)

rel. soln.quality [%]

run-time [CPU sec]

10.80.60.40.2

0

2.52

1.51

0.50

0.11

10100

Stochastic Local Search: Foundations and Applications 30

Typical SQT curves for SLS optimisation algorithms applied toinstance of hard combinatorial optimisation problem:

100

10

1

0.10 0.1 0.2 0.3 0.4 0.5 0.6 0.7

run-

time

[CP

U s

ec]

relative solution quality [%]

0.9 quantile0.75 quantile

median

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.1 1 10 100

rela

tive

solu

tion

qual

ity [%

]

run-time [CPU sec]

0.75 quantile0.9 quantile

median

Stochastic Local Search: Foundations and Applications 31

Note:

I The fraction of successful tuns, sr := k ′/k, is called thesuccess ratio; for large run-times t ′, it approximates theasymptotic success probability p∗s := limt→∞Ps(RTa,π ≤ t).

I In cases where the success ratio sr for a given cutoff time t ′

is smaller than 1, quantiles up to sr can still be estimated fromthe respective truncated RTD.

The mean run-time for a variant of the algorithm that restartsafter time t ′ can be estimated as:

E (RTs) + (1/sr − 1) · E (RTf )

where E (RTs) and E (RTf ) are the average times of successfuland failed runs, respectively.

Note: 1/sr − 1 is the expected number of failed runs required before

a successful run is observed.

Stochastic Local Search: Foundations and Applications 36

Note:

I The fraction of successful tuns, sr := k ′/k, is called thesuccess ratio; for large run-times t ′, it approximates theasymptotic success probability p∗s := limt→∞Ps(RTa,π ≤ t).

I In cases where the success ratio sr for a given cutoff time t ′

is smaller than 1, quantiles up to sr can still be estimated fromthe respective truncated RTD.

The mean run-time for a variant of the algorithm that restartsafter time t ′ can be estimated as:

E (RTs) + (1/sr − 1) · E (RTf )

where E (RTs) and E (RTf ) are the average times of successfuland failed runs, respectively.

Note: 1/sr − 1 is the expected number of failed runs required before

a successful run is observed.

Stochastic Local Search: Foundations and Applications 36

Note:

I The fraction of successful tuns, sr := k ′/k, is called thesuccess ratio; for large run-times t ′, it approximates theasymptotic success probability p∗s := limt→∞Ps(RTa,π ≤ t).

I In cases where the success ratio sr for a given cutoff time t ′

is smaller than 1, quantiles up to sr can still be estimated fromthe respective truncated RTD.

The mean run-time for a variant of the algorithm that restartsafter time t ′ can be estimated as:

E (RTs) + (1/sr − 1) · E (RTf )

where E (RTs) and E (RTf ) are the average times of successfuland failed runs, respectively.

Note: 1/sr − 1 is the expected number of failed runs required before

a successful run is observed.

Stochastic Local Search: Foundations and Applications 36

Protocol for obtaining the empirical RTD for an OLVA A′

applied to a given instance π′ of an optimisation problem:

I Perform k independent runs of A′ on π′ with cutoff time t ′.

I During each run, whenever the incumbent solution isimproved, record the quality of the improved incumbentsolution and the time at which the improvement was achievedin a solution quality trace.

I Let sq(t ′, j) denote the best solution quality encountered inrun j up to time t ′. The cumulative empirical RTD of A′ on π′

is defined by Ps(RT ≤ t ′,SQ ≤ q′) := #{j | sq(t ′, j) ≤ q′}/k.

Note: Qualified RTDs, SQDs and SQT curves can be easilyderived from the same solution quality traces.

Stochastic Local Search: Foundations and Applications 37

Protocol for obtaining the empirical RTD for an OLVA A′

applied to a given instance π′ of an optimisation problem:

I Perform k independent runs of A′ on π′ with cutoff time t ′.

I During each run, whenever the incumbent solution isimproved, record the quality of the improved incumbentsolution and the time at which the improvement was achievedin a solution quality trace.

I Let sq(t ′, j) denote the best solution quality encountered inrun j up to time t ′. The cumulative empirical RTD of A′ on π′

is defined by Ps(RT ≤ t ′,SQ ≤ q′) := #{j | sq(t ′, j) ≤ q′}/k.

Note: Qualified RTDs, SQDs and SQT curves can be easilyderived from the same solution quality traces.

Stochastic Local Search: Foundations and Applications 37

Protocol for obtaining the empirical RTD for an OLVA A′

applied to a given instance π′ of an optimisation problem:

I Perform k independent runs of A′ on π′ with cutoff time t ′.

I During each run, whenever the incumbent solution isimproved, record the quality of the improved incumbentsolution and the time at which the improvement was achievedin a solution quality trace.

I Let sq(t ′, j) denote the best solution quality encountered inrun j up to time t ′. The cumulative empirical RTD of A′ on π′

is defined by Ps(RT ≤ t ′,SQ ≤ q′) := #{j | sq(t ′, j) ≤ q′}/k.

Note: Qualified RTDs, SQDs and SQT curves can be easilyderived from the same solution quality traces.

Stochastic Local Search: Foundations and Applications 37