Automating estimation of warm-up length

Automating estimation of warm-up length

Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School

Simulation Workshop - April 2008

The AutoSimOA ProjectA 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.

http://www.wbs.ac.uk/go/autosimoa

Research Aim

• To create an automated system for dealing with the problem of initial bias, for implementation into simulation software.

• Target audience: non- (statistically) expert simulation users.

The Initial Bias Problem

• Model may not start in a “typical” state.• Can cause initial bias in the output.• Many methods proposed for dealing with

initial bias: e.g. Initial steady state conditions; run model for ‘long’ time…

• This project uses: Deletion of the initial transient data by specifying a warm-up period (or truncation point).

Question is:

How do you estimate the length of the warm-up period

required?

Methods fall into 5 main types :

1. Graphical Methods.

2. Heuristic Approaches.

3. Statistical Methods.

4. Initialisation Bias Tests.

5. Hybrid Methods.

Literature search – 42 methods

Summary of methods and

literature references on project

web site:

http://www.wbs.ac.uk/go/autosimoa

Short-listing warm-up methods for automation using literature

Short-listing Criteria

• Accuracy & robustness

• Simplicity

• Ease of automation

• Generality

• Number of parameters to estimate

• Computer running time

11

2 22

66 4

3

0

2

4

6

8

10

12

Graphical Statistical Heuristic

Warm-up method type

Nu

mb

er

of

me

tho

ds

Ease of AutomationAccuracyGeneralityRunning timeOther

Short-listing results: reasons for rejection of methods

• Statistical methods:– Goodness of Fit (GoF) test– Algorithm for a static data set (ASD)– Algorithm for a Dynamic data set (ADD)

• Heuristics:– MSER-5– Kimbler’s Double Exponential Smoothing– Euclidean Distance Method (ED)

Short-listing results: 6 Methods taken forward to testing

Testing Procedure

Test short-listed methods using:

1. Artificial data – controllable & comparable initial bias functions steady state functions

2. Set of performance criteria.

i) Length – proportion of data length.

ii) Severity – maximum bias value is a function of the difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data.

iii) Shape and Orientation – 7 shapes:

Initial bias functions - 3 Criteria:

1. Artificial Data Sets

• Mean Shift:

• Linear:

• Quadratic:

• Exponential:

• Oscillating (decreasing):

Quadratic ExponentialLinear

i) Constant steady state variance

ii) Error Terms: Normal or Exponential distribution

iii) Auto-Correlation:

No AutoCorrelation; AR(1); AR(2); AR(4); MA(2); ARMA(5,5).

Add Initial Bias to Steady state:

Superpostion: Bias Fn, a(t), added onto end of steady state function:

e.g. ...

)(1

etc

taXY

XX

tt

ttt

Steady state functions - 3 Criteria:

• Closeness of estimated truncation point (Lsol) to true truncation point (L).

• Coverage of true mean

• ½ width of 95% CI for average truncated mean.

• Bias and absolute bias in estimated mean.

• Number of failures of method.

2. Performance Criteria

Test Results• Rejections:

– ASD & ADD required a prohibitively large number of replications

– GoF & Kimbler’s method consistently severely underestimated truncation point.

– ED failed to give any result on majority of occasions

• MSER-5 most accurate and robust method.

MSER-5 Method

0

0.02

0.04

0.06

0.08

0.1

0.12

0 20

40

60

80

100

120

140

160

180

Batch number

sta

nd

ard

err

or

-4

-2

0

2

4

6

8

10

ba

tch

me

an

s

Standard Error

BatchMean

MSER-5 test statisticOutput data

(batched)

MSER-5 ResultsMSER5 result eg.xls

Does the true mean fall into the 95% CI for the estimated mean?

Non-truncated data sets

Truncated data sets

% of cases

yes yes 8.3%

yes no 0%

no no 19.3%

no yes 72.4%

Summary / Future Work

• 42 warm-up methods

• Short-listing and testing

• MSER-5 most promising method for automation

• Creation of heuristic framework around MSER-5 method for implementation into simulation software.

ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output

Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by

the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in

collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.

Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School

SW08

ii) SEVERITY OF BIAS FUNCTION

Set maximum value of bias fn, a(t), so that max |a(t)|t≤L = M×Q

Q = difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data.

M = relative maximum bias – user set: 1, 2, 5

M ≥ 1 → bias significantly separate from steady state data → easier to detect.

M ≤ 1 → bias absorbed into steady state data variance → harder to detect.