Upload
dolf
View
19
Download
0
Embed Size (px)
DESCRIPTION
The AutoSimOA Project. A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. http://www.wbs.ac.uk/go/autosimoa. Automating estimation of warm-up length. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School Simulation Workshop - April 2008. Research Aim. - PowerPoint PPT Presentation
Citation preview
Automating estimation of warm-up length
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
Simulation Workshop - April 2008
The AutoSimOA ProjectA 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.
http://www.wbs.ac.uk/go/autosimoa
Research Aim
• To create an automated system for dealing with the problem of initial bias, for implementation into simulation software.
• Target audience: non- (statistically) expert simulation users.
The Initial Bias Problem
• Model may not start in a “typical” state.• Can cause initial bias in the output.• Many methods proposed for dealing with
initial bias: e.g. Initial steady state conditions; run model for ‘long’ time…
• This project uses: Deletion of the initial transient data by specifying a warm-up period (or truncation point).
Question is:
How do you estimate the length of the warm-up period
required?
Methods fall into 5 main types :
1. Graphical Methods.
2. Heuristic Approaches.
3. Statistical Methods.
4. Initialisation Bias Tests.
5. Hybrid Methods.
Literature search – 42 methods
Summary of methods and
literature references on project
web site:
http://www.wbs.ac.uk/go/autosimoa
Short-listing warm-up methods for automation using literature
Short-listing Criteria
• Accuracy & robustness
• Simplicity
• Ease of automation
• Generality
• Number of parameters to estimate
• Computer running time
11
2 22
66 4
3
0
2
4
6
8
10
12
Graphical Statistical Heuristic
Warm-up method type
Nu
mb
er
of
me
tho
ds
Ease of AutomationAccuracyGeneralityRunning timeOther
Short-listing results: reasons for rejection of methods
• Statistical methods:– Goodness of Fit (GoF) test– Algorithm for a static data set (ASD)– Algorithm for a Dynamic data set (ADD)
• Heuristics:– MSER-5– Kimbler’s Double Exponential Smoothing– Euclidean Distance Method (ED)
Short-listing results: 6 Methods taken forward to testing
Testing Procedure
Test short-listed methods using:
1. Artificial data – controllable & comparable initial bias functions steady state functions
2. Set of performance criteria.
i) Length – proportion of data length.
ii) Severity – maximum bias value is a function of the difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data.
iii) Shape and Orientation – 7 shapes:
Initial bias functions - 3 Criteria:
1. Artificial Data Sets
• Mean Shift:
• Linear:
• Quadratic:
• Exponential:
• Oscillating (decreasing):
Quadratic ExponentialLinear
i) Constant steady state variance
ii) Error Terms: Normal or Exponential distribution
iii) Auto-Correlation:
No AutoCorrelation; AR(1); AR(2); AR(4); MA(2); ARMA(5,5).
Add Initial Bias to Steady state:
Superpostion: Bias Fn, a(t), added onto end of steady state function:
e.g. ...
)(1
etc
taXY
XX
tt
ttt
Steady state functions - 3 Criteria:
• Closeness of estimated truncation point (Lsol) to true truncation point (L).
• Coverage of true mean
• ½ width of 95% CI for average truncated mean.
• Bias and absolute bias in estimated mean.
• Number of failures of method.
2. Performance Criteria
Test Results• Rejections:
– ASD & ADD required a prohibitively large number of replications
– GoF & Kimbler’s method consistently severely underestimated truncation point.
– ED failed to give any result on majority of occasions
• MSER-5 most accurate and robust method.
MSER-5 Method
0
0.02
0.04
0.06
0.08
0.1
0.12
0 20
40
60
80
100
120
140
160
180
Batch number
sta
nd
ard
err
or
-4
-2
0
2
4
6
8
10
ba
tch
me
an
s
Standard Error
BatchMean
MSER-5 test statisticOutput data
(batched)
MSER-5 ResultsMSER5 result eg.xls
Does the true mean fall into the 95% CI for the estimated mean?
Non-truncated data sets
Truncated data sets
% of cases
yes yes 8.3%
yes no 0%
no no 19.3%
no yes 72.4%
Summary / Future Work
• 42 warm-up methods
• Short-listing and testing
• MSER-5 most promising method for automation
• Creation of heuristic framework around MSER-5 method for implementation into simulation software.
ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output
Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by
the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in
collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
SW08
ii) SEVERITY OF BIAS FUNCTION
Set maximum value of bias fn, a(t), so that max |a(t)|t≤L = M×Q
Q = difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data.
M = relative maximum bias – user set: 1, 2, 5
M ≥ 1 → bias significantly separate from steady state data → easier to detect.
M ≤ 1 → bias absorbed into steady state data variance → harder to detect.