Automating the Analysis of Simulation Output Data

Stewart Robinson, Katy Hoad, Ruth Davies

OR48, September 2006

Outline

The problem

A prototype automated output Analyser

Findings from prototype Analyser

The AutoSimOA Project

Current work - Collecting and characterising real and artificial models

The Problem

Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts.

Simulation software generally have very limited facilities for directing/advising on simulation experiments.

Main exception is directing scenario selection through ‘optimisers’.

With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

The Problem

Despite continued theoretical developments in simulation output analysis, little is being put into practical use.

There are 3 factors that seem to inhibit the adoption of output analysis methods:

• Limited testing of methods• Requirement for detailed statistical knowledge• Methods generally not implemented in simulation

software (AutoMod/AutoStat is an exception)

A solution would be to provide an automated output ‘Analyser’.

A Prototype Analyser

Simulationmodel

Warm-upanalysis

Run-lengthanalysis

Replicationsanalysis

Use replicationsor long-run?

Recommendationpossible?

Recommend-ation

Output data

Analyser

Obt

ain

mor

e ou

tput

dat

a

Masters Project (3 students).

The Analyser looked at:

• Warm-up

• Run-length

• Number of replications

Scenario analysis could be added.

A Prototype Analyser

A prototype Analyser has been developed in Microsoft Excel.

At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.

Illustration: Warm-up

Load Analyser into Excel.

Enter name of SIMUL8 model.

Specify initial number of replications and run-length to use.

Welch's Method: Plot of Moving Average (Window = 12 )

0.00

20.00

40.00

60.00

80.00

100.00

120.00

1 84 167 250 333 416 499 582 665 748 831 914

Observation

Mo

vin

g a

vera

ge

Illustration: Warm-up

Illustration: Replications

Findings from Prototype Analyser

It is possible to link an Automated Analyser in Excel to a simulation software tool.

This was just a proof of concept.

Key issues to address:• More thorough testing of output analysis methods for their

accuracy and their generality.• Adaptation of methods to sequential procedures and to

minimise the need for user intervention.

A 3 year, EPSRC funded project (GR EP/D033640/1) in collaboration with SIMUL8 Corporation.

The AutoSimOA Project

Objectives

• To determine the most appropriate methods for automating simulation output analysis

• To determine the effectiveness of the analysis methods• To revise the methods where necessary in order to

improve their effectiveness and capacity for automation• To propose a procedure for automated output analysis of

warm-up, replications and run-length

Only looking at analysis of a single scenario

The AutoSimOA Project

CURRENT WORK:

1. Literature review of warm-up, replications and run-length methods.

2. Development of artificial data sets (Auto-Regressive; Moving average; M/M/n/p Queues…)

3. Collection of ‘real’ simulation models.

Use models / data sets:

Provide a representative and sufficient set of models / data output for use in discrete event simulation research.

Use models / data sets to test the chosen simulation output analysis methods in the AutoSimOA Project..

Categorising Output Data Sets by Shape & Characteristics

Group A

…Group NGroup B

Auto Correlation Spread round mean

NormalityTrend

Cycling/Seasonality

Terminating

Non-terminating

Steady state

In/out of control

Transient

Model characteristics

Deterministic or random

Significant pre-determined model changes (by time)

Dynamic internal changes i.e. ‘feed-back’

Empty-to-empty pattern

Initial transient (warm-up)

Out of control trend ρ≥1

Cycle

Auto-correlation

Statistical distribution

Output data characteristics

ARTIFICIAL MODELS

Create simple models where theoretical value of some attribute is known.

E.g. M/M/1: mean waiting time.

Create simple models where value of some attribute is estimated but model characteristics can be controlled.

E.g. Single item inventory management system: Number-in-stock.

Construct output, which closely resembles real model output, with known value for some specific attribute.

E.g. AR(1) with Normal errors

Create different output types Transient

Steady state

Steady state cycle

Trend + Initial transient (warm-up)

Example artificial models:

1. Auto-Regressive (2) series

BiasFnXXX ttt 21 5.025.0

1005.010 te

220010 1005.0 t

Sine t

Exponential

Under Damped oscillations

Mean shift 2

Initial Bias Functions:

Run1 ~ AR(2) + "underdamped oscillations" initial bias

-30

-20

-10

0

10

20

30

40

0 100 200 300 400 500

t

Run1 ~ AR(2) + "mean shift" initial bias

-6

-4

-2

0

2

4

6

8

10

12

14

0 100 200 300 400 500t

Run1 ~ AR(2) + "exponential" initial bias

-10

-5

0

5

10

15

20

25

30

35

40

0 100 200 300 400 500t

Run1 ~ AR(2) with no initial bias

-4

-3

-2

-1

0

1

2

3

4

5

0 100 200 300 400 500

t

mean 1.8

Example artificial models:

2. E4 ~ Erlang(4) / M / 1 Queue

Traffic Intensity = 0.8

Queuing time for each customer in a E4/M/1Queuing System

0

2

4

6

8

10

12

14

16

1 432 863 1294 1725 2156 2587 3018 3449 3880 4311 4742

index

REAL MODELS

Models created in “real circumstances” that cover each

general type of model and output encountered in real life modeling.

e.g. Call centre: percentage of calls answered within 30

secs

e.g. Production Line Manufacturing Plant:

through-put / hour

e.g. Fast Food Store:

average queuing timee.g. Swimming

Pool complex: average

number in system

TransientSteady State

Cycle

Steady State

With or without warm-up

Trend

0

2

4

6

8

10

12

14

16

18

0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480

time (mins)

num

ber

of custo

mers

0

5

10

15

20

25

30

35

40

45

0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480

time (mins)

num

ber

of custo

mers

Example ‘ real ’ models:1. Argos – Number of customers in queue to pay

Stochastic model with changing arrival rates.

Empty to empty; transient; autocorrelated; non-normal output.

0

50

100

150

200

Time (hours)

Num

ber

of ite

ms

Example ‘ real ’ models:2. Leggings Manufacturing Plant – Through-put / hour

Stochastic model.

Steady state with warm-up; not autocorrelated; normal output.

0

50

100

150

200

Time (hours)

Num

ber

of ite

ms

0

10

20

30

40

50

60

1 5 9 13 17 21 25 29 33 37 41

Time (hours)

Num

ber

of

com

ple

ted ite

ms

Example ‘ real ’ models:3. Sanitory Towel Packing Plant – Through-put / hour

Stochastic model with changing productivity in work stations.

Steady state daily cycle.

40

41

42

43

44

45

1 10 19 28 37 46 55 64 73 82 91 100 109 118

Time (days)

Mea

n th

roug

h-pu

t for

the

day

Series of means of each cycle:

autocorrelated; non-normal output.

Use this representative and sufficient set of models/outputwhen

The AutoSimOA Project

• determining the most appropriate methods for automating simulation output analysis

• determining the effectiveness of the analysis methods

• revising the methods where necessary in order to improve their effectiveness and capacity for automation

In order to propose a procedure for automated output analysis of warm-up, replications and run-length.

Automating the Analysis of Simulation Output Data

Stewart Robinson, Katy Hoad, Ruth Davies

OR48, September 2006