Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9 1 SIMULATION MODELING AND ANALYSIS WITH ARENA T. Altiok and B. Melamed Chapter 9

Altiok / Melamed Simulation Modeling and Analysis with ArenaChapter 9

1

SIMULATION MODELING AND ANALYSIS SIMULATION MODELING AND ANALYSIS WITH ARENAWITH ARENA

T. Altiok and B. MelamedT. Altiok and B. Melamed

Chapter 9Chapter 9

Output AnalysisOutput Analysis


2

• Output Analysis is the modeling stage concerned with • simulation replications• computing statistics from replications• presenting statistics in textual or graphical format

• Output Analysis activities consist of the following activities:• replication design• estimation of performance metrics• system analysis and experimentation

Output Analysis Activities


3

• Simulation models can be classified into two main classes, based on their time horizon:

• terminating models

• steady-state models

Types of Simulation Models


4

Terminating Simulation Models• A terminating simulation model has a natural termination time for its replications

• the time horizon is inherent in the system and finite • the modeler is interested in short-term system dynamics • statistics are computed within the system’s natural time horizon

• The number of replications is a critical parameter of the

associated Output Analysis• it is the only means of controlling the sample size of any given estimator.


5

• A steady state simulation model has no natural termination time for its replications

• the time horizon could be potentially infinite• the modeler is interested in long-term dynamics and statistics

• While long-term statistics are of interest, initial system

conditions tend to bias its long-term statistics • it, therefore, makes sense to start statistics collection after an initial

period of system warm-up, namely, after the biasing effect of the initial

conditions decays to insignificance• the transient-state regime is characterized by statistics that vary as

function of time, while the steady-state regime prevails when statistics

stabilize and do not vary over time

Steady-State Simulation Models


6

Workstation throughput and average number of jobs as functions of time

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1 51 101 151 201 251 301 351 401 451

Time

Average Time in Buffer

Throughput

Sample Steady-State Simulation Output


7

• Since steady-state models have no natural termination time, how does one select a replication length?

• the replication can be stopped when statistics at the end of several

successive increments are sufficiently close

(e.g., within some difference, to be determined by the analyst)

• Since a warm-up period is needed to eliminate statistical bias, how does one select the warm-up length?

• in a similar vein, the length of a warm-up period is determined by

observing experimentally when the time variability of the statistics of

interest largely disappears

Steady-State Simulation Issues


8

• Suppose we are interested in a parameter of the system• e.g., mean flow time or blocking probability

• The simulation will then be programmed to produce a (variate) estimator, , which evaluates to some estimate

• For example, let denote the random variable of flow time through the system of job in replication , and let be a realization of

• if the estimator of the mean flow time is the sample mean of flow times in each replication, then the corresponding replication estimates are

where is the number of flow time observations recorded in replication

Statistics Collection From Replicationsq

Q̂ ˆQ̂ q=

( )j

X r( )j

x r( )j

X r

( )

( ) ( ), , ,( )

l r

jj

x r x r r nl r =

= = ¼å1

11

( )l r r

j r


9

• The Batch Means method is a practical way of collecting multiple estimates from a single replication

• the idea is to group observations into batches, which are iid or approximately so, and then collect one estimate from each batch

• Let the observed history be a discrete sample • the total of observations is divided into batches of size each,

such that (the batch size is selected to be large enough so as

to ensure that the corresponding estimators are iid or approximately so)

• this results in the following partition into sub-samples (batches)

• for each batch , an estimate is formed from the

observations of that batch only

• the replication thus yields a set of estimates

Batch Means Statistics Collection

{ , , }nx x1 Kn m kn mk=

{ , , }, { , , }, ,{ , , }k k m mkx x x x x x11 1 21 2 1K K K Km

, ,j m= 1 ˆj

q

{,,} jjk xx 1K

ˆ ˆ{ , , }mq q1 K


10

• Let a replication collect observations from a discrete sample, where• is a sequence of variates,

• is the corresponding sample of observations

• The estimator for the mean value is the sample mean

• The sample mean above is classified as a point estimator, because it estimates a scalar

• In a similar vein, the corresponding point estimate is obtained when the sample of observations (realizations) are substituted into the formula for the sample mean above

{ , , }nX X1 K

{ , , }nx x1 K

n

jj

X Xn =

= å1

1

Point Estimation for Discrete Samples

n


11

• Let a replication collect observations from a continuous sample over time, where

• is a stochastic process over some time interval

• is the corresponding sample of observations

• The estimator for the mean value is the time average

point estimate

so called because the variates involved are indexed by time

{ : }t

X t T£ £0

{:} t xtT ££ 0

T

tX X dtT

= ò0

1

Point Estimation for Continuous Samples

[ , ]T0


12

• Examples of time-continuous variates in queueing context

• the number of jobs, , at time in a queueing system (buffer and server) • the server utilization process, , is defined as the indicator

variate

where when the server is busy, and when the server is idle

• the utilization statistic is the time average

which is the fraction of time in that the server is busy, and as such

is an estimator of the probability that the server is busy

0 if 0

1 if 0

,

,t

tt

NU

N

ì =ïï= íï >ïî

T

tU U dtT

= ò0

1

tN t{ : }

tU t T£ £0

Point Estimation for Continuous Samples (Cont.)

1t

U = 0t

U =

[ , ]T0


13

Point Estimation in Arena

• The Statistic module allows the user to obtain estimates for • Tally statistics

• Time Persistent statistics

• Both Tally and Time Persistent statistics permit user access to a number of related Arena variables, such as

• TAVG(X) = tally average of variable X

• DAVG(X) = time average of variable X


14

• Consider a workstation, subject to failure, where • jobs arrive with exponential inter-arrival times of mean 1 hour

• jobs have a fixed processing time of 0.75 hours

• the workstation goes through up/down cycles as follows:• while busy, it fails randomly with exponentially distributed

time-to-failure of mean 20 hours• on failure, repair times are uniformly distributed between 1 and 5 hours

• Based on analytical calculations, we expect • the throughput to be 1 job per hour (same as the arrival rate)• the down time probability to be 0.1125• the average job delay (queue delay) to be 4.11 hours per job

[see Altiok (1997), Chapter 3]

Example: Point Estimation in Arena


15

• The results of 5 replications from a simulation of a workstation subject to failures are displayed in the table below

• Note that the estimates vary from replication to replication• the underlying estimator is a random variable!

• Any of the values can be used to estimate the true (unknown) parameter

• but how confident can the modeler be in their accuracy?

Example: Point Estimation in Arena (Cont.)

ReplicationNumber

AverageThroughput

AverageJob Delay

Probability ofDown State

1 1.0096 3.9111 0.11442 0.9858 3.9307 0.11333 1.0002 3.4373 0.10384 1.0176 3.4243 0.11165 0.9999 3.5828 0.1147


16

Confidence Interval Estimation• Confidence Interval (CI) estimation quantifies the confidence (probability) that an interval “covers” the true (but unknown) statistic

• the boundaries of the confidence interval are estimated using

appropriate point estimates

• therefore, those boundaries are random variables, and the CI is a

random interval which varies across experiments (replications)!

• the modeler predetermines the desired probability that the CI “covers”

the true statistic (the larger the probability, the wider the interval)

• Confidence intervals can be computed for• terminating simulations

• steady-state simulations


17

Confidence Intervals for Terminating Simulations

• Assume that to estimate a parameter by simulation, • independent fixed-length replications of the model were run • the runs produced a sample , where is the

point estimate of in replication

• The pooled point estimator for is the sample mean across replications

• this estimator is a random variable with mean and variance • thus, increasing the number of replications, , would decrease the

variance of , and consequently, increase our confidence in the

corresponding point estimate value,

n1ˆ ˆ{ ( ), , ( )}nq qK ˆ( )rq

q

1

1 ˆ( )n

rr

nQ Q

== å

m / ns 2

r

q

Q

Q

q

n

q


18

• We wish to quantify the confidence in an estimate of the true parameter by computing (at least approximately) the probability of events of the form where

• the estimators and define a (random) confidence interval

for the true parameter, • is the probability that the confidence interval

does not include the true parameter, • is a small probability, (often 0.01 or 0.05), called the

significance level

1 21ˆ ˆPr{ }Q q Q a£ £ = -

aq

Confidence Intervals for Terminating Simulations (Cont.)

1 a-

q

Q̂1 Q̂2

q

ˆ ˆ[ , ]Q Q1 2

ˆ ˆ[ , ]Q Q1 2


19

• By the Central Limit Theorem,

• the random variable in the estimator

is approximately normally distributed with mean and variance

• It follows that

• From properties of the normal distribution,

• the approximation improves as the sample size, , increases

• we have

(or at least approximately so, if the bias is tolerable)

ˆ( )n

rrQ

=å

1

nm ns 2

Norm( , / )nQ ms 2:

nˆE[ ] E[ ( )]rm Q Q q= = =


ˆ( )n

rr

nQ Q

== å

1

1


20

• It is known that has the standard normal

distribution with quantiles , where


0 1Norm( , )/

Zn

Q m

s

-= :

/za 2Pr{ }Z za a£ =

2/a2/a

1 a-

/za 2 /z a-1 2


21

• A confidence interval for the mean parameter, , at significance level is given by

• In case the variance is unknown, proceed as follows:

• estimate the variance by

• use the fact that the Student t distribution satisfies

• the requisite confidence interval is

/ /Pr{ / / }z n z na aQ s m Q s a- -- £ £ + = -2 21 2 1 2 1

ˆˆ( )

n

rS r

nQQ Q

== -

- å2 2

1

1[ ]

1

ˆ /nT S n

Q

Q m-

-=1

ˆ, / /nt S nQaQ - -± 2

1 1 2


q m=a

s2


22

• Consider confidence interval estimation for some mean in Batch Means setting with a single replication of batches (the underlying history may be discrete or continuous)

• The corresponding set of estimators, , gives rise to

• sample mean

• sample variance

• The confidence interval for at significance level is

Confidence Intervals for Steady-State Simulations

m

ˆ ˆ{ , , }mQ Q1K

ˆm

jjm

Q Q=

= å1

1

ˆ, / /mt S mQaQ - -± 2

1 1 2

q m=

m

q m= a

ˆˆ[ ]

m

jj

SmQ

Q Q=

= -- å2 2

1

1

1


23

Confidence Interval Estimation in Arena

• Standard Arena output provides 95% Batch Means confidence intervals for each replication

• computed for both Tally and Time Persistent statistics in terms of

half widths under the Half Width column heading• if, however, the estimated batch means are significantly dependent

or the underlying sample history is too short to yield a sufficient number

of batches, then that column will display the message (Correlated)

• Arena also supports the computation of confidence intervals from multiple replications in the Outputs element of the SIMAN summary report


24

• As a working example, consider a single-machine finishing operation workstation that processes two types of parts, denoted by G_1 and G_2, where

• parts of type G_1 arrive according to an exponential inter-arrival time distribution with mean 2 hours, and each part has a fixed processing time of 1 hour• parts of type G_2 arrive according to an exponential inter-arrival time distribution with mean 4 hours, and each part has a fixed processing time of 1.4 hours• all parts are processed in FIFO order• all part types have equal service priorities

• We wish to simulate the finishing operation for 10,000 hours in order to understand the behavior of the number of parts in the workstation buffer and the buffer delay for each part type

Example: Output Analysis via Standard Arena Output


25

Arena model of a finishing operation with two types of parts

Figure 9.3 Arena model of a finishing operation with two types of partsExample: Output Analysis via Standard Arena Output (Cont.)


26

Example: Observation Collection

Record module with Tally statistics for part delay in the buffer


27

Dialog boxes for the Set module to tally delay times for each part type (bottom) and its members (top)

Example: Observation Collection (Cont.)


28

Dialog box for the Statistic module with a Tally statistic for each part delay

Example: Observation Collection (Cont.)


29

Example: Output Summary

Output summary report for Resources


30

Example: Output Summary (Cont.)

Output summary report for Queues


31

• From the summary report we observe the following partial utilizations in the finishing machine:

• is utilization due to type G_1 parts workload

• is utilization due to type G_2 parts workload

• is utilization due to overall workload

• We, therefore, expect the probability of the finishing machine being in the Busy state to be around 0.85

• The estimated probability in the summary report for Resources is actually 0.84 over a replication of length 10,000 hours.

r== 21.4/40.35

r = =1 1 /2 0.5

0.85r =

Example: Output Analysis


32

• The table below shows the behavior of three performance measures as functions of replication length

• Observe how the estimates appear to converge to respective limiting values as the simulation length increases

• convergence is indicated by the fact that the values appear to stabilize and change very little for the higher range of replication lengths

• Since we know the true value of machine utilization, we can take advantage of this knowledge in deciding on the smallest replication length that gives rise to sufficiently accurate estimates

• high accuracy is indicated in the table by low variability in the estimates as function of replication length

• we naturally seek the smallest value, since we would like to reduce the computational effort as much as possible

.

Example: Output Analysis (Cont.)


33

PerformanceMeasure

AverageValue

HalfWidth

MinimumValue

MaximumValue

Number ofReplications

Machine Busy 0.84697 0.00669 0.83346 0.85974 10G_1 Buffer

Delay3.1366 0.28652 2.6073 4.1624 10

G_2 BufferDelay

3.1421 0.30979 2.6811 4.1841 10

Summary statistics based on 10 replications

Example: Output Analysis (Cont.)


34

Output Analysis via the Arena Output Analyzer

• The Arena Output Analyzer tool provides the following capabilities:

• Data transfer • data import from ASCII files• data export to ASCII files

• Statistical analysis • batching, correlogram, and basic statistics• point estimation and confidence interval estimation for means and standard deviations• statistical tests for comparing parameters of different samples

• Graphing • data plots and charts• statistics plots and charts


35

Example: Data Collection

Tallied buffer delay observations for parts of type G_1


36

Example: Graphical Statistics

Graphical statistics for parts of type G_1


37

Example: Batching Data for Independent Observations

Batch/Truncate dialog box


38

Batched observations stored in file : G_1_Delay_Batched.fltInitial Observations Truncated : 0Number of Batches : 499Number of Observations Per Batch : 1000Number of Trailing Obs'ns Truncated : 486Estimate of Covariance Between Batches : -0.002459

Batch/Truncate summary report for delay times of G_1 type parts

Example: Batching Data for Independent Observations (Cont.)


39

Confidence intervals for mean delays of parts of type G_1 and G_2

Example: Confidence Intervals for Means and Variances


40

Comparing Means and Variances • The Analyze menu in the Arena Output Analyzer tool provides the following options:

• Compare Means for comparing the means of two samples drawn from two populations• Compare Variances for comparing the variances of two samples drawn from two populations

• These options perform statistical analysis for testing statistically for the equality of means and variances, respectively• For example, to test the null hypothesis that the means are equal,

• a confidence interval is sets up for their difference• since under the null hypothesis the difference is zero, one accepts equality

if the confidence interval includes 0, and rejects it, otherwise


41

Dialog boxes for comparing mean buffer delays of type G_1 and G_2 parts

Example: Comparing Means


42

Test results for the equality of mean buffer delays of type G_1 and G_2 parts

Example: Testing for Equality of Means


43

• The Analyze menu in the Arena Output Analyzer tool provides the Correlogram option to gauge the statistical dependence among observations within a sample

• performs point estimation of the correlation between two samples by estimating the coefficient of correlation between the respective random variables

• This activity is called correlation analysis

Point Estimation for Correlation


44

Results of correlation analysis for type G_1 parts

• The left-hand correlogram was computed from simulation observations of successive delays• The right-hand correlogram was computed from the sample means of batches of 1000 delays• The two correlograms are strikingly different: the successive delays are strongly positively correlated, while the sample means of the batches are very nearly uncorrelated

Example: Correlation Analysis


45

• The term Parametric Analysis refers to the following set of activities:

• running a model multiple times with a different set of input parameters for each run • then comparing the resultant performance measures

• Its purpose is to carry out to Sensitivity Analysis• understand the impact of parameter changes on system behavior • often use this understanding to find the optimal configuration (parameter set) with respect to one or more performance measures, or combination thereof

• In Arena Process Analyzer parlance• input parameters are called controls• the resultant performance measures are called responses• a collection of controls and responses for a given set of runs is referred to as a scenario• a collection of scenarios is termed an Arena project

Parametric Analysis via the Arena Process Analyzer


46

A Process Analyzer window with an initial grid

The Process Analyzer GUI


47

A Process Analyzer window with a populated grid

The Process Analyzer GUI (Cont.)

Documents

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9 1 SIMULATION MODELING AND ANALYSIS WITH ARENA T. Altiok and B. Melamed Chapter 9