7/21/2019 All in One simulation notes
1/447
Chapter 1 Introduction toSimulation
Banks, Carson, Nelson & Nicol
Discrete-Event System Simulation
7/21/2019 All in One simulation notes
2/447
2
Outline
When Simulation Is the Appropriate Tool
When Simulation Is Not Appropriate
Advantages and Disadvantages of Simulation
Areas of Application
Systems and System Environment
Components of a System Discrete and Continuous Systems
Model of a System
Types of Models Discrete-Event System Simulation
Steps in a Simulation Study
7/21/2019 All in One simulation notes
3/447
3
Definition
A simulation is the imitation of the operation of real-world
process or system over time.
Generation of artificial history and observation of that
observation history
Amodelconstruct a conceptual framework that
describes a system The behavior of a system that evolves over time is
studied by developing a simulation model.
The model takes a set of expressed assumptions:
Mathematical, logical
Symbolic relationship between the entities
7/21/2019 All in One simulation notes
4/447
4
Goal of modeling and simulation
A model can be used to investigate a wide verity of what
if questions about real-world system.
Potential changes to the system can be simulated and predicate
their impact on the system.
Find adequate parameters before implementation
So simulation can be used as Analysis tool for predicating the effect of changes
Design tool to predicate the performance of new system
It is better to do simulation before Implementation.
7/21/2019 All in One simulation notes
5/447
5
How a model can be developed?
Mathematical Methods
Probability theory, algebraic method , Their results are accurate
They have a few Number of parameters
It is impossible for complex systems Numerical computer-based simulation
It is simple
It is useful for complex system
7/21/2019 All in One simulation notes
6/447
6
When Simulation Is the Appropriate Tool
Simulation enable the study of internal interaction of a subsystemwith complex system
Informational, organizational and environmental changes can besimulated and find their effects
A simulation model help us to gain knowledge about improvement ofsystem
Finding important input parameters with changing simulation inputs
Simulation can be used with new design and policies beforeimplementation
Simulating different capabilities for a machine can help determinethe requirement
Simulation models designed for training make learning possiblewithout the cost disruption
A plan can be visualized with animated simulation
The modern system (factory, wafer fabrication plant, serviceorganization) is too complex that its internal interaction can betreated only by simulation
7/21/2019 All in One simulation notes
7/447
7
When Simulation Is Not Appropriate
When the problem can be solved by common
sense. When the problem can be solved analytically.
If it is easier to perform direct experiments.
If cost exceed savings. If resource or time are not available.
If system behavior is too complex.
Like human behavior
7/21/2019 All in One simulation notes
8/447
8
Advantages and disadvantages of simulation
In contrast to optimization models, simulation
models are run rather than solved.Given as a set of inputs and model characteristics the
model is run and the simulated behavior is observed
7/21/2019 All in One simulation notes
9/447
9
Advantages of simulation New policies, operating procedures, information flows and son on
can be explored without disrupting ongoing operation of the realsystem.
New hardware designs, physical layouts, transportation systemsand can be tested without committing resources for theiracquisition.
Time can be compressed or expanded to allow for a speed-up orslow-down of the phenomenon( clock is self-control).
Insight can be obtained about interaction of variables and importantvariables to the performance.
Bottleneck analysis can be performed to discover where work inprocess, the system is delayed.
A simulation study can help in understanding how the systemoperates.
What if questions can be answered.
7/21/2019 All in One simulation notes
10/447
10
Disadvantages of simulation
Model building requires special training.
Vendors of simulation software have been activelydeveloping packages that contain models that only
need input (templates).
Simulation results can be difficult to interpret.
Simulation modeling and analysis can be time
consuming and expensive.
Many simulation software have output-analysis.
7/21/2019 All in One simulation notes
11/447
11
Areas of application Manufacturing Applications
Semiconductor Manufacturing
Construction Engineering and project management
Military application
Logistics, Supply chain and distribution application
Transportation modes and Traffic
Business Process Simulation
Health Care
Automated Material Handling System (AMHS) Test beds for functional testing of control-system software
Risk analysis Insurance, portfolio,...
Computer Simulation CPU, Memory,
Network simulation Internet backbone, LAN (Switch/Router), Wireless, PSTN (call center),...
7/21/2019 All in One simulation notes
12/447
12
Systems and System Environment
A system is defined as a groups of objects thatare joined together in some regular interactiontoward the accomplishment of some purpose.An automobile factory: Machines, components parts
and workers operate jointly along assembly line
A system is often affected by changes occurringoutside the system: system environment. Factory : Arrival orders
Effect of supply on demand : relationship between factory
output and arrival (activity of system)
Banks : arrival of customers
7/21/2019 All in One simulation notes
13/447
13
Components of system Entity
An object of interest in the system : Machines in factory
Attribute The property of an entity : speed, capacity
Activity A time period of specified length :welding, stamping
State A collection of variables that describe the system in any time : status of machine
(busy, idle, down,) Event
A instantaneous occurrence that might change the state of the system:breakdown
Endogenous Activities and events occurring with the system
Exogenous Activities and events occurring with the environment
7/21/2019 All in One simulation notes
14/447
14
Discrete and Continues Systems
A discrete system is one in which the state variables
change only at a discrete set of points in time : Bank
example
7/21/2019 All in One simulation notes
15/447
15
Discrete and Continues Systems (cont.)
A continues system is one in which the state variables
change continuously over time: Head of water behind the
dam
7/21/2019 All in One simulation notes
16/447
16
Model of a System
To study the system
it is sometimes possible to experiments with system This is not always possible (bank, factory,)
A new system may not yet exist
Model: construct a conceptual framework that
describes a system
It is necessary to consider those accepts of systems
that affect the problem under investigation
(unnecessary details must remove)
7/21/2019 All in One simulation notes
17/447
17
Types of Models
7/21/2019 All in One simulation notes
18/447
18
Characterizing a Simulation ModelCharacterizing a Simulation Model
Deterministic or Stochastic Does the model contain stochastic components?
Randomness is easy to add to a DES
Static or Dynamic Is time a significant variable?
Continuous or Discrete Does the system state evolve continuously or only at
discrete points in time?
Continuous: classical mechanics Discrete: queuing, inventory, machine shop models
7/21/2019 All in One simulation notes
19/447
19
Discrete-Event Simulation Model
Stochastic: some state variables are random
Dynamic: time evolution is importantDiscrete-Event: significant changes occur at
discrete time instances
7/21/2019 All in One simulation notes
20/447
20
Model TaxonomyModel Taxonomy
7/21/2019 All in One simulation notes
21/447
21
DES Model DevelopmentDES Model Development
How to develop a model:
1) Determine the goals and objectives2) Build a conceptualmodel
3) Convert into a specification model
4) Convert into a computational model
5) Verify
6) Validate
Typically an iterative process
7/21/2019 All in One simulation notes
22/447
22
Three Model LevelsThree Model Levels
Conceptual Very high level
How comprehensive should the model be? What are the state variables, which are dynamic, and which are
important?
Specification
On paper May involve equations, pseudocode, etc.
How will the model receive input?
Computational
A computer program General-purpose PL or simulation language?
7/21/2019 All in One simulation notes
23/447
23
Verification vs. ValidationVerification vs. Validation
Verification Computational model should be consistent with
specification model
Did we build the model right?
Validation
Computational model should be consistent with thesystem being analyzed
Did we build the right model?
Can an expert distinguish simulation output fromsystem output?
Interactive graphics can prove valuable
7/21/2019 All in One simulation notes
24/447
24
Steps in Simulation
Study
7/21/2019 All in One simulation notes
25/447
Computer Science, Informatik 4
Communication and Distributed Systems
Simulation
Discrete-Event System Simulation
Dr. Mesut Gne
7/21/2019 All in One simulation notes
26/447
Computer Science, Informatik 4
Communication and Distributed Systems
Chapter 9
Verification and Validation of Simulation Models
7/21/2019 All in One simulation notes
27/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
3hapter 9. Verification and Validation of Simulation Models
Purpose Overview
The goal of the validation process is:
To produce a model that represents true
behavior closely enough for decision-making
purposes
To increase the models credibility to an
acceptable level
Validation is an integral part of model
development:
Verification: building the model correctly,
correctly implemented with good input and
structure
Validation: building the correct model, anaccurate representation of the real system
Most methods are informal subjective
comparisons while a few are formal
statistical procedures
7/21/2019 All in One simulation notes
28/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
4hapter 9. Verification and Validation of Simulation Models
Modeling-Building, Verification Validation
Steps in Model-Building
Observing the real system
and the interactions amongtheir various components
and of collecting data on
their behavior.
Construction of aconceptual model
Implementation of an
operational model
7/21/2019 All in One simulation notes
29/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
5hapter 9. Verification and Validation of Simulation Models
Verification
Purpose: ensure the conceptual model is reflected accurately in the
computerized representation.
Many common-sense suggestions, for example:
Have someone else check the model.
Make a flow diagram that includes each logically possible action a
system can take when an event occurs.
Closely examine the model output for reasonableness under a variety of
input parameter settings.
Print the input parameters at the end of the simulation, make sure they
have not been changed inadvertently.
Make the operational model as self-documenting as possible.
If the operational model is animated, verify that what is seen in theanimation imitates the actual system.
Use the debugger.
If possible use a graphical representation of the model.
7/21/2019 All in One simulation notes
30/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
6hapter 9. Verification and Validation of Simulation Models
Examination of Model Output for Reasonableness
Two statistics that give a quick indication of model
reasonableness are current contents and total counts
Current content: The number of items in each component of thesystem at a given time.
Total counts: Total number of items that have entered each
component of the system by a given time.
Compute certain long-run measures of performance, e.g.
compute the long-run server utilization and compare to
simulation results.
7/21/2019 All in One simulation notes
31/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
7hapter 9. Verification and Validation of Simulation Models
Examination of Model Output for Reasonableness
A model of a complex network of queues consisting of
many service centers.
If the current content grows in a more or less linear fashion asthe simulation run time increases, it is likely that a queue is
unstable
If the total count for some subsystem is zero, indicates no items
entered that subsystem, a highly suspect occurrence If the total and current count are equal to one, can indicate that
an entity has captured a resource but never freed that resource.
7/21/2019 All in One simulation notes
32/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
8hapter 9. Verification and Validation of Simulation Models
Other Important Tools
Documentation
A means of clarifying the logic of a model and verifying its
completeness. Comment the operational model, definition of all variables and
parameters.
Use of a trace
A detailed printout of the state of the simulation model over time.- Can be very labor intensive if the programming language does not
support statistic collection.
- Labor can be reduced by a centralized tracing mechanism
- In object-oriented simulation framework, trace support can beintegrated into class hierarchy. New classes need only to add little for
the trace support.
7/21/2019 All in One simulation notes
33/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
9hapter 9. Verification and Validation of Simulation Models
Trace - Example
Simple queue from Chapter 2
Trace over a time interval [0, 16]
Allows the test of the results by pen-and-paper method
Definition of Variables:
CLOCK = Simulation clock
EVTYP = Event type (Start, Arrival, Departure, Stop)NCUST = Number of customers in system at time CLOCK
STATUS = Status of server (1=busy, 0=idle)
State of System Just After the Named Event Occurs:
CLOCK = 0 EVTYP = Start NCUST=0 STATUS = 0
CLOCK = 3 EVTYP = Arrival NCUST=1 STATUS = 0CLOCK = 5 EVTYP = Depart NCUST=0 STATUS = 0
CLOCK = 11 EVTYP = Arrival NCUST=1 STATUS = 0
CLOCK = 12 EVTYP = Arrival NCUST=2 STATUS = 1
CLOCK = 16 EVTYP = Depart NCUST=1 STATUS = 1
...
There is a customer,
but the status is 0
7/21/2019 All in One simulation notes
34/447
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
10hapter 9. Verification and Validation of Simulation Models
Calibration and Validation
Validation: the overall process of comparing the model and its behavior to the
real system.
Calibration: the iterative process of comparing the model to the real system
and making adjustments.
Comparison of the model to
real system
Subjective tests
- People who areknowledgeable about
the system
Objective tests
- Requires data on the
real systems behavior
and the output of the
model
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
35/447
Dr. Mesut Gne
Communication and Distributed Systems
11hapter 9. Verification and Validation of Simulation Models
Calibration and Validation
Danger during the calibration phase
Typically few data sets are available, in the worst case only one, and the
model is only validated for these.
Solution: If possible collect new data sets
No model is ever a perfect representation of the system
The modeler must weigh the possible, but not guaranteed, increase inmodel accuracy versus the cost of increased validation effort.
Three-step approach for validation:
1. Build a model that has high face validity.
2. Validate model assumptions.
3. Compare the model input-output transformations with the real systems
data.
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
36/447
Dr. Mesut Gne
Communication and Distributed Systems
12hapter 9. Verification and Validation of Simulation Models
High Face Validity
Ensure a high degree of realism:
Potential users should be involved in model construction from its
conceptualization to its implementation. Sensitivity analysis can also be used to check a models face
validity.
Example: In most queueing systems, if the arrival rate of
customers were to increase, it would be expected that serverutilization, queue length and delays would tend to increase.
For large-scale simulation models, there are many input
variables and thus possibly many sensitity tests.
- Sometimes not possible to perform all of theses tests, select themost critical ones.
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
37/447
Dr. Mesut Gne
Communication and Distributed Systems
13hapter 9. Verification and Validation of Simulation Models
Validate Model Assumptions
General classes of model assumptions:
Structural assumptions: how the system operates.
Data assumptions: reliability of data and its statistical analysis.
Bank example: customer queueing and service facility in a bank.
Structural assumptions
- Customer waiting in one line versus many lines
- Customers are served according FCFS versus priority
Data assumptions, e.g., interarrival time of customers, service times for
commercial accounts.
- Verify data reliability with bank managers
- Test correlation and goodness of fit for data
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
38/447
Dr. Mesut Gne
Communication and Distributed Systems
14hapter 9. Verification and Validation of Simulation Models
Validate Input-Output Transformation
Goal: Validate the models ability to predict future behavior
The only objective test of the model.
The structure of the model should be accurate enough to make goodpredictions for the range of input data sets of interest.
One possible approach: use historical data that have been reserved
for validation purposes only.
Criteria: use the main responses of interest.
SystemInput Output
ModelOutputInput
Model is viewed as an
input-outputtransformation
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
39/447
Dr. Mesut Gne
Communication and Distributed Systems
15hapter 9. Verification and Validation of Simulation Models
Bank Example
Example: One drive-in window serviced by one teller, only one or two
transactions are allowed.
Data collection: 90 customers during 11 am to 1 pm.
- Observed service times {Si, i = 1,2, , 90}.
- Observed interarrival times {Ai, i = 1,2, , 90}.
Data analysis let to the conclusion that:
- Interarrival times: exponentially distributed with rate = 45
- Service times:N(1.1, 0.22)Input variables
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
40/447
Dr. Mesut Gne
Communication and Distributed Systems
16hapter 9. Verification and Validation of Simulation Models
Bank Example The Black Box
A model was developed in close consultation with bank management and
employees
Model assumptions were validated
Resulting model is now viewed as a black box:
Input Variables
Possion arrivals
= 45/hr: X11, X12,
Services times,
N(D2, 0.22): X21, X22,
D1 = 1 (one teller)
D2 = 1.1 min
(mean service time)
D3 = 1 (one line)
Uncontrolled
variables, X
Controlled
Decision
variables, D
Model Output Variables, Y
Primary interest:Y1 = tellers utilization
Y2 = average delay
Y3 = maximum line length
Secondary interest:
Y4 = observed arrival rateY5 = average service time
Y6 = sample std. dev. of
service times
Y7 = average length of time
Model
black box
f(X,D) = Y
Computer Science, Informatik 4
7/21/2019 All in One simulation notes
41/447
Dr. Mesut Gne
Communication and Distributed Systems
17hapter 9. Verification and Validation of Simulation Models
Comparison with Real System Data
Real system data are necessary for validation.
System responses should have been collected during the same time
period (from 11am to 1pm on the same day.)
Compare the average delay from the model Y2
with the actual delay
Z2:
Average delay observed,Z2 = 4.3 minutes, consider this to be the true
mean value 0 = 4.3.
When the model is run with generated random variatesX1n andX2n, Y2should be close toZ2.
7/21/2019 All in One simulation notes
42/447
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
43/447
Dr. Mesut Gne
19hapter 9. Verification and Validation of Simulation Models
Hypothesis Testing
Compare the average delay from the model
Y2 with the actual delayZ2
Null hypothesis testing: evaluate whether the simulation and the realsystem are the same(w.r.t. output measures):
- IfH0 is not rejected, then, there is no reason to consider the model
invalid
- IfH0 is rejected, the current version of the model is rejected, and the
modeler needs to improve the model
minutes3.4
minutes34
21
20
=
): E(YH
.): E(YH
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
44/447
Dr. Mesut Gne
20hapter 9. Verification and Validation of Simulation Models
Hypothesis Testing
Conduct the
ttest:
Chose level of significance (= 0.5) and sample size (n = 6).
Compute the same mean and sample standard deviation overthe n replications:
Compute test statistics:
Hence, rejectH0.- Conclude that the model is inadequate.
Check: the assumptions justifying a ttest, that the observations(Y2i) are normally and independently distributed.
minutes51.21
1
22 == =
n
i
iYn
Yminutes82.0
1
)(1
2
22
=
==
n
YY
S
n
i
i
test)sided-2a(for571.25.346/82.0
3.451.2
/
020 =>=
=
= criticalt
nS
Yt
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
45/447
Dr. Mesut Gne
21hapter 9. Verification and Validation of Simulation Models
Hypothesis Testing
Similarly, compare the model output with the observed output for
other measures:
Y4
Z4,
Y5
Z5,
and Y6
Z6
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
46/447
Dr. Mesut Gne
22hapter 9. Verification and Validation of Simulation Models
Type II Error
For validation, the power of the test is:
Probability[ detecting an invalid model ] = 1
= P(Type II error) = P(failing to rejectH0|H
1is true)
Consider failure to rejectH0 as a strong conclusion, the modeler would
want to be small.
Value of depends on:
- Sample size, n
- The true difference, , betweenE(Y) and :
In general, the best approach to control error is:
Specify the critical difference, .
Choose a sample size, n, by making use of the operating characteristics
curve (OC curve).
=
)(YE
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
47/447
Dr. Mesut Gne
23hapter 9. Verification and Validation of Simulation Models
Type II Error
Operating characteristics curve
(OC curve).
Graphs of the probability of aType II Error () versus for agiven sample size n
For the same error
probability with smaller
difference the required
sample size increses!
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
48/447
Dr. Mesut Gne
24hapter 9. Verification and Validation of Simulation Models
Type I and II Error
Type I error (
):
Error of rejecting a valid model.
Controlled by specifying a small level of significance .
Type II error (
):
Error of accepting a model as valid when it is invalid.
Controlled by specifying critical difference and find the n.
For a fixed sample size
n, increasing
will decrease
.
Associated RiskModeling TerminologyStatistical Terminology
Failure to reject an invalid modelType II: failure to reject H0 when H1 is true
Rejecting a valid modelType I: rejecting H0 when H0 is true
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
49/447
Dr. Mesut Gne
25hapter 9. Verification and Validation of Simulation Models
Confidence Interval Testing
Confidence interval testing: evaluate whether the simulation and the
real system performance measures are close enough
If
Yis the simulation output, and
=
E(Y)
The confidence interval (CI) for is:
n
S
tY n 1,2
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
50/447
Dr. Mesut Gne
26hapter 9. Verification and Validation of Simulation Models
Confidence Interval Testing
Validating the model:
Suppose the CI does not contain 0:
- If the best-case error is > , model
needs to be refined.
- If the worst-case error is , acceptthe model.
- If best-case error is , additionalreplications are necessary.
Suppose the CI contains 0:
- If either the best-case or worst-case
error is > , additional replications
are necessary.
- If the worst-case error is , acceptthe model.
is a dif ference value chosen
by the analyst, that is small
enough to allow valid
decisions to be based on
simulations!
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
51/447
Dr. Mesut Gne
27hapter 9. Verification and Validation of Simulation Models
Confidence Interval Testing
Bank example:
= 4
, and close enough is
= 1 minute of
expected customer delay.
A 95% confidence interval, based on the 6 replications is
[1.65, 3.37] because:
0= 4.3 falls outside the confidence interval,
- the best case |3.37 4.3| = 0.93 < 1, but
- the worst case |1.65 4.3| = 2.65 > 1
Additional replications are needed to reach a decision.
6
82.0
571.251.2
5,025.0
n
StY
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
52/447
Dr. Mesut Gne
28hapter 9. Verification and Validation of Simulation Models
Using Historical Output Data
An alternative to generating input data:
Use the actual historical record.
Drive the simulation model with the historical record and then compare
model output to system data.
In the bank example, use the recorded interarrival and service times for
the customers {An, Sn, n = 1,2,}.
Procedure and validation process: similar to the approach used for
system generated input data.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
53/447
Dr. Mesut Gne
29hapter 9. Verification and Validation of Simulation Models
Using a Turing Test
Use in addition to statistical test, or when no statistical test is readily
applicable.
Utilize persons knowledge about the system.
For example:
Present 10 system performance reports to a manager of the system.
Five of them are from the real system and the rest are fake reports
based on simulation output data.
If the person identifies a substantial number of the fake reports, interview
the person to get information for model improvement.
If the person cannot distinguish between fake and real reports with
consistency, conclude that the test gives no evidence of model
inadequacy.
Turing TestDescribed by Alan Turing in 1950. A human jugde is involved in a natural languageconversation with a human and a machine. If the judge cannot reliably tell which of the
partners is the machine, then the machine has passed the test.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
54/447
Dr. Mesut Gne
30hapter 9. Verification and Validation of Simulation Models
Summary
Model validation is essential:
Model verification
Calibration and validation
Conceptual validation
Best to compare system data to model data, and make comparison
using a wide variety of techniques.
Some techniques that we covered: Insure high face validity by consulting knowledgeable persons.
Conduct simple statistical tests on assumed distributional forms.
Conduct a Turing test.
Compare model output to system output by statistical tests.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
55/447
Simulation
Discrete-Event System Simulation
Dr. Mesut Gne
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
56/447
Chapter 10
Output Analysis for a Single Model
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
57/447
Dr. Mesut Gne
3hapter 10. Output Analysis for a Single Model
Purpose
Objective: Estimate system performance via simulation
If is the system performance, the precision of the estimator can
be measured by:
The standard error of .
The width of a confidence interval (CI) for .
Purpose of statistical analysis:
To estimate the standard error or CI .
To figure out the number of observations required to achieve desirederror or CI.
Potential issues to overcome:
Autocorrelation, e.g. inventory cost for subsequent weeks lack statisticalindependence.
Initial conditions, e.g. inventory on hand and number of backorders attime 0 would most likely influence the performance of week 1.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
58/447
Dr. Mesut Gne
4hapter 10. Output Analysis for a Single Model
Outline
Distinguish the two types of simulation:
transient vs.
steady state
Illustrate the inherent variability in a stochastic discrete-event
simulation.
Cover the statistical estimation of performance measures.
Discusses the analysis of transient simulations.
Discusses the analysis of steady-state simulations.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
59/447
Dr. Mesut Gne
5hapter 10. Output Analysis for a Single Model
Type of Simulations
Terminating versus non-terminating simulations
Terminating simulation:
Runs for some duration of time TE, whereEis a specified event that
stops the simulation.
Starts at time 0 under well-specified initial conditions.
Ends at the stopping time TE
.
Bank example: Opens at 8:30 am (time 0) with no customers present
and 8 of the 11 teller working (initial conditions), and closes at 4:30 pm
(Time TE= 480 minutes).
The simulation analyst chooses to consider it a terminating system
because the object of interest is one days operation.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
60/447
Dr. Mesut Gne
6hapter 10. Output Analysis for a Single Model
Type of Simulations
Non-terminating simulation:
Runs continuously, or at least over a very long period of time.
Examples: assembly lines that shut down infrequently, hospital
emergency rooms, telephone systems, network of routers, Internet.
Initial conditions defined by the analyst.
Runs for some analyst-specified period of time TE.
Study the steady-state (long-run) properties of the system, properties
that are not influenced by the initial conditions of the model.
Whether a simulation is considered to be terminating or non-
terminating depends on both
The objectives of the simulation study and
The nature of the system
7/21/2019 All in One simulation notes
61/447
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
62/447
Dr. Mesut Gne
8hapter 10. Output Analysis for a Single Model
Stochastic Nature of Output Data
M/G/1 queueing example (cont.):
Batched average queue length for 3 independent replications:
Inherent variability in stochastic simulation both within a single replication
and across different replications.
The average across 3 replications, can be regarded as
independent observations, but averages within a replication, Y11, , Y15,
are not.
1, Y1j 2, Y2j 3, Y3j[0, 1000) 1 3.61 2.91 7.67
[1000, 2000) 2 3.21 9.00 19.53
[2000, 3000) 3 2.18 16.15 20.36
[3000, 4000) 4 6.92 24.53 8.11
[4000, 5000) 5 2.82 25.19 12.62
[0, 5000) 3.75 15.56 13.66
ReplicationBatching Interval
(minutes) Batch, j
,,, .3.2.1 YYY
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
63/447
Dr. Mesut Gne
9hapter 10. Output Analysis for a Single Model
Measures of performance
Consider the estimation of a performance parameter,
(or
), of a
simulated system.
Discrete time data: [Y1, Y2, , Yn], with ordinary mean:
Continuous-time data: {Y(t), 0 t TE} with time-weighted mean:
Point estimation for discrete time data.
The point estimator:
- Is unbiased if its expected value is , that is if:
- Is biased if: and is called bias of
=)(E Desired
)(E
=
=n
i
iYn 1
1
)(E
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
64/447
Dr. Mesut Gne
10hapter 10. Output Analysis for a Single Model
Point Estimator
Point estimation for continuous-time data.
The point estimator:
- Is biased in general where: .
- An unbiased or low-bias estimator is desired.
Usually, system performance measures can be put into the common
framework of or :
The proportion of days on which sales are lost through an out-of-stocksituation, let:
)(E
= ET
E
dttYT 0
)(1
=otherwise,0
dayonstockofoutif,1)(
iiY
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
65/447
Dr. Mesut Gne
11hapter 10. Output Analysis for a Single Model
Point Estimator
Performance measure that does not fit:
quantile or percentile:
Estimating quantiles: the inverse of the problem of estimating a proportion
or probability.
Consider a histogram of the observed values Y:
- Find such that 100p% of the histogram is to the left of (smaller than) .
A widely used used performance meausure is the median, which is the
0.5 quantile or 50-th percentile.
pY = }Pr{
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
66/447
Dr. Mesut Gne
12hapter 10. Output Analysis for a Single Model
Confidence-Interval Estimation
To understand confidence intervals fully, it is important to distinguish
between measures of error, and measures of risk, e.g., confidence
interval versus prediction interval.
Suppose the model is the normal distribution with mean , variance 2
(both unknown).
Let Yi. be the average cycle time for parts produced on the i-threplication
of the simulation (its mathematical expectation is ).
Average cycle time will vary from day to day, but over the long-run the
average of the averages will be close to .
Sample variance acrossR replications:
=
=R
i
i YYR
S1
2
...
2 )(1
1
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
67/447
Dr. Mesut Gne
13hapter 10. Output Analysis for a Single Model
Confidence-Interval Estimation
Confidence Interval (CI):
A measure of error.
Where Yi are normally distributed.
We cannot know for certain how far is from but CI attempts to bound
that error.
A CI, such as 95%, tells us how much we can trust the interval to actually
bound the error between and . The more replications we make, the less error there is in (converging
to 0 asR goes to infinity).
R
StY
R 1,..2
..Y
..Y
..Y
Quantile of the tdistribution with R-1
degrees of freedom.
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
68/447
Dr. Mesut Gne
14hapter 10. Output Analysis for a Single Model
Confidence-Interval Estimation
Prediction Interval (PI):
A measure of risk.
A good guess for the average cycle time on a particular day is our
estimator but it is unlikely to be exactly right.
PI is designed to be wide enough to contain the actualaverage cycle time
on any particular day with high probability.
Normal-theory prediction interval:
The length of PI will not go to 0 asR increases because we can never
simulate away risk. PIs limit is:
RStY R
111,2/.. +
2/z
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
69/447
Dr. Mesut Gne
15hapter 10. Output Analysis for a Single Model
A terminating simulation: runs over a simulated time interval [0
, TE].
A common goal is to estimate:
In general, independent replications are used, each run using a
different random number stream and independently chosen initial
conditions.
E
E
n
i
i
TttYdttY
T
E
Yn
E
=
=
=
0),(outputcontinuousfor,)(1
outputdiscretefor,1
ET
0
1
Computer Science, Informatik 4
Communication and Distributed Systems
Statistical Background
7/21/2019 All in One simulation notes
70/447
Dr. Mesut Gne
16hapter 10. Output Analysis for a Single Model
Important to distinguish within-replication data from across-replication
data.
For example, simulation of a manufacturing system
Two performance measures of that system: cycle time for parts and work
in process (WIP).
Let Yij be the cycle time for thej-thpart produced in the i-threplication.
Across-replication data are formed by summarizing within-replication
data ..iY
R
2
RRRnRR1
2
2
222n2221
1
2
111n1211
H,S,YYYY
H,S,YYYYH,S,YYYY
R
2
1
L
MMMM
L
L
2
Across-Replication
Data
Within-Replication Data
Computer Science, Informatik 4
Communication and Distributed Systems
Statistical Background
7/21/2019 All in One simulation notes
71/447
Dr. Mesut Gne
17hapter 10. Output Analysis for a Single Model
Across Replication:
For example: the daily cycle time averages (discrete time data)
- The average:
- The sample variance:
- The confidence-interval half-width:
Within replication:
For example: the WIP (a continuous time data)
- The average:
- The sample variance:
==R
i
iYR
Y1
... 1
=
=R
i
i YYR
S1
2
...
2 )(1
1
R
StH R 1,2/ =
= EiT
i
Ei
i dttYT
Y0
. )(1
( ) = EiT
ii
Ei
i dtYtY
T
S0
2
.
2 )(1
Computer Science, Informatik 4
Communication and Distributed Systems
Statistical Background
7/21/2019 All in One simulation notes
72/447
Dr. Mesut Gne
18hapter 10. Output Analysis for a Single Model
Overall sample average, , and the interval replication sample
averages, , are always unbiased estimators of the expected daily
average cycle time or daily average WIP.
Across-replication data are independent (different random numbers)
and identically distributed (same model), but within-replication data
do not have these properties.
..Y
.iY
7/21/2019 All in One simulation notes
73/447
Computer Science, Informatik 4
Communication and Distributed Systems
Confidence Intervals with Specified Precision
7/21/2019 All in One simulation notes
74/447
Dr. Mesut Gne
20hapter 10. Output Analysis for a Single Model
Assume that an initial sample of size
R0 (independent) replications has
been observed.
Obtain an initial estimate
S0
2of the population variance
2
.
Then, choose sample sizeR such thatR
R0:
Since t/2, R-1 z/2, an initial estimate ofR:
R is the smallest integer satisfyingRR0 and
Collect
R - R0additional observations.
The 100(1- ) CI for :
on.distributinormalstandardtheis,2/
2
02/
z
SzR
2
01,2/
StR
R
R
StY R 1,2/..
Computer Science, Informatik 4
Communication and Distributed Systems
Confidence Intervals with Specified Precision
7/21/2019 All in One simulation notes
75/447
Dr. Mesut Gne
21hapter 10. Output Analysis for a Single Model
Call Center Example: estimate the agents utilization over the first 2 hours of
the workday.
Initial sample of size R0 = 4 is taken and an initial estimate of the population varianceis S
02 = (0.072)2 = 0.00518.
The error criterion is = 0.04 and confidence coefficient is 1-= 0.95, hence, the finalsample size must be at least:
For the final sample size:
R = 15 is the smallest integer satisfying the error criterion, soR - R0 = 11 additionalreplications are needed.
After obtaining additional outputs, half-width should be checked.
14.1204.0
00518.0*96.12
22
0025.0 ==
Sz
R 13 14 15
t 0.025, R-1 2.18 2.16 2.14
15.39 15.1 14.83( )201,2/ / St R
Computer Science, Informatik 4
Communication and Distributed Systems
Quantiles
7/21/2019 All in One simulation notes
76/447
Dr. Mesut Gne
22hapter 10. Output Analysis for a Single Model
Here, a proportion or probability is treated as a special case of a
mean.
When the number of independent replications
Y1, ,YR is large
enough that
t
/2,n-1
= z
/2
, the confidence interval for a probability
pis
often written as:
A quantile is the inverse of the probability to the probability
estimation problem:
1
)1( 2/
R
ppzp
Find such that Pr(Y ) = p
p is given
The sample proportion
Computer Science, Informatik 4
Communication and Distributed Systems
Quantiles
7/21/2019 All in One simulation notes
77/447
Dr. Mesut Gne
23hapter 10. Output Analysis for a Single Model
The best way is to sort the outputs and use the (R*p)-
th smallest value,
i.e., find such that100p of the data in a histogram of Yis to the left of
.
Example: If we haveR=10 replications and we want thep = 0.8 quantile,first sort, then estimate by the (10)(0.8) = 8-thsmallest value (round ifnecessary).
5.6
sorted data
7.1
8.8
8.9
9.5
9.7
10.1
12.2 this is our point estimate
12.5
12.9
Computer Science, Informatik 4
Communication and Distributed Systems
Quantiles
7/21/2019 All in One simulation notes
78/447
Dr. Mesut Gne
24hapter 10. Output Analysis for a Single Model
1
)1(
1
)1(where
2/
2/
+=
=
R
ppzpp
R
ppzpp
u
l
Confidence Interval of Quantiles: An approximate (1
-
)100
confidence interval for can be obtained by finding two values
l
and
u
.
lcuts off 100pl%of the histogram (theRpl smallest value of the sorteddata).
u cuts off 100pu%of the histogram (theRpu smallest value of the sorted
data).
Computer Science, Informatik 4
Communication and Distributed Systems
Quantiles
7/21/2019 All in One simulation notes
79/447
Dr. Mesut Gne
25hapter 10. Output Analysis for a Single Model
Example: Suppose
R = 1000reps, to estimate the
p = 0.8quantile
with a 95
confidence interval.
First, sort the data from smallest to largest.
Then estimate of by the (1000)(0.8) = 800-thsmallest value, and thepoint estimate is 212.03.
And find the confidence interval:
The point estimate is
The 95% c.i. is [188.96, 256.79]
aluessmallest v820and780theisc.i.The
82.011000
)8.1(8.96.18.0
78.011000)8.1(8.96.18.0
thth
=
+=
=
=
up
pl
A portion of the 1000
sorted values:
Output Rank
180.92 779
188.96 780
190.55 781
208.58 799
212.03 800
216.99 801
250.32 819
256.79 820
256.99 821
Computer Science, Informatik 4
Communication and Distributed Systems
Output Analysis for Steady-State Simulation
7/21/2019 All in One simulation notes
80/447
Dr. Mesut Gne
26hapter 10. Output Analysis for a Single Model
Consider a single run of a simulation model to estimate a steady-
state or long-run characteristics of the system.
The single run produces observations Y1, Y2, ... (generally the samples of
an autocorrelated time series). Performance measure:
- Independent of the initial conditions.
measurediscretefor,1
1lim
=
=n
i
i
n
Yn
(with probability 1)
measurecontinuousfor,)(10lim
= E
E
T
ET
dttYT
(with probability 1)
7/21/2019 All in One simulation notes
81/447
Computer Science, Informatik 4
Communication and Distributed Systems
Initialization Bias
7/21/2019 All in One simulation notes
82/447
Dr. Mesut Gne
28hapter 10. Output Analysis for a Single Model
Methods to reduce the point-estimator bias caused by using artificial
and unrealistic initial conditions:
Intelligent initialization.
Divide simulation into an initialization phase and data-collection phase.
Intelligent initialization
Initialize the simulation in a state that is more representative of long-runconditions.
If the system exists, collect data on it and use these data to specify morenearly typical initial conditions.
If the system can be simplified enough to make it mathematically solvable,e.g. queueing models, solve the simplified model to find long-run expectedor most likely conditions, use that to initialize the simulation.
Computer Science, Informatik 4
Communication and Distributed Systems
Initialization Bias
7/21/2019 All in One simulation notes
83/447
Dr. Mesut Gne
29hapter 10. Output Analysis for a Single Model
Divide each simulation into two phases:
An initialization phase, from time 0 to time T0.
A data-collection phase, from T0 to the stopping time T0+TE.
The choice of T0 is important:- After T0 , system should be more nearly representative of steady-state behavior.
System has reached steady state: the probability distribution of the systemstate is close to the steady-state probability distribution (bias of responsevariable is negligible).
Computer Science, Informatik 4
Communication and Distributed Systems
Initialization Bias
7/21/2019 All in One simulation notes
84/447
Dr. Mesut Gne
30hapter 10. Output Analysis for a Single Model
M/G/1 queueing example: A total of 10 independent replications were
made.
Each replication beginning in the empty and idle state.
Simulation run length on each replication was T0+TE= 15000 minutes. Response variable: queue length,LQ(t,r) (at time tof ther-th replication).
Batching intervals of 1000 minutes, batch means
Ensemble averages:
To identify trend in the data due to initialization bias
The average corresponding batch means acrossreplications:
The preferred method to determine deletion point.
=
=R
r
rjj YR
Y1
.
1Rreplications
Computer Science, Informatik 4
Communication and Distributed Systems
Initialization Bias
7/21/2019 All in One simulation notes
85/447
Dr. Mesut Gne
31hapter 10. Output Analysis for a Single Model
A plot of the ensemble averages, , versus 1000
j,
forj = 1,2, ,15.
),..( dnY
Computer Science, Informatik 4
Communication and Distributed Systems
Initialization Bias
7/21/2019 All in One simulation notes
86/447
Dr. Mesut Gne
32hapter 10. Output Analysis for a Single Model
Cumulative average sample mean (after deletingdobservations):
Not recommended to determine the initialization phase.
It is apparent that downward bias is present and this bias can be reducedby deletion of one or more observations.
+=
=n
dj
jYdn
dnY
1
...
1),(
Computer Science, Informatik 4
Communication and Distributed Systems
Initialization Bias
7/21/2019 All in One simulation notes
87/447
Dr. Mesut Gne
33hapter 10. Output Analysis for a Single Model
No widely accepted, objective and proven technique to guide how
much data to delete to reduce initialization bias to a negligible level.
Plots can, at times, be misleading but they are still recommended.
Ensemble averages reveal a smoother and more precise trend as the # ofreplications,R, increases.
Ensemble averages can be smoothed further by plotting a movingaverage.
Cumulative average becomes less variable as more data are averaged.
The more correlation present, the longer it takes for to approachsteady state.
Different performance measures could approach steady state at differentrates.
jY.
Computer Science, Informatik 4
Communication and Distributed Systems
Error Estimation
7/21/2019 All in One simulation notes
88/447
Dr. Mesut Gne
34hapter 10. Output Analysis for a Single Model
If {Y1, , Yn} are not statistically independent, then S2/n is a biased
estimator of the true variance.
Almost always the case when {Y1, , Yn} is a sequence of outputobservations from within a single replication (autocorrelated sequence,time-series).
Suppose the point estimator is the sample mean
Variance of is very hard to estimate.
For systems with steady state, produce an output process that isapproximately covariance stationary (after passing the transient phase).
- The covariance between two random variables in the time series dependsonly on the lag, i.e. the number of observations between them.
Y
== n
i iY
nY
11
Computer Science, Informatik 4
Communication and Distributed Systems
Error Estimation
7/21/2019 All in One simulation notes
89/447
Dr. Mesut Gne
35hapter 10. Output Analysis for a Single Model
For a covariance stationary time series, {
Y1, , Yn}:
Lag-kautocovariance is:
Lag-kautocorrelation is:
If a time series is covariance stationary, then the variance of is:
The expected value of the variance estimator is:
Y
),cov(),cov( 11 kiikk YYYY ++ ==
2
kk=
+=
=
1
1
2
121)(n
k
kn
k
nYV
1
1/e wher,)(
2
==
n
cnBYVB
n
SE
c
11 k
Computer Science, Informatik 4
Communication and Distributed Systems
Error Estimation
7/21/2019 All in One simulation notes
90/447
Dr. Mesut Gne
36hapter 10. Output Analysis for a Single Model
a)
Stationary time series Yi exhibiting
positive autocorrelation.
Serie slowly drifts above and then
below the mean.
b)
Stationary time series Yi exhibiting
negative autocorrelation.
c) Nonstationary time series with an
upward trend
kk mostfor0>
kk mostfor0
7/21/2019 All in One simulation notes
91/447
Dr. Mesut Gne
37hapter 10. Output Analysis for a Single Model
The expected value of the variance estimator is:
If Yi are independent, then S2/n is an unbiased estimator of
If the autocorrelationkare primarily positive, then S2/n is biased low as
an estimator of .
If the autocorrelationkare primarily negative, then S2/n is biased high as
an estimator of .
)(YV
ofvariancetheis)(and
1
1/e wher,)(
2
YYV
n
cnBYVB
n
SE
==
)(YV
)(YV
Computer Science, Informatik 4
Communication and Distributed Systems
Replication Method
7/21/2019 All in One simulation notes
92/447
Dr. Mesut Gne
38hapter 10. Output Analysis for a Single Model
Use to estimate point-estimator variability and to construct a confidence
interval.
Approach: makeR replications, initializing and deleting from each one the
same way.
Important to do a thorough job of investigating the initial-condition bias:
Bias is not affected by the number of replications, instead, it is affected only by
deleting more data (i.e., increasing T0) or extending the length of each run (i.e.
increasing TE).
Basic raw output data {
Yrj, r = 1, ..., R; j = 1, , n} is derived by:
Individual observation from within replication r.
Batch mean from within replication rof some number of discrete-time
observations.
Batch mean of a continuous-time process over time intervalj.
7/21/2019 All in One simulation notes
93/447
Computer Science, Informatik 4
Communication and Distributed Systems
Replication Method
7/21/2019 All in One simulation notes
94/447
Dr. Mesut Gne
40hapter 10. Output Analysis for a Single Model
To estimate the standard error of , the sample variance and
standard error:
..Y
R
SYesYRY
RYY
RS
R
r
r
R
r
r =
=
=
==
).(.and1
1)(
1
1..
1
2
..
2
.
1
2
...
2
Mean of the
undeleted
observations
from the r-th
replication.
Mean of
),(,),,(1 dnYdnY R K
Standard error
Computer Science, Informatik 4
Communication and Distributed Systems
Replication Method
7/21/2019 All in One simulation notes
95/447
Dr. Mesut Gne
41hapter 10. Output Analysis for a Single Model
Length of each replication (
n) beyond deletion point (
d):
(n d ) > 10d or TE > 10T0
Number of replications (
R) should be as many as time permits, up to
about 25 replications.
For a fixed total sample size (n), as fewer data are deleted (d):
CI shifts: greater bias.
Standard error of decreases: decrease variance.),(.. dnY
Reducing
bias
Increasing
varianceTrade off
Computer Science, Informatik 4
Communication and Distributed Systems
Replication Method
7/21/2019 All in One simulation notes
96/447
Dr. Mesut Gne
42hapter 10. Output Analysis for a Single Model
The point estimator and standard error are:
The 95%CI for long-run mean queue length is:
A high degree of confidence that the long-run mean queue length is between 4.84and 12.02 (if dand n are large enough).
M/G/1 queueing example:
SupposeR = 10, each of length TE= 15000
minutes, starting at time 0 in the empty and
idle state, initialized for T0 = 2000 minutes
before data collection begins.
Each batch means is the average number of
customers in queue for a 1000-minute interval.
The 1-st two batch means are deleted (d= 2).
( ) 59.1)2,15(..and43.8)2,15( .... == YesY
)59.1(26.242.8)59.1(26.243.8
// 1,2/..1,2/..
+
+
Q
RR
L
RStYRStY
Computer Science, Informatik 4
Communication and Distributed Systems
Sample Size
7/21/2019 All in One simulation notes
97/447
Dr. Mesut Gne
43hapter 10. Output Analysis for a Single Model
To estimate a long-run performance measure, , within
with confidence 100(1
-
)
.
M/G/1 queueing example (cont.):
We know:R0 = 10, d= 2 andS02
= 25.30. To estimate the long-run mean queue length, LQ, within = 2 customers
with 90%confidence (= 10%).
Initial estimate:
Hence, at least 18 replications are needed, next tryR = 18,19, using . We found that:
Additional replications needed isR R0 = 19-10 = 9.
1.172
)30.25(645.12
22
005.0
==
Sz
R
( ) 93.18)4/3.25*73.1(/19 220119,05.0 === StR
( )201,05.0 / StR R
Computer Science, Informatik 4
Communication and Distributed Systems
Sample Size
7/21/2019 All in One simulation notes
98/447
Dr. Mesut Gne
44hapter 10. Output Analysis for a Single Model
An alternative to increasing
Ris to increase total run length
T0+TEwithin each replication.
Approach:
- Increase run length from (T0+TE) to (R/R0)(T0+TE), and- Delete additional amount of data, from time 0 to time (R/R0)T0.
Advantage: any residual bias in the point estimator should be further
reduced.
However, it is necessary to have saved the state of the model at time
T0+TEand to be able to restart the model.
Computer Science, Informatik 4
Communication and Distributed Systems
Batch Means for Interval Estimation
7/21/2019 All in One simulation notes
99/447
Dr. Mesut Gne
45hapter 10. Output Analysis for a Single Model
Using a single, long replication:
Problem: data are dependent so the usual estimator is biased.
Solution: batch means.
Batch means:
divide the output data from 1 replication (after appropriate
deletion) into a few large batches and then treat the means of these batches
as if they were independent.
A continuous-time process, {Y(t), T0 t T0+TE}:
kbatches of size m = TE/ k, batch means:
A discrete-time process
, {
Y
i
, i = d+1,d+2, , n
}:
kbatches of size m = (n d)/k, batch means:
+=
+=jm
mji
dij Ym
Y1)1(
1
+= jm
mjj dtTtY
mY
)1(0 )(
1
Computer Science, Informatik 4
Communication and Distributed Systems
Batch Means for Interval Estimation
YYYYYYYY
7/21/2019 All in One simulation notes
100/447
Dr. Mesut Gne
46hapter 10. Output Analysis for a Single Model
Starting either with continuous-time or discrete-time data, the
variance of the sample mean is estimated by:
If the batch size is sufficiently large, successive batch means will be
approximately independent, and the variance estimator will be
approximately unbiased.
No widely accepted and relatively simple method for choosing an
acceptable batch size
m. Some simulation software does it
automatically.
( )
==
=
=
k
j
jk
j
j
kk
YkY
k
YY
kk
S
1
22
1
22
)1(1
1
kmdmkdmdmdmddd YYYYYYYY ++++++++ ...,,,...,...,,,...,,,...,, 1)1(2111
deleted1Y 2Y kY
Computer Science, Informatik 4
Communication and Distributed Systems
Summary
7/21/2019 All in One simulation notes
101/447
Dr. Mesut Gne
47hapter 10. Output Analysis for a Single Model
Stochastic discrete-event simulation is a statistical experiment.
Purpose of statistical experiment: obtain estimates of the performance measures
of the system.
Purpose of statistical analysis: acquire some assurance that these estimates are
sufficiently precise.
Distinguish: terminating simulations and steady-state simulations.
Steady-state output data are more difficult to analyze
Decisions: initial conditions and run length
Possible solutions to bias: deletion of data and increasing run length
Statistical precision of point estimators are estimated by standard-error or
confidence interval
Method of independent replications was emphasized.
7/21/2019 All in One simulation notes
102/447
Computer Science, Informatik 4
Communication and Distributed Systems
7/21/2019 All in One simulation notes
103/447
Chapter 11
Comparison and Evaluation of Alternative System Designs
Computer Science, Informatik 4
Communication and Distributed Systems
Purpose
7/21/2019 All in One simulation notes
104/447
Dr. Mesut Gne
3hapter 11. Comparison and Evaluation of Alternative Designs
Purpose: comparison of alternative system designs.
Approach: discuss a few of many statistical methods that can be
used to compare two or more system designs.
Statistical analysis is needed to discover whether observed
differences are due to:
Differences in design or
The random fluctuation inherent in the models
Computer Science, Informatik 4
Communication and Distributed Systems
Outline
7/21/2019 All in One simulation notes
105/447
Dr. Mesut Gne
4hapter 11. Comparison and Evaluation of Alternative Designs
For two-system comparisons:
Independent sampling.
Correlated sampling (common random numbers).
For multiple system comparisons:
Bonferroni approach: confidence-interval estimation, screening, and
selecting the best.
Metamodels
Computer Science, Informatik 4
Communication and Distributed Systems
Comparison of Two System Designs
7/21/2019 All in One simulation notes
106/447
Dr. Mesut Gne
5hapter 11. Comparison and Evaluation of Alternative Designs
Goal: compare two possible configurations of a system
Two possible ordering policies in a supply-chain system, two possible
scheduling rules in a job shop.
Approach: the method of replications is used to analyze the output
data.
The mean performance measure for system i
Denoted by i , i = 1,2.
To obtain point and interval estimates for the difference in mean
performance, namely
1
2
.
7/21/2019 All in One simulation notes
107/447
Computer Science, Informatik 4
Communication and Distributed Systems
Comparison of Two System Designs
From replication r of system i, the simulation analyst obtains an estimate
7/21/2019 All in One simulation notes
108/447
Dr. Mesut Gne
7hapter 11. Comparison and Evaluation of Alternative Designs
From replication rof system i, the simulation analyst obtains an estimateYirof the mean performance measure i .
Assuming that the estimators Yirare (at least approximately) unbiased:
1 =E(Y1r), r = 1, , R1; 2 =E(Y2r), r = 1, , R2 Goal: compute a confidence interval for 1 2 to compare the two
system designs
Confidence interval for 1 2:
- If CI is totally to the left of 0, strong evidence for the hypothesis that1 2 < 0 (1 < 2 ).
- If CI is totally to the right of 0, strong evidence for the hypothesis that1 2 > 0 (1 > 2 ).
- If CI contains 0, no strong statistical evidence that one system is better thanthe other
If enough additional data were collected (i.e.,Ri increased), the CI would
most likely shift, and definitely shrink in length, until conclusion of1 < 2 or 1 > 2 would be drawn.
Computer Science, Informatik 4
Communication and Distributed Systems
Comparison of Two System Designs
7/21/2019 All in One simulation notes
109/447
Dr. Mesut Gne
8hapter 11. Comparison and Evaluation of Alternative Designs
In this chapter:
A two-sided 100(1-)% CI for 1 2 always takes the form of:
All three techniques assume that the basic data Yir are approximately
normally distributed.
).(. 2.1.,2/2.1. YYestYY
Sample
mean for
system i
Degree
of
freedom
Standard error
of the estimator
7/21/2019 All in One simulation notes
110/447
Computer Science, Informatik 4
Communication and Distributed Systems
Independent Sampling with Equal Variances
7/21/2019 All in One simulation notes
111/447
Dr. Mesut Gne
10hapter 11. Comparison and Evaluation of Alternative Designs
Different and independent random number streams are used
to simulate the two systems
All observations of simulated system 1 are statistically
independent of all the observations of simulated system 2.
The variance of the sample mean, , is:
For independent samples:
iY.
( ) ( )
21,
2.
. ,iRR
YV
YVi
i
i
i
i ===
( ) ( ) ( )2
22
1
21
2.1.2.1.RR
YVYVYYV
+=+=
7/21/2019 All in One simulation notes
112/447
Computer Science, Informatik 4
Communication and Distributed Systems
Independent Sampling with Unequal Variances
If the assumption of equal variances cannot safely be made,
7/21/2019 All in One simulation notes
113/447
Dr. Mesut Gne
12hapter 11. Comparison and Evaluation of Alternative Designs
an approximate 100(1-) CI can be computed as:
With degrees of freedom:
In this case, the minimum number of replicationsR1 > 7 andR2 > 7 is recommended.
( )( ) ( ) ( ) ( )
intergerantoround,
1//1//
//
2
2
2221
2
121
2
2221
21
+
+=RRSRRS
RSRS
( ) 2
2
2
1
2
12.1...R
S
R
SYYes +=
Computer Science, Informatik 4
Communication and Distributed Systems
Common Random Numbers (CRN)
For each replication, the same random numbers are used to simulate
7/21/2019 All in One simulation notes
114/447
Dr. Mesut Gne
13hapter 11. Comparison and Evaluation of Alternative Designs
both systems R1=R
2=R.
For each replication r, the two estimates, Yr1 and Yr2, are correlated.
However, independent streams of random numbers are used on differentreplications, so the pairs (Yr1 ,Ys2 ) are mutually independent for r s.
Purpose: induce positive correlation between (for eachr) to
reduce variance in the point estimator of .
2.1. ,YY
RRR
YYYVYVYYV
211222
21
2.1.2.1.2.1.
2
,cov2
+=
+=
2.1. YY
Correlation:
12
is positive
Computer Science, Informatik 4
Communication and Distributed Systems
Common Random Numbers (CRN)
Compare variance from independent sampling with variance
7/21/2019 All in One simulation notes
115/447
Dr. Mesut Gne
14hapter 11. Comparison and Evaluation of Alternative Designs
from CRN:
Variance of arising from CRN is less than that of
independent sampling (withR1=R2).2.1. YY
RVV INDCRN21122 =
7/21/2019 All in One simulation notes
116/447
Computer Science, Informatik 4
Communication and Distributed Systems
Common Random Numbers (CRN)
It is never enough to simply use the same seed for the random-
7/21/2019 All in One simulation notes
117/447
Dr. Mesut Gne
16hapter 11. Comparison and Evaluation of Alternative Designs
number generator(s):
The random numbers must be synchronized: each random number used
in one model for some purpose should be used for the same purpose inthe other model.
Example: if the i-thrandom number is used to generate a service time at
work station 2 for the 5-tharrival in model 1, the i-thrandom number
should be used for the very same purpose in model 2.
7/21/2019 All in One simulation notes
118/447
Computer Science, Informatik 4
Communication and Distributed Systems
Common Random Numbers (CRN)
Vehicle inspection example (cont.)
7/21/2019 All in One simulation notes
119/447
Dr. Mesut Gne
18hapter 11. Comparison and Evaluation of Alternative Designs
Each replication run of 16 hours
Model 1
Model 2 with independen random numbers
Model 2 with common random numbers
without synchronisation
Model 2 with common random numberswith synchronisation
7/21/2019 All in One simulation notes
120/447
Computer Science, Informatik 4
Communication and Distributed Systems
CRN with Specified Precision
Goal: The error in our estimate of
1
2
to be less than .
7/21/2019 All in One simulation notes
121/447
Dr. Mesut Gne
20hapter 11. Comparison and Evaluation of Alternative Designs
Approach: determine the of replicationsR such that the half-width of
CI:
Vehicle inspection example (cont.):
R0 = 10, complete synchronization of random numbers
yield 95% CI:
Suppose = 0.5 minutes for practical significance, we knowR is thesmallest integer satisfyingRR0 and:
Since , a conservation estimate of Ris:
Hence, 35 replications are needed (25 additional).
= 2.1.,2/ .. YYestH
minutes9.04.0
2
1,2/
DR StR
2
1,2/ 0
DR StR1,2/1,2/ 0 RR tt
7/21/2019 All in One simulation notes
122/447
7/21/2019 All in One simulation notes
123/447
xipxijxi2xi1
yi
i xipxijxi2xi1
yi
i
7/21/2019 All in One simulation notes
124/447
p xij j=1,2,,p
yi i=1,2,,n
i p xij
yi
7/21/2019 All in One simulation notes
125/447
ServerWaiting line
Calling population
7/21/2019 All in One simulation notes
126/447
ServerWaiting line
Calling populationti
ti+1
Arrivals
7/21/2019 All in One simulation notes
127/447
The interarrival and servicetimes are taken from
distributions!
7/21/2019 All in One simulation notes
128/447
The simulation run isbuild by meshingclock, arrival and
service times!
Chronological ordering of events
7/21/2019 All in One simulation notes
129/447
Number of customers in the system
7/21/2019 All in One simulation notes
130/447
7/21/2019 All in One simulation notes
131/447
min74.1100
174
customersofNumber
queueintimeWaiting===
w
46046waithocustomer wofNumber
)wait( ===p
7/21/2019 All in One simulation notes
132/447
46.0100customersofNunber
)wait( p
24.0418
101
run timeSimulation
serveroftimeIdle
server)idle( ===
p
min17.3100
317
customersofNumber
timeService===
s
min2.3605.0202.0101.0)()(0
=+++==
=
L
s
spssE
min19.499
415
1-arrivalsofNumber
arrivalsbetweenTimes===
min5.42
81
2)( =
+=
+=
baE
min22.354
174
that waitcustomersofNumber
queueintimeWaiting===
waitedw
min91.4100
491
customersofNumber
systeminspendcustomersTime===
t
min91.417.374.1 =+=+= swt
Average of 50 Trials
7/21/2019 All in One simulation notes
133/447
Histogram for the Average Customer Waiting Time
0
2
15
17
7
5
2
1
0 0
1
0
2
4
6
8
10
12
14
16
18
0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 >4.5
Average customer waiting time [min]
Occurrences(No.ofTrials)
7/21/2019 All in One simulation notes
134/447
Computer Science, Informatik 4Communication and Distributed Systems
7/21/2019 All in One simulation notes
135/447
Simulation
Discrete-Event System Simulation
Dr. Mesut Gne
Computer Science, Informatik 4Communication and Distributed Systems
7/21/2019 All in One simulation notes
136/447
Chapter 3
General Principles
Computer Science, Informatik 4Communication and Distributed Systems
General Principles Introduction
Framework for modeling systems by discrete-event
simulation
7/21/2019 All in One simulation notes
137/447
Dr. Mesut Gne3Chapter 3. General Principles
A system is modeled in terms of its state at each point in time
This is appropriate for systems where changes occur only atdiscrete points in time
Computer Science, Informatik 4Communication and Distributed Systems
Concepts in Discrete-Event Simulation
Concepts of dynamic, stochastic systems that change in a discretemanner
7/21/2019 All in One simulation notes
138/447
Dr. Mesut Gne4Chapter 3. General Principles
A record of an event to occur at the current or some future time, along with any
associated data necessary to execute the event.
Event notice
An instantaneous occurrence that changes the state of a system.Event
A collection of associated entities ordered in some logical fashion in a waiting line.
Holds entities and event notices
Entities on a list are always ordered by some rule, e.g. FIFO, LIFO, or ranked by
some attribute, e.g. priority, due date
List, Set
The properties of a given entity.Attributes
An object in the system that requires explicit representation in the model, e.g., people,
machines, nodes, packets, server, customer.
Entity
A collection of variables that contain all the information necessary to describe the
system at any time.
System state
An abstract representation of a system, usually containing structural, logical, or
mathematical relationships that describe the system.
Model
A collection of entities that interact together over time to accomplish one or more
goals.
System
Computer Science, Informatik 4Communication and Distributed Systems
Concepts in Discrete-Event Simulation
A duration of time of specified length which is known when it beginsActivity
A list of event notices for future events, ordered by time of occurrence; known as the
future event list (FEL).
Always ranked by the event time
Event list
7/21/2019 All in One simulation notes
139/447
Dr. Mesut Gne5Chapter 3. General Principles
A variable representing the simulated time.Clock
A duration of time of unspecified indefinite length, which is not known until it ends.
Customers delay in waiting line depends on the number and service times of othercustomers.
Typically a desired output of the simulation run.
Delay
A duration of time of specified length, which is known when it begins.
Represents a service time, interarrival time, or any other processing time whose
duration has been characterized by the modeler. The duration of an activity can bespecified as:
Deterministic Always 5 time units
Statistical Random draw from {2, 5, 7}
A function depending on system variables and entities
The duration of an activity is computable when it begins
The duration is not affected by other events
To track activities, an event notice is created for the completion time, e.g., let
clock=100 and service with duration 5 time units is starting
Schedule an end of service-event for clock + 5 = 105
Activity
7/21/2019 All in One simulation notes
140/447
Computer Science, Informatik 4Communication and Distributed Systems
Concepts in Discrete-Event Simulation
Activity vs. Delay
7/21/2019 All in One simulation notes
141/447
Dr. Mesut Gne7Chapter 3. General Principles
A1 A2 A3D1 D2
Activity1 Activity2
Delay
Delay
t
Computer Science, Informatik 4Communication and Distributed Systems
Concepts in Discrete-Event Simulation
A model consists of static description of the model and
the dynamic relationships and interactions between the components
7/21/2019 All in One simulation notes
142/447
Dr. Mesut Gne8Chapter 3. General Principles
Some questions that need to be answered for the dynamic behavior Events
- How does each event affect system state, entity attributes, and set contents?
Activities- How are activities defined?
- What event marks the beginning or end of each activity?
- Can the activity begin regardless of system state, or is its beginning conditioned on the
system being in a certain state? Delays
- Which events trigger the beginning (and end) of each delay?
- Under what condition does a delay begin or end?
System state initialization- What is the system state at time 0?
- What events should be generated at time 0 to prime the model that is, to get thesimulation started?
7/21/2019 All in One simulation notes
143/447
Computer Science, Informatik 4Communication and Distributed Systems
Event-scheduling/Time-advance algorithm
Future event list (FEL)
All event notices are chronologically ordered in the FEL
At current time t the FEL contains all scheduled events
7/21/2019 All in One simulation notes
144/447
Dr. Mesut Gne10Chapter 3. General Principles
At current time t, the FEL contains all scheduled events
The event times satisfy: t< t1 t2 t3 ... tn The event associated with t1 is the imminent event, i.e., the next
event to occur
Scheduling of an event
- At the beginning of an activity the duration is computed and an end-of-activity event is placed on the future event list
The content of the FEL is changing during simulation run
- Efficient management of the FEL has a major impact on the
performance of a simulation run- Class: Data structures and algorithms
Computer Science, Informatik 4Communication and Distributed Systems
Event-scheduling/Time-advance algorithm
(1,t2) Type 1 event to occur at t2
(3,t1) Type 3 event to occur at t1(5,1,6)t
Future event listStateClockOld system snapshot at time t
Event-scheduling/Time-advance algorithm
Step 1: Remove the event notice for the
7/21/2019 All in One simulation notes
145/447
Dr. Mesut Gne11Chapter 3. General Principles
(2,tn) Type 2 event to occur at tn
...
(1,t3) Type 1 event to occur at t3
(2,tn) Type 2 event to occur at tn
...
(1,t3) Type 1 event to occur at t3
(4,t*) Type 4 event to occur at t*
(1,t2) Type 1 event to occur at t2(5,1,5)t1
Future event listStateClock
New system snapshot at time t1
Step 1: Remove the event notice for the
imminent event from FEL
event (3, t1) in the example
Step 2: Advance Clock to imminent event time
Set clock = t1
Step 3: Execute imminent event
update system state
change entity attributes
set membership as needed
Step 4: Generate future events and place their
event notices on FEL
Event (4, t*)
Step 5: Update statistics and counters
Computer Science, Informatik 4Communication and Distributed Systems
Event-scheduling/Time-advance algorithm
System snapshot at time 0
Initial conditions
Generation of exogenous events
7/21/2019 All in One simulation notes
146/447
Dr. Mesut Gne12Chapter 3. General Principles
Generation of exogenous events
- Exogenous event, is an event which happens outside the system,but impinges on the system, e.g., arrival of a customer
Computer Science, Informatik 4Communication and Distributed Systems
Event-scheduling/Time-advance algorithm
Generation of events
Arrival of a customer
- At t=0 first arrival is generated and scheduled
7/21/2019 All in One simulation notes
147/447
Dr. Mesut Gne13Chapter 3. General Principles
At t 0 first arrival is generated and scheduled
- When the clock is advanced to the time of thefirst arrival, a second arrival is generated
- Generate an interarrival time a*
- Calculate t* = clock+ a*
- Place event notice at t* on the FEL
Bootstrapping
Computer Science, Informatik 4Communication and Distributed Systems
Event-scheduling/Time-advance algorithm
Generation of events
Service completion of a customer
- A customer completes service at t
7/21/2019 All in One simulation notes
148/447
Dr. Mesut Gne14Chapter 3. General Principles
p
- If the next customer is present a new service time s* is generated- Calculate t* = clock + s*
- Schedule next service completion at t*
- Additionally: Service completion event will scheduled at the arrival
time, when there is an idle server
- Service time is an activity
- Beginning service is a conditional event
Conditions: Customer is present and server is idle
- Service completion is a primary event
Computer Science, Informatik 4Communication and Distributed Systems
Event-scheduling/Time-advance algorithm
Generation of events
Alternate generation of runtimes and downtimes
- At time 0, the first runtime will be generated and an end-of-runtime
7/21/2019 All in One simulation notes
149/447
Dr. Mesut Gne15Chapter 3. General Principles
, g
event will be scheduled
- Whenever an end-of-runtime event occurs, a downtime will be
generated, and a end-of-downtime event will be scheduled
- At the end-of-downtime event, a runtime is generated and