Performance Evaluation - Startseite TU Ilmenau · Metrics = objectives in this example . Performance Evaluation (WS 14/15): 02 – Introduction to Simulation 5 A simple model ! Map

1 Performance Evaluation (WS 14/15): 02 – Introduction to Simulation

Performance Evaluation Chapter 2

Introduction to Discrete Event Simulation


Overview

!  Problem statement !  Performing a simulation !  Parameters, metrics, and measurements


Problem statement

!  Perform a performance evaluation of a checkout counter in a store !  Objectives: Determine

!  How long do customers have to wait? !  How many customers are in line? !  How much of the time is the cashier busy?


Problem statement

!  What are relevant parts? !  The person at the counter !  People waiting in line

!  What are relevant parameters? !  How do customers arrive at the queue? !  How long does it take to serve a customer? !  How is the queue organized?

!  Metrics = objectives in this example


A simple model

!  Map the real-world system parts to abstract representations

Server

Queue

Customer

New customers Customers leave

system


A simple model – Algorithm

!  How does such a checkout queue work? !  Server is idle (no customer present)

!  If customer arrives, customer is serviced immediately, until completion !  Server is then busy until customer is completely served

!  Server is busy !  If customer arrives, the customer joins the end of the queue !  When server becomes idle (a customer has been finished), server checks

the queue ■  If one or more customer waits in queue, the first customer leaves the

queue and is now serviced ■  If no customer in queue, the server becomes idle


Overview

!  Problem statement !  Performing a simulation

!  Executing a manual simulation !  Extracting the structure of a simulation

!  Parameters, metrics, and measurements


Manual simulation

!  Let us start the simulation at some arbitrary time, say t = 0 !  Initially, the server is idle, no customers are waiting !  Observe the model in its operation as customers enter and leave the

model, as the model changes its state

Server


Manual simulation

!  At some point in time, a customer A arrives, say t = 2.1 !  The server is now busy !  Assume this customer needs 1.2 time units to be served

Server

A is being served


Manual simulation

!  At t = 3.3, the customer leaves the system !  The queue is empty, the server becomes idle again

Server


Manual simulation

!  At t = 4.5, the next customer B arrives !  The server is idle, the customer begins service immediately, the server

is then busy !  Assume this customer needs 3.7 time units for service

Server

B is being served


Manual simulation

!  At t = 4.9, the next customer C arrives at the counter !  Server is busy, customer joins the queue !  Assume this customer will need 1.8 time units for service

Server

B is being served


Manual simulation

!  When does this third customer start service? !  Wait a second: When did the other guy finish?

!  Oh, it arrived at 4.5, it will take 3.7 time units, so it will leave at t = 8.2 !  Maybe it would be more convenient to write down the time the currently

serviced customer will finish (= time the server can accept the next customer)!

Server

Will finish: 8.2

B is being served


Manual simulation

!  Did I mention that customer D will arrive at t = 5.6 (requiring 3.5 units of service time)?

!  So at that time, a new customer enters the queue !  I guess the server was busy at that time?

!  Maybe it would be a good idea to write down the time that is represented by these figures!

Server

Will finish: 8.2 Clock: 5.6

B is being served


Manual simulation

!  By the way, the next customer E will arrive at t = 9.3 !  But that is only after the second customer has already finished its

service !  So have a look first what happens at t = 8.2, then consider the next event !  At 8.2, customer leaves system, next one is taken into service, and that

will finish after (oh, what was it) 1.8 time units

Server

Will finish: N/A Clock: 8.2

B leaves the system


Customer will arrive: 9.3

Manual simulation

!  Start to serve customer C, set the time it will finish !  Now what was the next thing that will happen?

!  Add a customer? Finish customer? !  Maybe it´s a good idea to write down the time the next customer will arrive

as well!

Server

Will finish: 10 Clock: 8.2

C is being served

Note that no time passes!


Manual simulation

!  Compare the time the current customer will finish with the time the next customer will arrive

!  Next event that changes the state of the model is the arrival of a new customer at t = 9.3

!  Set the clock to this time and update the state

Server


C is being served



Manual simulation

!  At t = 9.3, customer E arrives and is put into the queue !  E will need, say, 0.7 time units for processing by the server

!  Write down the arrival of the next customer at, say, t = 12.2 !  Next event to process is the finishing of customer C at t = 10

Server


C is being served



Manual simulation

!  C finishes, remove D from queue and put it on the server !  Write down the finish time of D, which took 3.5 time units

!  Maybe it would be convenient to also write down the time each customer will take once it is being served

!  Next event would be at t = 12.2, arrival of a new customer

Server

Will finish: 13.5 Clock: 10

D is being served


Will require: 0.7


Overview

!  Problem statement !  Performing a simulation

!  Executing a manual simulation !  Extracting the structure of a simulation

!  Parameters, metrics, and measurements


What did we do here?

!  You got the idea! Generalize? !  We observed the model only at the points in time when the state of the

model changed !  These points in time were explicitly represented by the simulation

clock variable !  The state of the model changed due to two different kinds of events:

!  Arrival of customers !  Customers finishing service and leaving the server


What did we do here?

!  Both kinds of events were processed = the state was manipulated according to the algorithms describing the model (cp. Slide 6)

!  Two variables were used to determine which kind of event is the next one (arrival or departure of customer)

!  In a nutshell: we advanced time from one event to the next, checking to see which kind of event would be the next one

!  This is the essence of the �next-event time advance algorithm� !  Note: time was incremented as necessary, not in fixed amounts !  Fixed-increment algorithms are also possible, yet rarely used !  Result: periods of inactivity are skipped with next-event algorithms


Next-event time advance algorithm

!  Initialize simulation clock to 0 !  Determine time of occurrence of future events (might be infinite for

some events) !  Might be arbitrarily many – stored in the future event list

!  As long as there are events to be processed !  Increment the simulation clock to the time of the next, most imminent

event !  Update the system state as required by the occurrence of this event

(usually done by an event routine specific for each kind of event) !  Compute times for future events

Time e1 e2 e3 e4 e5

e6 e7 e8

e9 e10 0


Simulation time and simulated time

!  Carefully distinguish between !  Simulated time:

■  The time as measured by the simulation clock ■  Virtual time within the simulated system ■  Units can be chosen arbitrarily

!  Simulation time: ■  Time that is necessary to run a given simulation ■  Wall clock time ■  Depends on parameters, models, equipment used, accuracy, …

!  Ideally, simulation time should be much shorter than simulated time !  Do not let the common term �simulation clock� confuse you: it is

measuring simulated time


Overview


!  Parameters and metrics !  How to measure typical types of metrics !  On the meaning of measurements


And what about parameters?

!  In this manual simulation, parameters have not been taken into account

!  Parameters are !  Pattern of customer arrival

■  Deterministic modeling impossible ■  Stochastic modeling: consider the time between customers as a

random variable, choose a suitable distribution for this variable ■  Common case: interarrival time of customers is exponentially

distributed, characterized by the distribution�s mean !  Pattern of service times

■  Similar to arrival patterns ■  Service time modeled as a random variable, e.g. also with an

exponential distribution !  Speed of the server

■  Can arbitrarily be set to 1 by scaling the time ■  Direct representation is simple, too ■  In practice, values for server speed are obtained from the server�s

technical specification or by explicit measurements


And what about metrics?

!  Initial goal was to investigate !  Utilization of the server (�what percentage of time is the server busy�) !  Length of the queue (�how much space do we need in the store?�)

■  What does this mean? At most? On average? !  Waiting time of customers

■  Again: At most? On average? Fridays? !  How to capture such information from a simulation?


Overview




Measuring server utilization

!  Directly measuring utilization is difficult !  Simple to measure the absolute time the server has been busy, and at

the end divide it by the total time that has been simulated !  Introduce a counter �busytime�, initialize to zero !  At every event: if the server has been busy in the time before this event,

add the time since the last event to �busytime� ■  Easy to check the �old� server state if this check is done before the

state is updated according to the current event !  Additional counter �time_of_last_event� necessary

■  Trivial to compute time since last event from this ■  Before setting the simulation clock to the new time, store it in �time_of_last_event�


Measuring server utilization serverbusy

Timetime_of_last_event = 0

busy_time = 0

time_of_last_event = 2busy_time = 4




Measuring customer waiting time

!  How much time does a customer spend in the queue? !  When putting customer in queue, mark it with the time this was done !  When retrieving customer from queue, take difference between current

simulation clock and time of entry into queue !  If customer does not enter queue, waiting time is 0



!  Example from above: waiting time for customer C

Server C

4.9

B is being served

Will finish: 8.2 Clock: 4.9


Server D

5.6

C is being served



❑  Waiting time for C is current simulation clock – time of enqueuing = 8.2 – 4.9 = 3.3



!  This yields waiting times for each customer !  How to aggregate this information? !  Build the average!

!  Let Di stand for the delay of customer i (possibly 0) !  Compute the usual arithmetic average of the Di´s

!  Note: (Arithmetic) average over a discrete number of data

∑=

n

iiDn

1

/1


Measuring queue length

!  How does the number of customers in the queue behave?

!  Discrete-valued function over time, changing at arbitrary points in time (customers joining and leaving)

# of customers in

queue

Time

1

2

3



!  Appropriate representation: average of this function over time

!  Area of this curve = 17 between time 0 and 16.9 !  Average length of the queue is hence roughly 1.006

# of customers in

queue

Time

1

2

3


# of customers in

queue

Time

1

2

3


!  How to easily compute this average? !  Many complicated approaches possible "

!  Simplest way !  Compute the total area under the curve, and at the end of the simulation,

divide by the total simulated time !  At every event that manipulates the queue: add the area since the last

event to the total area


Different kinds of metrics

!  Average waiting time of customers !  Average is taken over a discrete set of individuals !  Measured value is time !  Example for a discrete-time metric (or statistic)

!  Average queue length !  Average is continuously taken over time !  Measured value is a number, a discrete metric !  Example for a continuous-time metric (or statistic)


Overview




Meaning of measurements

!  What is the correct interpretation of such simulation-based measurements?

!  Look e.g. at waiting time in queue !  Let Di represent the waiting time as measured for customer i !  This refers to one particular simulation run – or to observations on one

particular day (for the real system)! !  For different simulations/on different days, the Di´s will be different !  Di´s are averaged over n customers to obtain aggregate information, here:

average waiting time in queue !  Other possible aggregations: max, min, proportions, …

!  Both Di´s and their aggregate will change from one set of observations to the next!



!  Why look at aggregated information of measurements? !  Aggregated information gives a more concise representation/

description of the system under study !  It is easier to compare aggregated information from different system/

simulation runs !  The average waiting times in a supermarket on weekdays as opposed to

Saturdays is more informative than waiting times of individual customers (which are not really important)

!  So what about looking at distributions instead of averages? !  Successive values might not be distributed in the same way

!  In the queuing example, D1 is always 0, whereas D2, D3, etc. are not !  Hence, the average of such values are not the usual statistical average

which is usually computed over independent, but identically distributed observations



!  The �truly typical� (in a statistical sense) behavior of the model can be considered to be the �average� over all possible behaviors the model can exhibit

!  Weighted by the probabilities of these behaviors !  Sometimes possible to analytically derive closed-form descriptions for

such behavior



!  What is the relationship between truly typical behavior and the result of a simulation run?

!  How good is such an estimator? How to improve the estimation? What are typical sources of errors?

!  We will look at that in some more detail!

Average siD )(ˆ nd )(ndn observations from one run

Statistically typical

behavior

Estimator for true behavior

Estimates


How long to run simulations?

!  When have sufficiently many observations been collected? !  Depends on the purpose (again) !  Sometimes (rarely): behavior of system for a well-defined finite amount

of time is of interest !  Supermarket closes at 8pm, no matter what !  Simulate for a certain fixed amount of simulated time, or a fixed number of

events !  Problem: quality of estimations is variable


How long to run simulations?

!  Often: certain metrics (depending on input parameters) are of interest !  Simulate as long as it takes for the estimation of these metrics to have

reached a desired quality level !  Correctly computing the quality level is difficult !  Amount of necessary simulated time can vary

!  �Stopping rules� regulate when to stop a simulation !  We will look at that in more detail, too


Conclusion

!  A simple cashier queue served as example for many problems in simulation

!  Performed a manual simulation of such a model !  Next-event time advance algorithm !  How to do bookkeeping for this algorithm !  Difference simulated time and simulation time !  Stopping rules

!  Parameters and metrics !  Relationship between statistical properties of the model as such and

estimations of these properties by simulation !  How to extract measurements from simulations


References [CL99] Christos G. Cassandras and Stephane Lafortune. Introduction to

Discrete Event Systems. Kluwer Academic Publishers, Boston, 1999. [HP03] John L. Hennessy and David A. Patterson. Computer Architecture – A

Quantitative Approach. Morgan Kaufmann, Amsterdam, Boston, 3rd edition, 2003.

[Jain91] Raj Jain. The Art of Computer Systems Performance Analysis – Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley Professional Computing. John Wiley and Sons, New York, Chichester, 1991.

[Karl05] H. Karl. Praxis der Simulation. course slides, Universität Paderborn, 2005.

[Law00] A. M. Law, W. D. Kelton. Simulation Modeling and Analysis. 3rd edition, McGraw-Hill. 2000.

Documents

Performance Evaluation - Startseite TU Ilmenau · Metrics = objectives in this example . Performance Evaluation (WS 14/15): 02 – Introduction to Simulation 5 A simple model ! Map