View
0
Download
0
Category
Preview:
Citation preview
1Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Telematics 2 &
Performance Evaluation
Chapter 7
Short Probability Primer
& Obtaining Results
(Acknowledgement: these slides have mostly been compiled from [Kar04, Rob01, BLK02])
2Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Motivation
q Recall:
q Usually, a simulation program is built to help answer a series of
(quantitative) questions about a system under study
q For this, a model (the simulation program) is built
q This model needs to represent the system under study with sufficient
accuracy concerning the characteristics of the system that are relevant
for the questions of interest
q Typically, this requires modeling of the following:
q System
q Load
q Faults
3Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
System, Load, and Fault Models (1)
q System model:
q Describes the composition of an entire system out of simpler subsystems
q Represents the behavior, the way a system works
q In communication networks:
q Entities communicate by exchanging messages over links
q Protocols implemented in entities, can have parameters like processing
delays, limited queue length
q Links can have parameters like bandwidth and delay
(compiled from [Kar04])
4Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
System, Load, and Fault Models (2)
q Load model:
q Describes the pattern with which requests are made to the system to
perform different kinds of activities
■ When do such requests occur?
■ What is the time between requests?
■ What are the parameters of such a request?
q In communication networks:
q Load is the desire of a �user� to send a packet to another user
■ Example parameter: How big is the packet?
q On a coarser abstraction level:
■ How often are connections established?
■ How much data is transmitted within a connection?
■ What is the mix between different types of connections (QoS) within a
session?
■ ...
5Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
System, Load, and Fault Models (3)
q Fault model:
q Describes which parts of a system can deviate from their
proscribed/desired behavior, and in which form
■ When do such deviations occur?
■ Can they be repeated (are faulty entities repaired)?
■ How long do deviations last?
■ Which kind of faulty behavior occurs?
q In communication networks:
q Entities can be faulty (e.g., a node can crash, often considered a
permanent error)
q Communication links can be faulty (e.g., some bits are not transmitted
correctly, due to electromagnetic noise, usually considered a transient
error)
6Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
System, Load, and Fault Models (4)
q Neither the arrival of requests nor the occurrence of faults can be
described deterministically
q Random distributions needed to model these events along with their
parameters:
q What distributions are available, appropriate and easy to use in
simulations?
q How can random numbers be generated such that the simulated events
occur according to these distributions? (Based on general-purpose
random number generators?)
q How well do standard distributions match observed behavior of real
(communication) systems? How to choose parameters for distributions to
model real systems?
q What to do if no simple standard distributions can be found that match a
real system’s behavior?
q Is it enough to just look at distributions?
q By the way, what exactly is a “distribution”???
q We will first review some basics of probability...
7Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Short Review of Some Probability Basics
q Probability is a numerical measure of the likelihood that an
event will occur.
q Probability values are always assigned on a scale from 0 to 1.
q A probability near 0 indicates an event is very unlikely to occur.
q A probability near 1 indicates an event is almost certain to occur.
q A probability of 0.5 indicates the occurrence of the event is just
as likely as it is unlikely.
0 1.5
Increasing Likelihood of Occurrence
Probability:
The occurrence of the event is
just as likely as it is unlikely.
(compiled from [Rob01])
8Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
An Experiment and Its Sample Space
q An experiment is any process that generates well-defined
outcomes.
q The sample space for an experiment is the set of all
experimental outcomes.
q A sample point is an element of the sample space, any one
particular experimental outcome.
q Examples:
Experiment
q Draw a card from a pack
q Telephone sales call
q First number drawn in
National Lottery
Outcomes
q {Ace hearts, 2 hearts,
…, King of spades}
q Sale or no sale
q {1,2,3,…,49}
9Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Constructing Sample Spaces
q A good way to construct the sample space is to write down examples
of typical outcomes and try to identify the complete set
q Example:
Toss a coin four times
One typical outcome is four consecutive heads (H,H,H,H), another is
a head, tail, head and head (H,T,H,H)
A little thought results in identifying the sample space as the set of all
such 4-tuples
S={ (H,H,H,H), (H,H,H,T), (H,H,T,H), (H,T,H,H),
(T,H,H,H), (H,H,T,T), (H,T,H,T), (H,T,T,H),
(T,H,T,H), (T,T,H,H), (T,H,H,T), (H,T,T,T),
(T,H,T,T), (T,T,H,T), (T,T,T,H), (T,T,T,T) }
10Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
A Counting Rule for Multiple-Step Experiments
q If an experiment consists of a sequence of k steps in which
there are n1 possible results for the first step, n2 possible results
for the second step, and so on, then the total number of
experimental outcomes is given by (n1)(n2) . . . (nk)
q A helpful graphical representation of a multiple-step experiment
is a tree diagram
11Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Another useful counting rule enables us to count the
number of experimental outcomes when n objects are to
be selected from a set of N objects
q Number of combinations of N objects taken n at a time
where N! = N(N - 1)(N - 2) . . . (2)(1)
n! = n(n - 1)( n - 2) . . . (2)(1)
0! = 1
Counting Rule for Combinations
)!(!!nNn
NnN
CN n -=÷÷
ø
öççè
æ=
12Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Counting Rule for Permutations
A third useful counting rule enables us to count the
number of experimental outcomes when n objects are to
be selected from a set of N objects where the order of
selection is important
q Number of permutations of N objects taken n at a time:
n)!(NN!
nN
n!PN n -=÷÷
ø
öççè
æ=
13Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Assigning Probabilities
q Classical Method:
q Assigning probabilities based on the assumption of equally likely
outcomes
q Relative Frequency Method:
q Assigning probabilities based on experimentation or historical data
q Subjective Method:
q Assigning probabilities based on the assignor’s judgment
q Applied in economics and related sciences
14Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Classical Method
If an experiment has n possible outcomes, this method
would assign a probability of 1/n to each outcome.
q Example:
Experiment: Rolling a die
Sample Space: S = {1, 2, 3, 4, 5, 6}
Probabilities: Each sample point has a 1/6 chance
of occurring.
15
On
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Relative Frequency of an Outcome
q Suppose that, in N repetitions of the experiment, the outcome O,
occurs times.
The relative frequency of O is,
We can think of the probability of O as the value to which the relative
frequency settles down as N gets larger and larger.
NnO
16Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Events and Their Probability
q An event is a collection of sample points
q The probability of any event is equal to the sum of the
probabilities of the sample points in the event
q If we can identify all the sample points of an experiment and
assign a probability to each, we can compute the probability of
an event
q There are some basic probability relationships that can be used
to compute the probability of an event without knowledge of all
the sample point probabilities:
q Complement of an Event
q Union of Two Events
q Intersection of Two Events
q Mutually Exclusive Events
17Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Complement of an Event
q The complement of event A is defined to be the event consisting
of all sample points that are not in A
q The complement of A is denoted by Ac
q The Venn diagram below illustrates the concept of a
complement
Event A Ac
Sample Space S
18Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
q The union of events A and B is the event containing all sample
points that are in A or B or both
q The union is denoted by A È B
q The union of A and B is illustrated below
Sample Space S
Event A Event B
Union of Two Events
19Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Intersection of Two Events
q The intersection of events A and B is the set of all sample points
that are in both A and B
q The intersection is denoted by A Ç Bq The intersection of A and B is the area of overlap in the
illustration below
Sample Space S
Event A Event B
Intersection
20Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Addition Law
q The addition law provides a way to compute the probability of
event A, or B, or both A and B occurring
q It is written as:
P(A È B) = P(A) + P(B) - P(A Ç B)
21Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Mutually Exclusive Events
q Two events are said to be mutually exclusive if the events have
no sample points in common
q That is, two events are mutually exclusive if, when one event
occurs, the other cannot occur
Sample
Space S
Event BEvent A
q Addition Law for Mutually Exclusive Events:
P(A È B) = P(A) + P(B)
22Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Conditional Probability
q The probability of an event given that another event has
occurred is called a conditional probability
q The conditional probability of A given B is denoted by P(A|B)
q A conditional probability is computed as follows:
P PP
( | ) ( )( )
A B A BB
=Ç
23Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Multiplication Law
q The multiplication law provides a way to compute the probability
of an intersection of two events
q The law is written as:
P(A Ç B) = P(B)P(A|B)
24Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Independent Events
q Events A and B are independent if P(A|B) = P(A)
q Multiplication Law for Independent Events
P(A Ç B) = P(A)P(B)
q The multiplication law also can be used as a test to see if two
events are independent
25Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Random Variables
q A random variable is a numerical expression of the outcome of an
experiment
q Example 1: toss a die twice; count the number of times the number 4
appears (0, 1 or 2 times)
q Example 2: toss a coin; assign $10 to head and -$30 to a tail
q Discrete random variable:
q Can only take discrete values
q Obtained by counting (0, 1, 2, 3, etc.)
q Usually a finite number of different values
q E.g., toss a coin 5 times; count the number of tails (0, 1, 2, 3, 4, or 5 times)
(compiled from [BLK02])
26Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Discrete Probability Distribution
q A discrete probability distribution is the list of all possible [Xj , P(Xj)]
pairs, with:
q Xj = Value of random variable
q P(Xj) = Probability associated with value
q This list can be visualized with a histogram
q Mutually Exclusive (Nothing in Common)
q Collective Exhaustive (Nothing Left Out)
( ) ( )0 1 1j jP X P X£ £ =å
27Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Basic Summary Measures (1)
q Expected Value (The Mean):
q Weighted average of the probability distribution
q E.g., toss 2 coins, count the number of tails, compute expected
value:
( ) ( )j jj
E X X P Xµ = =å
( )
( ) ( ) ( ) ( ) ( ) ( ) 0 .25 1 .5 2 .25 1
j jjX P Xµ =
= + + =
å
28Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Basic Summary Measures (2)
q Variance:
q Weighted average squared deviation about the mean
q E.g., Toss 2 coins, count number of tails, compute variance:
q The standard deviation is the square root of the variance:
( ) ( ) ( )222j jE X X P Xs µ µé ù= - = -ë û å
( ) ( )( ) ( ) ( ) ( ) ( ) ( )
22
2 2 2 0 1 .25 1 1 .5 2 1 .25 .5
j jX P Xs µ= -
= - + - + -
=
å
2ss =
29Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Other Summary Measures
q Coefficient of Variation:
q Normalizes standard deviation:
q Median & Percentiles
q Pro: Better protection against outliers
q Con: Better protection against outliers
)()(
XEXC s
=
30Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Binomial Probability Distribution
q �n� Identical Trials
q E.g., 15 tosses of a coin; 10 light bulbs taken from a warehouse
q 2 Mutually Exclusive Outcomes on Each Trial
q E.g., Heads or tails in each toss of a coin; defective or not defective light
bulb
q Trials are Independent
q The outcome of one trial does not affect the outcome of the other
q Constant Probability for Each Trial
q E.g., Probability of getting a tail is the same each time we toss the coin
q 2 Sampling Methods
q Infinite population without replacement
q Finite population with replacement
31Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Binomial Probability Distribution Function
P X( ) = n!X ! n− X( )!
pX 1− p( )n−X
P X( ) : probability of X successes given n and p
X : number of "successes" in sample X = 0,1,,n( )p : the probability of each "success"n : sample size
Tails in 2 Tosses of Coin
X P(X)0 1/4 = .25
1 2/4 = .50
2 1/4 = .25
32Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Binomial Distribution Characteristics
q Mean
q E.g.,
q Variance and
Standard Deviation
q
q E.g.,
( )E X npµ = =
( )5 .1 .5npµ = = =
n = 5 p = 0.1
0.2.4.6
0 1 2 3 4 5X
P(X)
( ) ( )( )1 5 .1 1 .1 .6708np ps = - = - =
( )( )
2 1
1
np p
np p
s
s
= -
= -
33Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Geometric Distribution
q Number of Bernoulli trials until we have a “success”
q E.g. number of retransmits
q PMF:
q Mean:
q Variance:
P (X) = (1� p)xp<latexit sha1_base64="ipHQQw2s3RiwgaAhAg+labDD2KI=">AAACCnicbVDLTgIxFO3gC/GFunQzkZjAQjKDJroxIbpxiYk8EhhJp9yBhk6nth0jmfAH7t3qL7gzbv0J/8DPsMAsFDzJTU7Oube39/iCUaUd58vKLC2vrK5l13Mbm1vbO/ndvYaKYkmgTiIWyZaPFTDKoa6pZtASEnDoM2j6w6uJ33wAqWjEb/VIgBfiPqcBJVgbyasVW6WLonssSnePopsvOGVnCnuRuCkpoBS1bv6704tIHALXhGGl2q4jtJdgqSlhMM51YgUCkyHuQ9tQjkNQXjL99Ng+MkrPDiJpimt7qv6eSHCo1Cj0TWeI9UDNexPxX49TAoHEZG6/Ds69hHIRa+Bktj6Ima0je5KL3aMSiGYjQzCR1FxgkwE2z2iTXs5E484HsUgalbJ7Uq7cnBaql2lIWXSADlERuegMVdE1qqE6IugePaMX9Go9WW/Wu/Uxa81Y6cw++gPr8wdT0pnR</latexit>
µ =1
p<latexit sha1_base64="URyUeabCyMYPI6N2J37wcZjZtn0=">AAACEHicbVDLSsNAFL3xWesr1aWbwSK4KkkVdCMU3bisYB/QhDKZTtqhk0mYmSgl9Cfcu9VfcCdu/QP/wM9w0mahrQcGDufcx9wTJJwp7Thf1srq2vrGZmmrvL2zu7dvVw7aKk4loS0S81h2A6woZ4K2NNOcdhNJcRRw2gnGN7nfeaBSsVjc60lC/QgPBQsZwdpIfbviRSm6Ql4oMcncaZZM+3bVqTkzoGXiFqQKBZp9+9sbxCSNqNCEY6V6rpNoP8NSM8LptOyliiaYjPGQ9gwVOKLKz2Zfn6ITowxQGEvzhEYz9XdHhiOlJlFgKiOsR2rRy8V/PcEIzY9a2K/DSz9jIkk1FWS+Pkw50jHK00EDJinRfGIIJpKZCxAZYTNGmwzLJhp3MYhl0q7X3LNa/e682rguQirBERzDKbhwAQ24hSa0gMAjPMMLvFpP1pv1bn3MS1esoucQ/sD6/AHBDZzT</latexit>
https://www.probabilitycourse.com�2 =
1� p
p2<latexit sha1_base64="in36lAwmDKNr1Z3sY6Ts3jtBuuM=">AAACGXicbVDLSsNAFJ3UV62vqDvdDBbBjSWJgm6EohuXFewDmrRMppN26GQSZiZCCQH/w71b/QV34taVf+BnOGmz0NYDA4dz7p3LOX7MqFSW9WWUlpZXVtfK65WNza3tHXN3ryWjRGDSxBGLRMdHkjDKSVNRxUgnFgSFPiNtf3yT++0HIiSN+L2axMQL0ZDTgGKktNQ3D1xJhyHqOfAKuoFAOLVP4yyNe07WN6tWzZoCLhK7IFVQoNE3v91BhJOQcIUZkrJrW7HyUiQUxYxkFTeRJEZ4jIakqylHIZFeOs2QwWOtDGAQCf24glP190aKQiknoa8nQ6RGct7LxX89TjHJg83dV8Gll1IeJ4pwPDsfJAyqCOY1wQEVBCs20QRhQXUCiEdIf6N0mRVdjT1fxCJpOTX7rObcnVfr10VJZXAIjsAJsMEFqINb0ABNgMEjeAYv4NV4Mt6Md+NjNloyip198AfG5w8BW6AZ</latexit>
34Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Poisson Distribution
q Discrete events (�successes�) occurring in a given area of
opportunity (�interval�)
q �Interval� can be time, length, surface area, etc.
q The probability of a �success� in a given �interval� is the same for
all the �intervals�
q The number of �successes� in one �interval� is independent of
the number of �successes� in other �intervals�
q The probability of two or more �successes� occurring in an
�interval� approaches zero as the �interval� becomes smaller
q E.g., # customers arriving in 15 minutes
q E.g., # defects per case of light bulbs
35Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Poisson Probability Distribution Function
( )
( )!
: probability of "successes" given : number of "successes" per unit: expected (average) number of "successes": 2.71828 (base of natural logs)
XeP XX
P X XX
e
ll
l
l
-
=
E.g., Find the probability of 4
customers arriving in 3 minutes when
the mean is 3.6. ( )
3.6 43.6 .19124!
eP X-
= =
Warning: Sometimes the definition is P (X) =(�t)X
X!e��t
36Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Poisson Distribution Characteristics
q Mean
q Standard Deviation
and Variance
( )
( )1
N
i ii
E X
X P X
µ l
=
= =
=å l = 0.5
l = 6
0.2.4.6
0 1 2 3 4 5X
P(X)
0.2.4.6
0 2 4 6 8 10X
P(X)2 s l s l= =
37Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Continuos Random Variables
q Continuous random variable:
q Values from interval of numbers (absence of gaps)
q This implies, that for every single value, the probability that the variable
takes on this value is 0
q Continuous probability distribution:
q Distribution of continuous random variable
q Transition from a histogram to a continuous function
q The probability that the random variable takes on a value in the interval
[a, b] is the area under the function between the values a and b
q Probability density function:
q The derivation of the probability distribution
q Important continuous probability distributions:
q The uniform distribution
q The exponential distribution
q The normal distribution
38Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
The Uniform Distribution
q Properties:
q The probability of occurrence of a value is equally likely to occur anywhere
in the range between the smallest value a and the largest value b
q Also called the rectangular distribution
q Mean:
q Variance:
( )2a b
µ+
=
( )22
12b a
s-
=
39Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
The Uniform Distribution
q The probability density function of the uniform distribution
q Application: Selection of random numbers
q E.g., A wooden wheel is spun on a horizontal surface and allowed to
come to rest. What is the probability that a mark on the wheel will point to
somewhere between the North and the East?
( ) ( )1 if f X a X bb a
= £ £-
( ) 900 90 0.25360
P X< < = =!
! !!
40Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Exponential Distributions
( )arrival time 1: any value of continuous random variable
: the population average number of arrivals per unit of time1/ : average time between arrivals
2.71828
XP X eX
e
l
l
l
-< = -
=
E.g., Drivers arriving at a toll bridge; customers arriving at an ATM
q Sometimes also called negative exponential distribution
41
Warning: Sometimes the definition is . Then
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Exponential Distributions
q Describe Time or Distance between Events
q Used for queues
q Density Function
q
q Parameters
q
f(X)
X
l = 0.5l = 2.0
( ) 1 x
f x e l
l-
=
µ l s l= =
l1)( =TEtetf ll -×=)(
42Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Exponential Distributions: Example
( ) ( )( )( )30 5/ 60
30 5 / 60 hoursarrival time > 1 arrival time
1 1
.0821
XP X P X
e
l
-
= =
= - £
= - -
=
q Customers arrive at the checkout line of a supermarket at the rate
of 30 per hour.
q What is the probability that the arrival time between consecutive
customers will be greater than 5 minutes?
43
The Beta distribution
q Often use in the absence of better knowledge
q Distributed between two distinct points
q PDF:
q The only
normalizes the function
q For uniform
distribution
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
f(x) =x↵�1 (1� x)��1
B(↵,�)
B(↵,�)
↵ = � = 1
[Stolen from Wikipedia]
44
The Pareto distribution
q Used to model systems with “preferential attachment” or “riches get
richer” effects
q Distributed in
q PDF for
q For low values of k, i.e. <= 2,
the distribution is heavy-tailed
q Used for traffic modelling
q Sizes of ASes etc.
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
[xmin,1]
f(x) =
(k xk
min x�k�1 x � xmin
0 x < xmin
k > 0, xmin > 0
0 0,8 1,6 2,4 3,2 4
0,8
1,6
2,4
3,2
4
E(X) = V ar(X) = 1
45
The Pareto distribution
q Often plotted in log-log-scale
q Observed variables follow a straight line
q Following the “power law”
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
1 10
0,01
0,1
46Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
The Normal Distribution
q Properties:
q �Bell Shaped�
q Symmetrical
q Mean, Median and
Mode are Equal
q Random Variable
Has Infinite Range Mean Median Mode
X
f(X)
µ
47Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
The Mathematical Model
( ) ( )
( )
( )
2(1/ 2) /12
: density of random variable 3.14159; 2.71828
: population mean: population standard deviation: value of random variable
Xf X e
f X Xe
X X
µ s
ps
pµs
- -é ùë û=
» »
-¥ < < ¥
48Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Many Normal Distributions
Varying the parameters s and µ, we obtain different normal distributions
There are an infinite number of Normal distributions (as for all previous)
49Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
The Standardized Normal Distribution
When X is normally distributed with a mean and a standard deviation ,
follows a standardized (normalized) normal distribution with a mean 0 and a
standard deviation 1.
XZ µs-
=
X
f(X)
µ
Z
s
0Zµ =
1Zs =
f(Z)
µ s
50Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Finding Probabilities
Probability is the
area under the
curve!
c dX
f(X)
( ) ?P c X d£ £ =
51Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Which Table to Use?
Infinitely many Normal distributions would mean infinitely
many tables to look up!
52Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Solution: The Cumulative Standardized Normal Distribution
Z .00 .01
0.0 .5000 .5040 .5080
.5398 .5438
0.2 .5793 .5832 .5871
0.3 .6179 .6217 .6255
.5478
.02
0.1 .5478
Cumulative Standardized Normal
Distribution Table (Portion)
Probabilities
Only One Table is Needed
0 1Z Zµ s= =
Z = 0.120
53Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Standardizing Example
6.2 5 0.1210
XZ µs- -
= = =
Normal DistributionStandardized
Normal Distribution
10s = 1Zs =
5µ =6.2 X Z
0Zµ =0.12
54Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Example:
Normal DistributionStandardized
Normal Distribution
10s = 1Zs =
5µ =7.1 X Z0Zµ =
0.21
2.9 5 7.1 5.21 .2110 10
X XZ Zµ µs s- - - -
= = = - = = =
2.9 0.21-
.0832
( )2.9 7.1 .1664P X£ £ =
.0832
55Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Z .00 .01
0.0 .5000 .5040 .5080
.5398 .5438
0.2 .5793 .5832 .5871
0.3 .6179 .6217 .6255
.5832.02
0.1 .5478
Cumulative Standardized Normal Distribution Table (Portion) 0 1Z Zµ s= =
Z = 0.210
Example: ( )2.9 7.1 .1664P X£ £ =
56Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Z .00 .01
-0.3 .3821 .3783 .3745
.4207 .4168
-0.1 .4602 .4562 .4522
0.0 .5000 .4960 .4920
.4168.02
-0.2 .4129
Cumulative Standardized Normal
Distribution Table (Portion)
0 1Z Zµ s= =
Z = -0.210
Example: ( )2.9 7.1 .1664P X£ £ =
57Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Back to Simulation...
q When performing a simulation study, we are usually interested in
obtaining answers to some quantitative questions
q One very common question in this respect is:
q What is the mean value of some defined metric of the system under
study?
q Example:
q Packets arrive with an average inter-arrival time of 1/ l at a router
q The router has two outgoing links and arriving packet joins link i with
probability fi
q The service time on link i is µi
µ1
µ2
l
(mostly compiled from [Tow04])
58Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Gathering Performance Statistics
q Average delay at queue i:
q Record Dij : delay of customer j at queue i
q Let Ni be # customers passing through queue i
iii TN g=q Average queue length at i:
iNtotal simulated time
q Throughput at queue i, gi =
i
N
jij
i N
DT
i
å== 1
Little�s Law
(treated later in detail)
59Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Analyzing Output Results (1)
q Each time we run a simulation (using different random number
streams), we will get different output results!
distribution of random numbers
to be used during simulation
(interarrival, service times)
random number sequence 1 simulation output results 1input output
random number sequence 2 simulation output results 2input output
random number sequence M simulation output results Minput output
… … … … … …
60Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Analyzing Output Results (2)
q Example: Delay
experienced by each
customer in queue 2
q W2,n: delay of nth
departing customer from
queue 2
µ1
µ2
l
61Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Analyzing Output Results (3)
q Each run shows variation in
customer delay
q One run different from next
q Statistical characterization of
delay must be made
q Expected delay of n-th
customer
q Behavior as n approaches
infinity
q Average of n customers
µ1
µ2
l
62Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Transient Behavior
q Simulation outputs that depend on initial condition (i.e., output value
changes when initial conditions change) are called transient
characteristics
q �Early� part of simulation
q Example: The first customer entering a supermarket in the morning
will always find an empty queue at the counter
q Later part of simulation less dependent on initial conditions
µ1
µ2
l
63Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Effect of Initial Conditions
q Histogram of delay of 20th
customer, given initially empty
(1000 runs)
q Histogram of delay of 20th
customer, given non-empty
conditions
64Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Steady state behavior
q Output results may converge to limiting �steady state� value if
simulation run �long enough�
q Discard statistics gathered during transient phase, e.g., ignore first k
measurements of delay at queue 2
avg. delay
of packets
[n, n+10]
kN
DT
i
N
kjij
i
i
-=å=
q Pick k so statistic is �approximately
the same� for different random
number streams and remains same
as n increases
avg. of 5 simulations�Knee�
65Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Estimating the Transient Phase (1)
q Perform m independent replications containing n observations each:
{ }miji njX1, ...,,1,=
=
q Calculate mean across all replications:
njXm
Xm
ijij ...,,1,1
1, == å
=
q Calculate overall mean across all replications:
å=
=n
jjXn
X1
1
q Remove first k observations when calculating overall mean:
1...,,1,11
-=-
= å+=
nkXkn
Xn
kjjk
66Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Estimating the Transient Phase (2)
q Compute and plot as a function of k:
XXX k -
q Identify the position k0 of the �knee� and discard the first k0
observations in subsequent runs
q In order to facilitate computation of , it is better to
record the running sum of all prior values instead of
recording the values themselves:
q Individual values can still be easily obtained by simple subtraction
q With this we get:
å=
=k
jjiki XS
1,,
å=
=m
iniSmn
X1
,11
and ( )å=
--
=m
ikinik SS
mknX
1,,
11
kX
67Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals (1)
q Run simulation: get estimate X1 as estimate of performance metrics
of interest
q Repeat simulation M times (each with new set of random numbers),
get X2, … XM – all different!
q Which of X1, … XM is �right�?
q Intuitively, average of M samples should be �better� than choosing
any one of M samples
M
XX
M
jjå
== 1How �confident�are we in X?
68Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals (2)
q We can not get perfect estimate of true mean, µ, with finite # samples
q Instead, we look for bounds: find c1 and c2 such that:
Probability(c1 < µ < c2) = 1 – a
[c1,c2]: confidence interval
100(1-a)%: confidence level
q One approach for finding c1, c2 (suppose a = 0.1)
q Take k samples (e.g., k independent simulation runs)
q Sort
q Find largest value is smallest 5% � c1
q Find smallest value in largest 5% � c2
69Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
q Central Limit Theorem: If samples X1, … XM are independent and from
same population (independent identically distributed, i.i.d.) with
population mean µ and standard deviation s (both finite!), then
M
XX
M
jjå
== 1
is approximately normally distributed with mean µ and standard deviation
the sample mean:
Ms
Confidence Intervals – The Central Limit Theorem
70Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals (3)
q However, we usually do not know the population�s standard
deviation
q So, we estimate it using the sample�s (observed) standard
deviation:
s2 = 1M −1
(Xm − X)2
m=1
M
∑
q Given we can now find upper and lower tails of normal distributions
containing a ⨉ 100% of the mass
X, s2
71Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
q If we calculate confidence
intervals as in the above
recipe for a = 0.05, 95% of
the confidence intervals thus
computed will contain the
true (unknown) population
mean
Interpretation of Confidence Intervals
72Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Computing a Sample�s Variance Simplified
q Computing the sum of squared samples leads to simpler computation:
s2 = 1M −1
(Xi − X)2
i=1
M
∑
=1
M −1Xi2 − 2XiX + X
2( )i=1
M
∑
=1
M −1Xi2 − 2X Xi + X
2 1i=1
M
∑i=1
M
∑i=1
M
∑#
$%
&
'(
=1
M −1Xi2 − 2XMX
i=1
M
∑ +MX 2#
$%
&
'(
=Xi2 −MX 2
i=1
M
∑M −1
But: This may lead to numerical
problems, if numbers get
big and variance is small!
73Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Computing a Sample�s Variance (Numerically Stable)
q In 1962, B. P. Welford proposed a method in an article that has been
incorporated into Donald Knuth�s book „The Art of Programming�(2nd volume, page 232, 3rd edition):
For 2 ≤ k ≤ n, the kth estimate of the variance is:
M1 = x1; S1 = 0
Mk =Mk−1 +xk −Mk−1( )
kSk = Sk−1 + xk −Mk−1( ) ⋅ xk −Mk( )
s2 = Skk −1( )
(see also: http://www.johndcook.com/standard_deviation.html)
74Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
s2 = 1M −1
(Xm − X)2
m=1
M
∑
q Given samples X1, … XM, (e.g., having repeated simulation M
times), compute
M
XX
M
jjå
== 1
95% confidence interval: X ±1.96sM
Confidence Intervals: The Recipe (1)
75Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals: The Recipe (2)
q Why does this work?
q If the Xi have been observed running the same simulation program
with different seed values for the random number generation (with
enough �distance� between the seed values), then they are
independent and identically distributed (i.i.d.) with some mean µ and
variance s2
q With
we have:
X =1
M
MX
i=1
Xi
<latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="KRbDiPMi2Av7/klvnFQUmFgN3OU=">AAACEXicbVDLTgIxFO3gCxAV1m4aiYkLQ2bY6NLEjUtM5JHghHQ6F2joY9IHCZnwA279Ab/GnXHt39gBFgKepOnpub3tPSfJODM2DH+C0sHh0fFJuVI9rVXPzi/qtZ5RTlPoUsWVHiTEAGcSupZZDoNMAxEJh34yeyzq/Tlow5R8sYsMYkEmko0ZJdZLnVG9GbbCFfA+iTakiTYYNYLKa6qoEyAt5cSYYRRmNs6JtoxyWFZfnYGM0BmZwNBTSQSYOF/NucTXXknxWGm/pMUr9W9HToQxC5H4m4LYqdmtFeJ/taGz4/s4ZzJzFiRdfzR2HFuFC9M4ZRqo5QtPCNXMz4rplGhCrY9m66UiRyOdSEBDeovnxdnv2nFIvU0+Ub59Ktreqs8u2k1qn/Tarcjz5xCV0SW6QjcoQnfoAT2hDuoiilL0ht6Dj+Az+FpnXAo2YTfQFoLvX4V7n4E=</latexit><latexit sha1_base64="hOz90MU90fLIcnQxWgdUFH6D4xk=">AAACQXicbZDLShxBFIZPa4zjROPoNptCEQKKdLtJNoLgJpsBhYwOTI9NdfXpmcK6NHURhqYfwqdxG1/Ave/hTgKpHmfhJQeK/uv/61T1+fJKcOvi+DFaWv608nm1s9b9sr7xdbO3tX5htTcMB0wLbYY5tSi4woHjTuCwMkhlLvAyvz5t88sbNJZr9dvNKhxLOlG85Iy6YGW9/VSHuO2uhw05JmlpKKuTpu43JLVeXvWzmh8nzTDj3ay3Gx/G8yIfRbIQu7Cos2wrWksLzbxE5Zig1o6SuHLjmhrHmcCmm3qLFWXXdIKjIBWVaMf1fKqG7AWnIKU2YSlH5u7rjppKa2cyDycldVP7PmvN/2Uj78qf45qryjtU7OWh0gviNGkRkYIbZE7MgqDM8PCvhE1p4OICyDc3tdys8jJHg8UBuWn34Wu8wCKMKSY6tE/lURg1wEveo/ooLo4Ok6DPY+jAN9iB75DADziBX3AGA2BwC3fwB+6jh+gpen7BvBQteG/Dm4r+/gNFkrHq</latexit><latexit sha1_base64="hOz90MU90fLIcnQxWgdUFH6D4xk=">AAACQXicbZDLShxBFIZPa4zjROPoNptCEQKKdLtJNoLgJpsBhYwOTI9NdfXpmcK6NHURhqYfwqdxG1/Ave/hTgKpHmfhJQeK/uv/61T1+fJKcOvi+DFaWv608nm1s9b9sr7xdbO3tX5htTcMB0wLbYY5tSi4woHjTuCwMkhlLvAyvz5t88sbNJZr9dvNKhxLOlG85Iy6YGW9/VSHuO2uhw05JmlpKKuTpu43JLVeXvWzmh8nzTDj3ay3Gx/G8yIfRbIQu7Cos2wrWksLzbxE5Zig1o6SuHLjmhrHmcCmm3qLFWXXdIKjIBWVaMf1fKqG7AWnIKU2YSlH5u7rjppKa2cyDycldVP7PmvN/2Uj78qf45qryjtU7OWh0gviNGkRkYIbZE7MgqDM8PCvhE1p4OICyDc3tdys8jJHg8UBuWn34Wu8wCKMKSY6tE/lURg1wEveo/ooLo4Ok6DPY+jAN9iB75DADziBX3AGA2BwC3fwB+6jh+gpen7BvBQteG/Dm4r+/gNFkrHq</latexit><latexit sha1_base64="6HiQWV4IxgUYo1z7ZaLTke+8/lI=">AAACTHicbVDLahsxFNW4aZo4aeM0y25ETCCQEmaySTaG0G66CaRQP8DjDBrNHVtEj0EPgxnmI/o13bY/kH3+I7sQqMb2IrZzQejonHsk3ZMWnBkbho9B493W++0PO7vNvf2Pnw5ah597RjlNoUsVV3qQEgOcSehaZjkMCg1EpBz66f33Wu9PQRum5C87K2AkyFiynFFiPZW0zmLl5dpdDircwXGuCS2jqrypcGycuLtJStaJqkHCmkmrHZ6H88KbIFqCNlrWbXIY7MaZok6AtJQTY4ZRWNhRSbRllEPVjJ2BgtB7Moahh5IIMKNyPlWFTzyT4Vxpv6TFc/a1oyTCmJlIfacgdmLWtZp8Sxs6m1+NSiYLZ0HSxUO549gqXEeEM6aBWj7zgFDN/F8xnRCfi/VBrtxU52akEyloyL7iaX32u3YcMj8mHytvn4gLP6oPL1qPahP0Ls4jj3+G7etvyxh30Bd0jE5RhC7RNfqBblEXUfQb/UF/0b/gIXgKnoOXRWsjWHqO0Eo1tv8D9wiyng==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit>
E⇥X
⇤= E
"1
M
MX
i=1
Xi
#=
1
ME
"MX
i=1
Xi
#
=1
M
MX
i=1
E[Xi ] =1
MMµ = µ
<latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit><latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit><latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit><latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit>
76Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals: The Recipe (3)
q Furthermore,
and
q Because of the central limit theorem, for large M the XM are normal
distributed with expected value µ and standard deviation
q In practice, depending on your setting, this gives a good estimation
already for M>30
2
1][ sMXVARMXVAR i
M
ii =×=úû
ùêë
éå=
[ ]MM
ii XVAR
MM
MX
MVAR ===ú
û
ùêë
é å=
22
21
11 ss
Ms
77Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals: The Recipe (4)
q Consider now the random variable
q Z is normal distributed with mean 0 and variance 1
q We now look for and so that
q From the table of the N(0,1) normal distribution we obtain the value
for a given a (~�the amount of confidence we are aiming at�)
q For a = 0.05, we obtain
M
XZ M
sµ-
=
2aZ-
2aZ
aaa -=úûù
êëé ££- 1
22ZZZP
2aZ
96.12=aZ
78Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals: The Recipe (5)
q This leads to our formula for the confidence interval
asµs
asµs
asµ
a
aa
aa
aa
aa
-=úûù
êëé +££-Û
-=úûù
êëé +-£-£--Û
-=úúú
û
ù
êêê
ë
é£
-£-Û
-=úûù
êëé ££-
1
1
1
1
22
22
22
22
MZX
MZXP
MZX
MZXP
ZM
XZP
ZZZP
MM
MM
M
MX Xs96.1±
79Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
q In order to obtain enough i.i.d. observations, we have two simple
alternatives (others exist):
q Independent replications: perform M independent runs with different
seeds (also remember to delete k observations from transient phase):
■ However, if we want to use the central limit theorem, we should at
least perform 30 independent simulation runs...
■ Alternatively, we can use a student-t distribution instead of the
normal distribution if M<30 (see next slides)
q Batch means: take a single run, delete first k observations, divide
remainder into n groups and obtain Xi for i-th group, i = 1,…,n
■ Follow procedure for independent replications
■ Complication due to non-independence of Xis
■ Potential efficiency gain due to deletion of only k observations
Generating Confidence Intervals for Steady State Measures
80Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals with Student-t Distribution (1)
q When the sample size is relatively small, however, the approximation
of the confidence interval with the normal distribution is poor
q In this case, one can make use of the student-t distribution
q If X is the sample mean of a random sample of size M from a random
population having the mean µ and standard deviation S, then the
random variable
has student-t distribution with M-1 degrees of freedom
q Thus, the 100(1-a)% confidence interval of µ can be obtained as
T = X −µS
M
X − tM−1;α 2S
M< µ < X + tM−1;α 2
SM
(source: [Tri02])
81Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
82Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Confidence Intervals with Student-t Distribution (2)
q In the above formula, is defined such that the area under the
t probability density function to its right is equal to , that is
q This value can be read from a table (see next slide)
q Example:
q Estimate the average execution time of a program, that was run six time
with randomly chosen data sets, obtaining sample mean X = 230 ms and a
sample standard deviation of s = 14
q To obtain a 98% confidence interval of the true mean execution time µ, we
read t 5;0.01 from the table of student-t distribution with n-1 = 5 degrees of
freedom to be 3.365, leading to the following confidence interval:
2;1a-n
t2
a
2)(2;1
aa =>
-ntTP
614365.3230
614365.3230 ×
+<<×
- µ
233.249767.210 << µor (with 98% confidence)
83Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Important Values of the Student-t Distribution
dfa = degrees of freedom
84
Thought experiment: Medians and confidence intervals
q Can we use confidence intervals to estimate the position of the true
median of our samples?
q Theory: Why not?
q Median and average value are equal in normal distribution
� Same confidence intervals
q Practice: Why would we use the median then?
q The median is usually used if the distribution is unknown!
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
85
Confidence intervals for median values (I)
q Given n independent samples of some experimental outcome (no
assumption on distribution)
q Observation: The (n/2)th value (sample median) is a good estimator for
the true median of the distribution
q Confidence?
q By definition: independent samples are either larger or smaller than
the true median with probability 0.5
q Probability of each sample being bigger than the true median:
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
X1 X0X4 X3 X2median?
P (median < min(X)) = 0.5n
86
Confidence intervals for median values (II)
q Probability of each sample being bigger than the true median:
q Probability of median being between min(X) and max(X):
q This is a confidence interval!
q Example confidence: 93,75%
q (even though the probability is not predetermined)
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
X1 X0X4 X3 X2 median?
P (median > max(X)) = 0.5n
P (min(X) < median < max(X)) = 1� 2⇥ 0.5n
87
Confidence intervals for median values (III)
q How large is the confidence for
q More general approach: bionomial distribution!
q Calculate probability of each possible outcome:
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
X1 X0X4 X3 X2median? X5
P (X4 < median < X2)
✓6
2
◆= 15
✓6
4
◆= 15
✓6
3
◆= 20
88
Confidence intervals for median values (IV)
q Sum of probabilities:
q In example:
q No assumptions on distribution required!
Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
P (Xx < median < Xy) =y�1X
k=x
✓n
k
◆0.5n
P (X4 < median < X2) =
0.56 ⇥✓✓
6
2
◆+
✓6
3
◆+
✓6
4
◆◆=
0.78125
89Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results
Additional References
[BLK02] M. L. Berenson, D. M. Levine, T. K. Krehbiel. Basic Business Statistics. course
slides to the ninth edition of the book. Prentice-Hall, 2002.
http://myphlip.pearsoncmg.com/cw/mpbookhome.cfm?vbookid=462
[Jain91] R. Jain. The Art of Computer Systems Performance Analysis: Techniques for
Experimental Design, Measurement, Simulation, and Modeling. Wiley, 1991.
[Karl04] H. Karl. Praxis der Simulation. course slides, Technische Universität Berlin.
[Rob01] T. Robinson. Probability. course slides, University of Bath,
http://www.bath.ac.uk/~masar/UNIV0037/probability.ppt
[Tow04] Don Towsley. Network Simulation. slides for course “Advanced Foundations of
Computer Networks�, University of Massachusetts, USA, 2004.
http://www-net.cs.umass.edu/cs653/
[Tri02] K. S. Trivedi. Probability and Statistics with Reliability, Queueing and Computer
Science Applications. Wiley, 2002.
[Var04] A. Varga. OMNeT++: Object-Oriented Discrete Event Simulator.
http://www.omnetpp.org/
Recommended