Telematics 2 & Performance Evaluation · Tele2 / Perf Eval (WS 19/20): 07 –Probability & Results 7 Short Review of Some Probability Basics qProbabilityis a numerical measure of

1Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results

Telematics 2 &

Performance Evaluation

Chapter 7

Short Probability Primer

& Obtaining Results

(Acknowledgement: these slides have mostly been compiled from [Kar04, Rob01, BLK02])


Motivation

q Recall:

q Usually, a simulation program is built to help answer a series of

(quantitative) questions about a system under study

q For this, a model (the simulation program) is built

q This model needs to represent the system under study with sufficient

accuracy concerning the characteristics of the system that are relevant

for the questions of interest

q Typically, this requires modeling of the following:

q System

q Load

q Faults


System, Load, and Fault Models (1)

q System model:

q Describes the composition of an entire system out of simpler subsystems

q Represents the behavior, the way a system works

q In communication networks:

q Entities communicate by exchanging messages over links

q Protocols implemented in entities, can have parameters like processing

delays, limited queue length

q Links can have parameters like bandwidth and delay

(compiled from [Kar04])



q Load model:

q Describes the pattern with which requests are made to the system to

perform different kinds of activities

■ When do such requests occur?

■ What is the time between requests?

■ What are the parameters of such a request?


q Load is the desire of a �user� to send a packet to another user

■ Example parameter: How big is the packet?

q On a coarser abstraction level:

■ How often are connections established?

■ How much data is transmitted within a connection?

■ What is the mix between different types of connections (QoS) within a

session?

■ ...



q Fault model:

q Describes which parts of a system can deviate from their

proscribed/desired behavior, and in which form

■ When do such deviations occur?

■ Can they be repeated (are faulty entities repaired)?

■ How long do deviations last?

■ Which kind of faulty behavior occurs?


q Entities can be faulty (e.g., a node can crash, often considered a

permanent error)

q Communication links can be faulty (e.g., some bits are not transmitted

correctly, due to electromagnetic noise, usually considered a transient

error)



q Neither the arrival of requests nor the occurrence of faults can be

described deterministically

q Random distributions needed to model these events along with their

parameters:

q What distributions are available, appropriate and easy to use in

simulations?

q How can random numbers be generated such that the simulated events

occur according to these distributions? (Based on general-purpose

random number generators?)

q How well do standard distributions match observed behavior of real

(communication) systems? How to choose parameters for distributions to

model real systems?

q What to do if no simple standard distributions can be found that match a

real system’s behavior?

q Is it enough to just look at distributions?

q By the way, what exactly is a “distribution”???

q We will first review some basics of probability...


Short Review of Some Probability Basics

q Probability is a numerical measure of the likelihood that an

event will occur.

q Probability values are always assigned on a scale from 0 to 1.

q A probability near 0 indicates an event is very unlikely to occur.

q A probability near 1 indicates an event is almost certain to occur.

q A probability of 0.5 indicates the occurrence of the event is just

as likely as it is unlikely.

0 1.5

Increasing Likelihood of Occurrence

Probability:

The occurrence of the event is

just as likely as it is unlikely.

(compiled from [Rob01])


An Experiment and Its Sample Space

q An experiment is any process that generates well-defined

outcomes.

q The sample space for an experiment is the set of all

experimental outcomes.

q A sample point is an element of the sample space, any one

particular experimental outcome.

q Examples:

Experiment

q Draw a card from a pack

q Telephone sales call

q First number drawn in

National Lottery

Outcomes

q {Ace hearts, 2 hearts,

…, King of spades}

q Sale or no sale

q {1,2,3,…,49}


Constructing Sample Spaces

q A good way to construct the sample space is to write down examples

of typical outcomes and try to identify the complete set

q Example:

Toss a coin four times

One typical outcome is four consecutive heads (H,H,H,H), another is

a head, tail, head and head (H,T,H,H)

A little thought results in identifying the sample space as the set of all

such 4-tuples

S={ (H,H,H,H), (H,H,H,T), (H,H,T,H), (H,T,H,H),

(T,H,H,H), (H,H,T,T), (H,T,H,T), (H,T,T,H),

(T,H,T,H), (T,T,H,H), (T,H,H,T), (H,T,T,T),

(T,H,T,T), (T,T,H,T), (T,T,T,H), (T,T,T,T) }


A Counting Rule for Multiple-Step Experiments

q If an experiment consists of a sequence of k steps in which

there are n1 possible results for the first step, n2 possible results

for the second step, and so on, then the total number of

experimental outcomes is given by (n1)(n2) . . . (nk)

q A helpful graphical representation of a multiple-step experiment

is a tree diagram


Another useful counting rule enables us to count the

number of experimental outcomes when n objects are to

be selected from a set of N objects

q Number of combinations of N objects taken n at a time

where N! = N(N - 1)(N - 2) . . . (2)(1)

n! = n(n - 1)( n - 2) . . . (2)(1)

0! = 1

Counting Rule for Combinations

)!(!!nNn

NnN

CN n -=÷÷

ø

öççè

æ=


Counting Rule for Permutations

A third useful counting rule enables us to count the

number of experimental outcomes when n objects are to

be selected from a set of N objects where the order of

selection is important

q Number of permutations of N objects taken n at a time:

n)!(NN!

nN

n!PN n -=÷÷

ø

öççè

æ=


Assigning Probabilities

q Classical Method:

q Assigning probabilities based on the assumption of equally likely

outcomes

q Relative Frequency Method:

q Assigning probabilities based on experimentation or historical data

q Subjective Method:

q Assigning probabilities based on the assignor’s judgment

q Applied in economics and related sciences


Classical Method

If an experiment has n possible outcomes, this method

would assign a probability of 1/n to each outcome.

q Example:

Experiment: Rolling a die

Sample Space: S = {1, 2, 3, 4, 5, 6}

Probabilities: Each sample point has a 1/6 chance

of occurring.

15

On

Tele2 / Perf Eval (WS 19/20): 07 – Probability & Results

Relative Frequency of an Outcome

q Suppose that, in N repetitions of the experiment, the outcome O,

occurs times.

The relative frequency of O is,

We can think of the probability of O as the value to which the relative

frequency settles down as N gets larger and larger.

NnO


Events and Their Probability

q An event is a collection of sample points

q The probability of any event is equal to the sum of the

probabilities of the sample points in the event

q If we can identify all the sample points of an experiment and

assign a probability to each, we can compute the probability of

an event

q There are some basic probability relationships that can be used

to compute the probability of an event without knowledge of all

the sample point probabilities:

q Complement of an Event

q Union of Two Events

q Intersection of Two Events

q Mutually Exclusive Events


Complement of an Event

q The complement of event A is defined to be the event consisting

of all sample points that are not in A

q The complement of A is denoted by Ac

q The Venn diagram below illustrates the concept of a

complement

Event A Ac

Sample Space S


q The union of events A and B is the event containing all sample

points that are in A or B or both

q The union is denoted by A È B

q The union of A and B is illustrated below

Sample Space S

Event A Event B

Union of Two Events


Intersection of Two Events

q The intersection of events A and B is the set of all sample points

that are in both A and B

q The intersection is denoted by A Ç Bq The intersection of A and B is the area of overlap in the

illustration below

Sample Space S

Event A Event B

Intersection


Addition Law

q The addition law provides a way to compute the probability of

event A, or B, or both A and B occurring

q It is written as:

P(A È B) = P(A) + P(B) - P(A Ç B)


Mutually Exclusive Events

q Two events are said to be mutually exclusive if the events have

no sample points in common

q That is, two events are mutually exclusive if, when one event

occurs, the other cannot occur

Sample

Space S

Event BEvent A

q Addition Law for Mutually Exclusive Events:

P(A È B) = P(A) + P(B)


Conditional Probability

q The probability of an event given that another event has

occurred is called a conditional probability

q The conditional probability of A given B is denoted by P(A|B)

q A conditional probability is computed as follows:

P PP

( | ) ( )( )

A B A BB

=Ç


Multiplication Law

q The multiplication law provides a way to compute the probability

of an intersection of two events

q The law is written as:

P(A Ç B) = P(B)P(A|B)


Independent Events

q Events A and B are independent if P(A|B) = P(A)

q Multiplication Law for Independent Events

P(A Ç B) = P(A)P(B)

q The multiplication law also can be used as a test to see if two

events are independent


Random Variables

q A random variable is a numerical expression of the outcome of an

experiment

q Example 1: toss a die twice; count the number of times the number 4

appears (0, 1 or 2 times)

q Example 2: toss a coin; assign $10 to head and -$30 to a tail

q Discrete random variable:

q Can only take discrete values

q Obtained by counting (0, 1, 2, 3, etc.)

q Usually a finite number of different values

q E.g., toss a coin 5 times; count the number of tails (0, 1, 2, 3, 4, or 5 times)

(compiled from [BLK02])


Discrete Probability Distribution

q A discrete probability distribution is the list of all possible [Xj , P(Xj)]

pairs, with:

q Xj = Value of random variable

q P(Xj) = Probability associated with value

q This list can be visualized with a histogram

q Mutually Exclusive (Nothing in Common)

q Collective Exhaustive (Nothing Left Out)

( ) ( )0 1 1j jP X P X£ £ =å


Basic Summary Measures (1)

q Expected Value (The Mean):

q Weighted average of the probability distribution

q E.g., toss 2 coins, count the number of tails, compute expected

value:

( ) ( )j jj

E X X P Xµ = =å

( )

( ) ( ) ( ) ( ) ( ) ( ) 0 .25 1 .5 2 .25 1

j jjX P Xµ =

= + + =

å


Basic Summary Measures (2)

q Variance:

q Weighted average squared deviation about the mean

q E.g., Toss 2 coins, count number of tails, compute variance:

q The standard deviation is the square root of the variance:

( ) ( ) ( )222j jE X X P Xs µ µé ù= - = -ë û å

( ) ( )( ) ( ) ( ) ( ) ( ) ( )

22

2 2 2 0 1 .25 1 1 .5 2 1 .25 .5

j jX P Xs µ= -

= - + - + -

=

å

2ss =


Other Summary Measures

q Coefficient of Variation:

q Normalizes standard deviation:

q Median & Percentiles

q Pro: Better protection against outliers

q Con: Better protection against outliers

)()(

XEXC s

=


Binomial Probability Distribution

q �n� Identical Trials

q E.g., 15 tosses of a coin; 10 light bulbs taken from a warehouse

q 2 Mutually Exclusive Outcomes on Each Trial

q E.g., Heads or tails in each toss of a coin; defective or not defective light

bulb

q Trials are Independent

q The outcome of one trial does not affect the outcome of the other

q Constant Probability for Each Trial

q E.g., Probability of getting a tail is the same each time we toss the coin

q 2 Sampling Methods

q Infinite population without replacement

q Finite population with replacement


Binomial Probability Distribution Function

P X( ) = n!X ! n− X( )!

pX 1− p( )n−X

P X( ) : probability of X successes given n and p

X : number of "successes" in sample X = 0,1,,n( )p : the probability of each "success"n : sample size

Tails in 2 Tosses of Coin

X P(X)0 1/4 = .25

1 2/4 = .50

2 1/4 = .25


Binomial Distribution Characteristics

q Mean

q E.g.,

q Variance and

Standard Deviation

q

q E.g.,

( )E X npµ = =

( )5 .1 .5npµ = = =

n = 5 p = 0.1

0.2.4.6

0 1 2 3 4 5X

P(X)

( ) ( )( )1 5 .1 1 .1 .6708np ps = - = - =

( )( )

2 1

1

np p

np p

s

s

= -

= -


Geometric Distribution

q Number of Bernoulli trials until we have a “success”

q E.g. number of retransmits

q PMF:

q Mean:

q Variance:

P (X) = (1� p)xp<latexit sha1_base64="ipHQQw2s3RiwgaAhAg+labDD2KI=">AAACCnicbVDLTgIxFO3gC/GFunQzkZjAQjKDJroxIbpxiYk8EhhJp9yBhk6nth0jmfAH7t3qL7gzbv0J/8DPsMAsFDzJTU7Oube39/iCUaUd58vKLC2vrK5l13Mbm1vbO/ndvYaKYkmgTiIWyZaPFTDKoa6pZtASEnDoM2j6w6uJ33wAqWjEb/VIgBfiPqcBJVgbyasVW6WLonssSnePopsvOGVnCnuRuCkpoBS1bv6704tIHALXhGGl2q4jtJdgqSlhMM51YgUCkyHuQ9tQjkNQXjL99Ng+MkrPDiJpimt7qv6eSHCo1Cj0TWeI9UDNexPxX49TAoHEZG6/Ds69hHIRa+Bktj6Ima0je5KL3aMSiGYjQzCR1FxgkwE2z2iTXs5E484HsUgalbJ7Uq7cnBaql2lIWXSADlERuegMVdE1qqE6IugePaMX9Go9WW/Wu/Uxa81Y6cw++gPr8wdT0pnR</latexit>

µ =1

p<latexit sha1_base64="URyUeabCyMYPI6N2J37wcZjZtn0=">AAACEHicbVDLSsNAFL3xWesr1aWbwSK4KkkVdCMU3bisYB/QhDKZTtqhk0mYmSgl9Cfcu9VfcCdu/QP/wM9w0mahrQcGDufcx9wTJJwp7Thf1srq2vrGZmmrvL2zu7dvVw7aKk4loS0S81h2A6woZ4K2NNOcdhNJcRRw2gnGN7nfeaBSsVjc60lC/QgPBQsZwdpIfbviRSm6Ql4oMcncaZZM+3bVqTkzoGXiFqQKBZp9+9sbxCSNqNCEY6V6rpNoP8NSM8LptOyliiaYjPGQ9gwVOKLKz2Zfn6ITowxQGEvzhEYz9XdHhiOlJlFgKiOsR2rRy8V/PcEIzY9a2K/DSz9jIkk1FWS+Pkw50jHK00EDJinRfGIIJpKZCxAZYTNGmwzLJhp3MYhl0q7X3LNa/e682rguQirBERzDKbhwAQ24hSa0gMAjPMMLvFpP1pv1bn3MS1esoucQ/sD6/AHBDZzT</latexit>

https://www.probabilitycourse.com�2 =

1� p

p2<latexit sha1_base64="in36lAwmDKNr1Z3sY6Ts3jtBuuM=">AAACGXicbVDLSsNAFJ3UV62vqDvdDBbBjSWJgm6EohuXFewDmrRMppN26GQSZiZCCQH/w71b/QV34taVf+BnOGmz0NYDA4dz7p3LOX7MqFSW9WWUlpZXVtfK65WNza3tHXN3ryWjRGDSxBGLRMdHkjDKSVNRxUgnFgSFPiNtf3yT++0HIiSN+L2axMQL0ZDTgGKktNQ3D1xJhyHqOfAKuoFAOLVP4yyNe07WN6tWzZoCLhK7IFVQoNE3v91BhJOQcIUZkrJrW7HyUiQUxYxkFTeRJEZ4jIakqylHIZFeOs2QwWOtDGAQCf24glP190aKQiknoa8nQ6RGct7LxX89TjHJg83dV8Gll1IeJ4pwPDsfJAyqCOY1wQEVBCs20QRhQXUCiEdIf6N0mRVdjT1fxCJpOTX7rObcnVfr10VJZXAIjsAJsMEFqINb0ABNgMEjeAYv4NV4Mt6Md+NjNloyip198AfG5w8BW6AZ</latexit>


Poisson Distribution

q Discrete events (�successes�) occurring in a given area of

opportunity (�interval�)

q �Interval� can be time, length, surface area, etc.

q The probability of a �success� in a given �interval� is the same for

all the �intervals�

q The number of �successes� in one �interval� is independent of

the number of �successes� in other �intervals�

q The probability of two or more �successes� occurring in an

�interval� approaches zero as the �interval� becomes smaller

q E.g., # customers arriving in 15 minutes

q E.g., # defects per case of light bulbs


Poisson Probability Distribution Function

( )

( )!

: probability of "successes" given : number of "successes" per unit: expected (average) number of "successes": 2.71828 (base of natural logs)

XeP XX

P X XX

e

ll

l

l

-

=

E.g., Find the probability of 4

customers arriving in 3 minutes when

the mean is 3.6. ( )

3.6 43.6 .19124!

eP X-

= =

Warning: Sometimes the definition is P (X) =(�t)X

X!e��t


Poisson Distribution Characteristics

q Mean

q Standard Deviation

and Variance

( )

( )1

N

i ii

E X

X P X

µ l

=

= =

=å l = 0.5

l = 6

0.2.4.6

0 1 2 3 4 5X

P(X)

0.2.4.6

0 2 4 6 8 10X

P(X)2 s l s l= =


Continuos Random Variables

q Continuous random variable:

q Values from interval of numbers (absence of gaps)

q This implies, that for every single value, the probability that the variable

takes on this value is 0

q Continuous probability distribution:

q Distribution of continuous random variable

q Transition from a histogram to a continuous function

q The probability that the random variable takes on a value in the interval

[a, b] is the area under the function between the values a and b

q Probability density function:

q The derivation of the probability distribution

q Important continuous probability distributions:

q The uniform distribution

q The exponential distribution

q The normal distribution


The Uniform Distribution

q Properties:

q The probability of occurrence of a value is equally likely to occur anywhere

in the range between the smallest value a and the largest value b

q Also called the rectangular distribution

q Mean:

q Variance:

( )2a b

µ+

=

( )22

12b a

s-

=


The Uniform Distribution

q The probability density function of the uniform distribution

q Application: Selection of random numbers

q E.g., A wooden wheel is spun on a horizontal surface and allowed to

come to rest. What is the probability that a mark on the wheel will point to

somewhere between the North and the East?

( ) ( )1 if f X a X bb a

= £ £-

( ) 900 90 0.25360

P X< < = =!

! !!


Exponential Distributions

( )arrival time 1: any value of continuous random variable

: the population average number of arrivals per unit of time1/ : average time between arrivals

2.71828

XP X eX

e

l

l

l

-< = -

=

E.g., Drivers arriving at a toll bridge; customers arriving at an ATM

q Sometimes also called negative exponential distribution

41

Warning: Sometimes the definition is . Then


Exponential Distributions

q Describe Time or Distance between Events

q Used for queues

q Density Function

q

q Parameters

q

f(X)

X

l = 0.5l = 2.0

( ) 1 x

f x e l

l-

=

µ l s l= =

l1)( =TEtetf ll -×=)(


Exponential Distributions: Example

( ) ( )( )( )30 5/ 60

30 5 / 60 hoursarrival time > 1 arrival time

1 1

.0821

XP X P X

e

l

-

= =

= - £

= - -

=

q Customers arrive at the checkout line of a supermarket at the rate

of 30 per hour.

q What is the probability that the arrival time between consecutive

customers will be greater than 5 minutes?

43

The Beta distribution

q Often use in the absence of better knowledge

q Distributed between two distinct points

q PDF:

q The only

normalizes the function

q For uniform

distribution


f(x) =x↵�1 (1� x)��1

B(↵,�)

B(↵,�)

↵ = � = 1

[Stolen from Wikipedia]

44

The Pareto distribution

q Used to model systems with “preferential attachment” or “riches get

richer” effects

q Distributed in

q PDF for

q For low values of k, i.e. <= 2,

the distribution is heavy-tailed

q Used for traffic modelling

q Sizes of ASes etc.


[xmin,1]

f(x) =

(k xk

min x�k�1 x � xmin

0 x < xmin

k > 0, xmin > 0

0 0,8 1,6 2,4 3,2 4

0,8

1,6

2,4

3,2

4

E(X) = V ar(X) = 1

45

The Pareto distribution

q Often plotted in log-log-scale

q Observed variables follow a straight line

q Following the “power law”


1 10

0,01

0,1


The Normal Distribution

q Properties:

q �Bell Shaped�

q Symmetrical

q Mean, Median and

Mode are Equal

q Random Variable

Has Infinite Range Mean Median Mode

X

f(X)

µ


The Mathematical Model

( ) ( )

( )

( )

2(1/ 2) /12

: density of random variable 3.14159; 2.71828

: population mean: population standard deviation: value of random variable

Xf X e

f X Xe

X X

µ s

ps

pµs

- -é ùë û=

» »

-¥ < < ¥


Many Normal Distributions

Varying the parameters s and µ, we obtain different normal distributions

There are an infinite number of Normal distributions (as for all previous)


The Standardized Normal Distribution

When X is normally distributed with a mean and a standard deviation ,

follows a standardized (normalized) normal distribution with a mean 0 and a

standard deviation 1.

XZ µs-

=

X

f(X)

µ

Z

s

0Zµ =

1Zs =

f(Z)

µ s


Finding Probabilities

Probability is the

area under the

curve!

c dX

f(X)

( ) ?P c X d£ £ =


Which Table to Use?

Infinitely many Normal distributions would mean infinitely

many tables to look up!


Solution: The Cumulative Standardized Normal Distribution

Z .00 .01

0.0 .5000 .5040 .5080

.5398 .5438

0.2 .5793 .5832 .5871

0.3 .6179 .6217 .6255

.5478

.02

0.1 .5478

Cumulative Standardized Normal

Distribution Table (Portion)

Probabilities

Only One Table is Needed

0 1Z Zµ s= =

Z = 0.120


Standardizing Example

6.2 5 0.1210

XZ µs- -

= = =

Normal DistributionStandardized

Normal Distribution

10s = 1Zs =

5µ =6.2 X Z

0Zµ =0.12


Example:

Normal DistributionStandardized

Normal Distribution

10s = 1Zs =

5µ =7.1 X Z0Zµ =

0.21

2.9 5 7.1 5.21 .2110 10

X XZ Zµ µs s- - - -

= = = - = = =

2.9 0.21-

.0832

( )2.9 7.1 .1664P X£ £ =

.0832


Z .00 .01

0.0 .5000 .5040 .5080

.5398 .5438

0.2 .5793 .5832 .5871

0.3 .6179 .6217 .6255

.5832.02

0.1 .5478

Cumulative Standardized Normal Distribution Table (Portion) 0 1Z Zµ s= =

Z = 0.210

Example: ( )2.9 7.1 .1664P X£ £ =


Z .00 .01

-0.3 .3821 .3783 .3745

.4207 .4168

-0.1 .4602 .4562 .4522

0.0 .5000 .4960 .4920

.4168.02

-0.2 .4129

Cumulative Standardized Normal

Distribution Table (Portion)

0 1Z Zµ s= =

Z = -0.210

Example: ( )2.9 7.1 .1664P X£ £ =


Back to Simulation...

q When performing a simulation study, we are usually interested in

obtaining answers to some quantitative questions

q One very common question in this respect is:

q What is the mean value of some defined metric of the system under

study?

q Example:

q Packets arrive with an average inter-arrival time of 1/ l at a router

q The router has two outgoing links and arriving packet joins link i with

probability fi

q The service time on link i is µi

µ1

µ2

l

(mostly compiled from [Tow04])


Gathering Performance Statistics

q Average delay at queue i:

q Record Dij : delay of customer j at queue i

q Let Ni be # customers passing through queue i

iii TN g=q Average queue length at i:

iNtotal simulated time

q Throughput at queue i, gi =

i

N

jij

i N

DT

i

å== 1

Little�s Law

(treated later in detail)


Analyzing Output Results (1)

q Each time we run a simulation (using different random number

streams), we will get different output results!

distribution of random numbers

to be used during simulation

(interarrival, service times)

random number sequence 1 simulation output results 1input output

random number sequence 2 simulation output results 2input output

random number sequence M simulation output results Minput output

… … … … … …



q Example: Delay

experienced by each

customer in queue 2

q W2,n: delay of nth

departing customer from

queue 2

µ1

µ2

l



q Each run shows variation in

customer delay

q One run different from next

q Statistical characterization of

delay must be made

q Expected delay of n-th

customer

q Behavior as n approaches

infinity

q Average of n customers

µ1

µ2

l


Transient Behavior

q Simulation outputs that depend on initial condition (i.e., output value

changes when initial conditions change) are called transient

characteristics

q �Early� part of simulation

q Example: The first customer entering a supermarket in the morning

will always find an empty queue at the counter

q Later part of simulation less dependent on initial conditions

µ1

µ2

l


Effect of Initial Conditions

q Histogram of delay of 20th

customer, given initially empty

(1000 runs)

q Histogram of delay of 20th

customer, given non-empty

conditions


Steady state behavior

q Output results may converge to limiting �steady state� value if

simulation run �long enough�

q Discard statistics gathered during transient phase, e.g., ignore first k

measurements of delay at queue 2

avg. delay

of packets

[n, n+10]

kN

DT

i

N

kjij

i

i

-=å=

q Pick k so statistic is �approximately

the same� for different random

number streams and remains same

as n increases

avg. of 5 simulations�Knee�


Estimating the Transient Phase (1)

q Perform m independent replications containing n observations each:

{ }miji njX1, ...,,1,=

=

q Calculate mean across all replications:

njXm

Xm

ijij ...,,1,1

1, == å

=

q Calculate overall mean across all replications:

å=

=n

jjXn

X1

1

q Remove first k observations when calculating overall mean:

1...,,1,11

-=-

= å+=

nkXkn

Xn

kjjk


Estimating the Transient Phase (2)

q Compute and plot as a function of k:

XXX k -

q Identify the position k0 of the �knee� and discard the first k0

observations in subsequent runs

q In order to facilitate computation of , it is better to

record the running sum of all prior values instead of

recording the values themselves:

q Individual values can still be easily obtained by simple subtraction

q With this we get:

å=

=k

jjiki XS

1,,

å=

=m

iniSmn

X1

,11

and ( )å=

--

=m

ikinik SS

mknX

1,,

11

kX


Confidence Intervals (1)

q Run simulation: get estimate X1 as estimate of performance metrics

of interest

q Repeat simulation M times (each with new set of random numbers),

get X2, … XM – all different!

q Which of X1, … XM is �right�?

q Intuitively, average of M samples should be �better� than choosing

any one of M samples

M

XX

M

jjå

== 1How �confident�are we in X?



q We can not get perfect estimate of true mean, µ, with finite # samples

q Instead, we look for bounds: find c1 and c2 such that:

Probability(c1 < µ < c2) = 1 – a

[c1,c2]: confidence interval

100(1-a)%: confidence level

q One approach for finding c1, c2 (suppose a = 0.1)

q Take k samples (e.g., k independent simulation runs)

q Sort

q Find largest value is smallest 5% � c1

q Find smallest value in largest 5% � c2


q Central Limit Theorem: If samples X1, … XM are independent and from

same population (independent identically distributed, i.i.d.) with

population mean µ and standard deviation s (both finite!), then

M

XX

M

jjå

== 1

is approximately normally distributed with mean µ and standard deviation

the sample mean:

Ms

Confidence Intervals – The Central Limit Theorem



q However, we usually do not know the population�s standard

deviation

q So, we estimate it using the sample�s (observed) standard

deviation:

s2 = 1M −1

(Xm − X)2

m=1

M

∑

q Given we can now find upper and lower tails of normal distributions

containing a ⨉ 100% of the mass

X, s2


q If we calculate confidence

intervals as in the above

recipe for a = 0.05, 95% of

the confidence intervals thus

computed will contain the

true (unknown) population

mean

Interpretation of Confidence Intervals


Computing a Sample�s Variance Simplified

q Computing the sum of squared samples leads to simpler computation:

s2 = 1M −1

(Xi − X)2

i=1

M

∑

=1

M −1Xi2 − 2XiX + X

2( )i=1

M

∑

=1

M −1Xi2 − 2X Xi + X

2 1i=1

M

∑i=1

M

∑i=1

M

∑#

$%

&

'(

=1

M −1Xi2 − 2XMX

i=1

M

∑ +MX 2#

$%

&

'(

=Xi2 −MX 2

i=1

M

∑M −1

But: This may lead to numerical

problems, if numbers get

big and variance is small!


Computing a Sample�s Variance (Numerically Stable)

q In 1962, B. P. Welford proposed a method in an article that has been

incorporated into Donald Knuth�s book „The Art of Programming�(2nd volume, page 232, 3rd edition):

For 2 ≤ k ≤ n, the kth estimate of the variance is:

M1 = x1; S1 = 0

Mk =Mk−1 +xk −Mk−1( )

kSk = Sk−1 + xk −Mk−1( ) ⋅ xk −Mk( )

s2 = Skk −1( )

(see also: http://www.johndcook.com/standard_deviation.html)


s2 = 1M −1

(Xm − X)2

m=1

M

∑

q Given samples X1, … XM, (e.g., having repeated simulation M

times), compute

M

XX

M

jjå

== 1

95% confidence interval: X ±1.96sM

Confidence Intervals: The Recipe (1)



q Why does this work?

q If the Xi have been observed running the same simulation program

with different seed values for the random number generation (with

enough �distance� between the seed values), then they are

independent and identically distributed (i.i.d.) with some mean µ and

variance s2

q With

we have:

X =1

M

MX

i=1

Xi

<latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="KRbDiPMi2Av7/klvnFQUmFgN3OU=">AAACEXicbVDLTgIxFO3gCxAV1m4aiYkLQ2bY6NLEjUtM5JHghHQ6F2joY9IHCZnwA279Ab/GnXHt39gBFgKepOnpub3tPSfJODM2DH+C0sHh0fFJuVI9rVXPzi/qtZ5RTlPoUsWVHiTEAGcSupZZDoNMAxEJh34yeyzq/Tlow5R8sYsMYkEmko0ZJdZLnVG9GbbCFfA+iTakiTYYNYLKa6qoEyAt5cSYYRRmNs6JtoxyWFZfnYGM0BmZwNBTSQSYOF/NucTXXknxWGm/pMUr9W9HToQxC5H4m4LYqdmtFeJ/taGz4/s4ZzJzFiRdfzR2HFuFC9M4ZRqo5QtPCNXMz4rplGhCrY9m66UiRyOdSEBDeovnxdnv2nFIvU0+Ub59Ktreqs8u2k1qn/Tarcjz5xCV0SW6QjcoQnfoAT2hDuoiilL0ht6Dj+Az+FpnXAo2YTfQFoLvX4V7n4E=</latexit><latexit sha1_base64="hOz90MU90fLIcnQxWgdUFH6D4xk=">AAACQXicbZDLShxBFIZPa4zjROPoNptCEQKKdLtJNoLgJpsBhYwOTI9NdfXpmcK6NHURhqYfwqdxG1/Ave/hTgKpHmfhJQeK/uv/61T1+fJKcOvi+DFaWv608nm1s9b9sr7xdbO3tX5htTcMB0wLbYY5tSi4woHjTuCwMkhlLvAyvz5t88sbNJZr9dvNKhxLOlG85Iy6YGW9/VSHuO2uhw05JmlpKKuTpu43JLVeXvWzmh8nzTDj3ay3Gx/G8yIfRbIQu7Cos2wrWksLzbxE5Zig1o6SuHLjmhrHmcCmm3qLFWXXdIKjIBWVaMf1fKqG7AWnIKU2YSlH5u7rjppKa2cyDycldVP7PmvN/2Uj78qf45qryjtU7OWh0gviNGkRkYIbZE7MgqDM8PCvhE1p4OICyDc3tdys8jJHg8UBuWn34Wu8wCKMKSY6tE/lURg1wEveo/ooLo4Ok6DPY+jAN9iB75DADziBX3AGA2BwC3fwB+6jh+gpen7BvBQteG/Dm4r+/gNFkrHq</latexit><latexit sha1_base64="hOz90MU90fLIcnQxWgdUFH6D4xk=">AAACQXicbZDLShxBFIZPa4zjROPoNptCEQKKdLtJNoLgJpsBhYwOTI9NdfXpmcK6NHURhqYfwqdxG1/Ave/hTgKpHmfhJQeK/uv/61T1+fJKcOvi+DFaWv608nm1s9b9sr7xdbO3tX5htTcMB0wLbYY5tSi4woHjTuCwMkhlLvAyvz5t88sbNJZr9dvNKhxLOlG85Iy6YGW9/VSHuO2uhw05JmlpKKuTpu43JLVeXvWzmh8nzTDj3ay3Gx/G8yIfRbIQu7Cos2wrWksLzbxE5Zig1o6SuHLjmhrHmcCmm3qLFWXXdIKjIBWVaMf1fKqG7AWnIKU2YSlH5u7rjppKa2cyDycldVP7PmvN/2Uj78qf45qryjtU7OWh0gviNGkRkYIbZE7MgqDM8PCvhE1p4OICyDc3tdys8jJHg8UBuWn34Wu8wCKMKSY6tE/lURg1wEveo/ooLo4Ok6DPY+jAN9iB75DADziBX3AGA2BwC3fwB+6jh+gpen7BvBQteG/Dm4r+/gNFkrHq</latexit><latexit sha1_base64="6HiQWV4IxgUYo1z7ZaLTke+8/lI=">AAACTHicbVDLahsxFNW4aZo4aeM0y25ETCCQEmaySTaG0G66CaRQP8DjDBrNHVtEj0EPgxnmI/o13bY/kH3+I7sQqMb2IrZzQejonHsk3ZMWnBkbho9B493W++0PO7vNvf2Pnw5ah597RjlNoUsVV3qQEgOcSehaZjkMCg1EpBz66f33Wu9PQRum5C87K2AkyFiynFFiPZW0zmLl5dpdDircwXGuCS2jqrypcGycuLtJStaJqkHCmkmrHZ6H88KbIFqCNlrWbXIY7MaZok6AtJQTY4ZRWNhRSbRllEPVjJ2BgtB7Moahh5IIMKNyPlWFTzyT4Vxpv6TFc/a1oyTCmJlIfacgdmLWtZp8Sxs6m1+NSiYLZ0HSxUO549gqXEeEM6aBWj7zgFDN/F8xnRCfi/VBrtxU52akEyloyL7iaX32u3YcMj8mHytvn4gLP6oPL1qPahP0Ls4jj3+G7etvyxh30Bd0jE5RhC7RNfqBblEXUfQb/UF/0b/gIXgKnoOXRWsjWHqO0Eo1tv8D9wiyng==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit><latexit sha1_base64="opBkqdS4xGG/lrsrnjPWUXxrcro=">AAACTHicbVDLahsxFNU4reO4LztddiNqCoWWMBMK7SZgmk03hhTq2OBxB43mji2sx6CHwQzzEfmabpMfyL7/0V0IVGN7Udu9IHR0zj2S7kkLzowNw99B4+jJ0+Zx66T97PmLl6863dNro5ymMKSKKz1OiQHOJAwtsxzGhQYiUg6jdHFZ66MlaMOU/GFXBUwFmUmWM0qsp5LOh1h5uXaX4wpf4DjXhJZRVQ4qHBsnfg6Skl1E1Thh7aTTC8/CdeFDEG1BD23rKukGJ3GmqBMgLeXEmEkUFnZaEm0Z5VC1Y2egIHRBZjDxUBIBZlqup6rwO89kOFfaL2nxmv3XURJhzEqkvlMQOzf7Wk3+T5s4m3+ZlkwWzoKkm4dyx7FVuI4IZ0wDtXzlAaGa+b9iOic+F+uD3Lmpzs1IJ1LQkH3Ey/rsd+04ZH5MPlPePhfnflQfXrQf1SG4Pj+LPP7+qdf/uo2xhd6gt+g9itBn1Eff0BUaIopu0C90i+6C++BP8BA8blobwdbzGu1Uo/kX+Eiyog==</latexit>

E⇥X

⇤= E

"1

M

MX

i=1

Xi

#=

1

ME

"MX

i=1

Xi

#

=1

M

MX

i=1

E[Xi ] =1

MMµ = µ

<latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit><latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit><latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit><latexit sha1_base64="Vaw1ZGaZcgem9MNhv/aCMGu1scM=">AAAC9HicbVHLattAFB2pr8R9xGmX3UxrGrowQQqFtgtDaCl0Y0ihTgweVYxGV9aQmZGYh8EI/Ul3pdv+TFf9m44cl/qRC0Jnzj1nHudmteDGRtGfILxz9979BweHvYePHj856h8/vTSV0wwmrBKVnmbUgOAKJpZbAdNaA5WZgKvs+mPXv1qANrxSX+2yhkTSueIFZ9R6Ku3//kReEAGFnZEhqbyy26iZtn6l+by0CT4Z4f+aQlPWxG0zbjExTn4bpw0fxe005f/kI7whIsMNb6f33K6DkN7JnmtbvHHHzje89awxka5jpEv7g+g0WhXeB/EaDNC6LtLj4JDkFXMSlGWCGjOLo9omDdWWMwFtjzgDNWXXdA4zDxWVYJJmlX6LX3kmx0Wl/acsXrGbjoZKY5Yy80pJbWl2ex15W2/mbPEuabiqnQXFbg4qnMC2wt0occ41MCuWHlCmub8rZiX1gVg/8K2duqEa5WQGGvIhXnRr/9dOQO6fKeaVt5fyzD/VhxfvRrUPLs9OY4+/vBmcf1jHeICeo5foNYrRW3SOPqMLNEEseB+kQRnwcBF+D3+EP2+kYbD2PENbFf76C+n+7M0=</latexit>



q Furthermore,

and

q Because of the central limit theorem, for large M the XM are normal

distributed with expected value µ and standard deviation

q In practice, depending on your setting, this gives a good estimation

already for M>30

2

1][ sMXVARMXVAR i

M

ii =×=úû

ùêë

éå=

[ ]MM

ii XVAR

MM

MX

MVAR ===ú

û

ùêë

é å=

22

21

11 ss

Ms



q Consider now the random variable

q Z is normal distributed with mean 0 and variance 1

q We now look for and so that

q From the table of the N(0,1) normal distribution we obtain the value

for a given a (~�the amount of confidence we are aiming at�)

q For a = 0.05, we obtain

M

XZ M

sµ-

=

2aZ-

2aZ

aaa -=úûù

êëé ££- 1

22ZZZP

2aZ

96.12=aZ



q This leads to our formula for the confidence interval

asµs

asµs

asµ

a

aa

aa

aa

aa

-=úûù

êëé +££-Û

-=úûù

êëé +-£-£--Û

-=úúú

û

ù

êêê

ë

é£

-£-Û

-=úûù

êëé ££-

1

1

1

1

22

22

22

22

MZX

MZXP

MZX

MZXP

ZM

XZP

ZZZP

MM

MM

M

MX Xs96.1±


q In order to obtain enough i.i.d. observations, we have two simple

alternatives (others exist):

q Independent replications: perform M independent runs with different

seeds (also remember to delete k observations from transient phase):

■ However, if we want to use the central limit theorem, we should at

least perform 30 independent simulation runs...

■ Alternatively, we can use a student-t distribution instead of the

normal distribution if M<30 (see next slides)

q Batch means: take a single run, delete first k observations, divide

remainder into n groups and obtain Xi for i-th group, i = 1,…,n

■ Follow procedure for independent replications

■ Complication due to non-independence of Xis

■ Potential efficiency gain due to deletion of only k observations

Generating Confidence Intervals for Steady State Measures


Confidence Intervals with Student-t Distribution (1)

q When the sample size is relatively small, however, the approximation

of the confidence interval with the normal distribution is poor

q In this case, one can make use of the student-t distribution

q If X is the sample mean of a random sample of size M from a random

population having the mean µ and standard deviation S, then the

random variable

has student-t distribution with M-1 degrees of freedom

q Thus, the 100(1-a)% confidence interval of µ can be obtained as

T = X −µS

M

X − tM−1;α 2S

M< µ < X + tM−1;α 2

SM

(source: [Tri02])



Confidence Intervals with Student-t Distribution (2)

q In the above formula, is defined such that the area under the

t probability density function to its right is equal to , that is

q This value can be read from a table (see next slide)

q Example:

q Estimate the average execution time of a program, that was run six time

with randomly chosen data sets, obtaining sample mean X = 230 ms and a

sample standard deviation of s = 14

q To obtain a 98% confidence interval of the true mean execution time µ, we

read t 5;0.01 from the table of student-t distribution with n-1 = 5 degrees of

freedom to be 3.365, leading to the following confidence interval:

2;1a-n

t2

a

2)(2;1

aa =>

-ntTP

614365.3230

614365.3230 ×

+<<×

- µ

233.249767.210 << µor (with 98% confidence)


Important Values of the Student-t Distribution

dfa = degrees of freedom

84

Thought experiment: Medians and confidence intervals

q Can we use confidence intervals to estimate the position of the true

median of our samples?

q Theory: Why not?

q Median and average value are equal in normal distribution

� Same confidence intervals

q Practice: Why would we use the median then?

q The median is usually used if the distribution is unknown!


85

Confidence intervals for median values (I)

q Given n independent samples of some experimental outcome (no

assumption on distribution)

q Observation: The (n/2)th value (sample median) is a good estimator for

the true median of the distribution

q Confidence?

q By definition: independent samples are either larger or smaller than

the true median with probability 0.5

q Probability of each sample being bigger than the true median:


X1 X0X4 X3 X2median?

P (median < min(X)) = 0.5n

86

Confidence intervals for median values (II)

q Probability of each sample being bigger than the true median:

q Probability of median being between min(X) and max(X):

q This is a confidence interval!

q Example confidence: 93,75%

q (even though the probability is not predetermined)


X1 X0X4 X3 X2 median?

P (median > max(X)) = 0.5n

P (min(X) < median < max(X)) = 1� 2⇥ 0.5n

87

Confidence intervals for median values (III)

q How large is the confidence for

q More general approach: bionomial distribution!

q Calculate probability of each possible outcome:


X1 X0X4 X3 X2median? X5

P (X4 < median < X2)

✓6

2

◆= 15

✓6

4

◆= 15

✓6

3

◆= 20

88

Confidence intervals for median values (IV)

q Sum of probabilities:

q In example:

q No assumptions on distribution required!


P (Xx < median < Xy) =y�1X

k=x

✓n

k

◆0.5n

P (X4 < median < X2) =

0.56 ⇥✓✓

6

2

◆+

✓6

3

◆+

✓6

4

◆◆=

0.78125


Additional References

[BLK02] M. L. Berenson, D. M. Levine, T. K. Krehbiel. Basic Business Statistics. course

slides to the ninth edition of the book. Prentice-Hall, 2002.

http://myphlip.pearsoncmg.com/cw/mpbookhome.cfm?vbookid=462

[Jain91] R. Jain. The Art of Computer Systems Performance Analysis: Techniques for

Experimental Design, Measurement, Simulation, and Modeling. Wiley, 1991.

[Karl04] H. Karl. Praxis der Simulation. course slides, Technische Universität Berlin.

[Rob01] T. Robinson. Probability. course slides, University of Bath,

http://www.bath.ac.uk/~masar/UNIV0037/probability.ppt

[Tow04] Don Towsley. Network Simulation. slides for course “Advanced Foundations of

Computer Networks�, University of Massachusetts, USA, 2004.

http://www-net.cs.umass.edu/cs653/

[Tri02] K. S. Trivedi. Probability and Statistics with Reliability, Queueing and Computer

Science Applications. Wiley, 2002.

[Var04] A. Varga. OMNeT++: Object-Oriented Discrete Event Simulator.

http://www.omnetpp.org/

Documents

Telematics 2 & Performance Evaluation · Tele2 / Perf Eval (WS 19/20): 07 –Probability & Results 7 Short Review of Some Probability Basics qProbabilityis a numerical measure of