Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Slide 1
18-660: Numerical Methods for Engineering Design and Optimization
Xin Li Department of ECE
Carnegie Mellon University Pittsburgh, PA 15213
Slide 2
Overview
Monte Carlo Analysis Random variable Probability distribution Random sampling
Slide 3
Random Variables
A random variable is a real-valued function of the outcome of the experiment
x (random variable) =
+1.0 (Experiment 1) −0.3 (Experiment 2) ... −2.4 (Experiment N)
We get different results from different experiments (i.e., the output is random)
Slide 4
Probability Distribution
A continuous random variable x is defined by its probability distribution function
Probability density function (PDF) pdfx(tx) denotes the probability per unit
length near x = tx
Cumulative distribution function (CDF) cdfx(tx) equals the probability of x ≤ tx
( ) ( ) ( )x
t
xxxxx txPdpdftcdfx
≤=⋅= ∫∞−
ττ
( ) ( )bxaPdpdfb
axxx ≤≤=⋅∫ ττ x 0
x 0
CDF
Slide 5
Expectation
Given a random variable x and a function f(x), the expectation of f(x) is the weighted average of the possible values of f(x)
A useful equation for expected value calculation
( )[ ] ( ) ( )∫+∞
∞−
⋅⋅= xxxx dpdffxfE τττ
( ) ( )[ ] ( )[ ] ( )[ ]xgxfExgxfE +=+
Slide 6
Mean, Variance and Standard Deviation
Mean
Variance
Standard deviation
[ ] ( )∫+∞
∞−
⋅⋅= xxxx dpdfxE τττ
[ ] [ ]( )[ ] ( )[ ] ( )∫+∞
∞−
⋅⋅−=−= xxxx dpdfxExExExVAR τττ 22
[ ] [ ]xVARxSTD =
VAR[x] is always positive!
Slide 7
Mean, Variance and Standard Deviation
Mean measures the “average position” of x
Variance measures the “spread” of the distribution
x 0
PDF1
x 0
PDF2
Small mean Large mean
x 0
PDF1
x 0
PDF2
Small variance Large variance
Δ
Slide 8
Moments and Central Moments
k-th order moment
Mean is the first order moment
k-th order central moments
Variance is the second order central moment
[ ]( )[ ] ( )[ ] ( )∫+∞
∞−
⋅⋅−=− xxxk
xk dpdfxExExE τττ
[ ] ( )∫+∞
∞−
⋅⋅= xxxkx
k dpdfxE τττ
Slide 9
Normal Distribution
A random variable x is Normal if
μ: mean σ: standard deviation Denoted as N(μ, σ2)
If μ = 0 and σ = 1, it is called standard Normal distribution
( )( )
2
2
2
2
1 σ
µ
πσ
−−
⋅=t
x etpdf
-5 0 50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
x
PD
FStandard Normal distribution
Why is Normal distribution important to us?
Slide 10
Normal Distribution
Many physical variations are Normal
Central limit theorem: the variation caused by a large number of independent random factors is “almost” Normal
1xy = 21 xxy +=
Assume that all xi’s are independent and have the same uniform distribution
54321 xxxxxy ++++=
-0.5 0 0.50
200
400
600
y
# of
Sam
ples
-1 -0.5 0 0.5 10
200
400
600
800
1000
y
# of
Sam
ples
-4 -2 0 2 40
500
1000
1500
y
# of
Sam
ples
Slide 11
Multiple Random Variables
Two continuous random variables x and y are defined by their joint probability distribution
Joint probability density function
Joint cumulative distribution function
( ) ( ) ( )∫ ∫∞− ∞−
⋅⋅=≤≤=x yt t
yxyxyxyxyxyx ddpdftytxPttcdf ττττ ,,, ,,
( ) ( )dycbxaPddpdfd
c
b
ayxyxyx ≤≤≤≤=⋅⋅∫ ∫ ,,, ττττ
Applicable to more than two random variables
Slide 12
Joint Probability Distribution
Example: bivariate Normal distribution
Joint probability density function Joint cumulative distribution function
Slide 13
Marginal Distribution Function
Marginal probability density function
Marginal cumulative distribution function
( ) ( )
( ) ( )∫
∫∞+
∞−
+∞
∞−
⋅=
⋅=
xyxyxyy
yyxyxxx
dtpdftpdf
dtpdftpdf
ττ
ττ
,
,
,
,
( ) ( ) ( )( ) ( ) ( )yxyxtyyy
yxyxtxxx
ttcdftyxPtcdf
ttcdfytxPtcdf
x
y
,lim,
,lim,
,
,
+∞→
+∞→
=≤+∞≤=
=+∞≤≤=
Slide 14
Marginal Distribution Function
Example: bivariate Normal distribution
Marginal PDF for x Marginal
PDF for y
Slide 15
Covariance and Correlation
Covariance
If COV[x,y] = 0, then x and y are uncorrelated
Covariance matrix
Σ is always symmetric Diagonal components are corresponding to variance values Σ is diagonal if x and y are uncorrelated
[ ] [ ]( ) [ ]( )[ ]yEyxExEyxCOV −⋅−=,
[ ] [ ][ ] [ ]
=Σ
yyCOVxyCOVyxCOVxxCOV
,,,,
Slide 16
Covariance and Correlation
Correlation (normalized covariance)
Correlation between two random variables can be visualized by scatter plot
[ ] [ ][ ] [ ]ySTDxSTD
yxCOVyxCOR⋅
=,,
-4 -2 0 2 4-4
-2
0
2
4
x
y Zero correlation
Slide 17
Covariance and Correlation
Example: correlated random variables
-4 -2 0 2 4-4
-2
0
2
4
x
y
-4 -2 0 2 4-4
-2
0
2
4
xy
Positive correlation
Negative correlation
Slide 18
Monte Carlo Analysis
Problem definition Find probability distribution and/or moments of
In general, the distribution and/or moments of f cannot be calculated analytically, because f(X) is nonlinear f(X) may not have closed-form expression (we can only
numerically calculate f for a given X value)
( )Xf
Function of interest
Random variable with known distribution
Slide 19
Monte Carlo Analysis
Monte Carlo analysis for f(X) Randomly select M samples for X Evaluate function f(X) at each sampling point Estimate distribution of f using these M samples
Distribution of f(X)
Evaluate f(X)
Random samples {X(1), X(2), ...}
Samples of f(X)
Slide 20
Monte Carlo Analysis Example
Example: estimate the probability distribution of
x ~ N(0,1) (standard Normal distribution)
( )xy exp=
Slide 21
Monte Carlo Analysis Example
Step 1: draw random samples for x
Step 2: calculate y at each sampling point
Samples 1 2 3 4 5 6 ... x -0.4326 -1.6656 0.1253 0.2877 -1.1465 1.1909 ...
M random samples for x
M random samples for y
Samples 1 2 3 4 5 6 ... y 0.6488 0.1891 1.1335 1.3333 0.3178 3.2901 ...
Slide 22
Monte Carlo Analysis Result
Monte Carlo result is typically represented by a histogram A big table of data is not intuitive
Histogram of y based on 1000 random samples
QUESTION: how accurate is Monte Carlo analysis?
0 5 10 15 200
100
200
300
400
500
y
Num
ber o
f Sam
ples
Slide 23
Monte Carlo Analysis Accuracy
Monte Carlo analysis is not deterministic We cannot get identical results when running MC twice The analysis error is not deterministic
Monte Carlo accuracy depends on the number of samples Examples: histogram of y
100 samples 1000 samples 10000 samples
0 2 4 6 8 100
5
10
15
20
25
y
Num
ber o
f Sam
ples
0 5 10 15 200
100
200
300
400
500
y
Num
ber o
f Sam
ples
0 5 10 15 200
1000
2000
3000
4000
5000
6000
y
Num
ber o
f Sam
ples
Slide 24
Monte Carlo Analysis Accuracy
Example: bivariate Normal distribution x and y are independent and jointly standard Normal
Slide 25
Monte Carlo Analysis Accuracy
Statistical methods exist to analyze Monte Carlo accuracy
Example: Monte Carlo accuracy analysis
Estimate the mean value μx by Monte Carlo analysis Our question: how accurate is the estimated μx (dependent on
the number of Monte Carlo samples)?
( )1,0~ Nx Standard Normal distribution
Slide 26
Monte Carlo Analysis Accuracy
Monte Carlo analysis for the mean value μx Randomly draw M sampling points {x(1),x(2),...,x(M)} Estimate μx by the following equation
Assumptions in our accuracy analysis Each x(i) is random and satisfies standard Normal distribution –
it is randomly created for x ~ N(0,1) All x(i)’s are mutually independent – samples from a good
random number generator should be independent
μx is a function of {x(1),x(2),...,x(M)}, which is a random variable
( ) ( ) ( )
Mxxx M
x+++
=
21
µ Called an estimator
Slide 27
Monte Carlo Analysis Accuracy
Mean of μx
Variance of μx
{ }( ) ( ) ( ) ( ){ } ( ){ } ( ){ } 0
2121
=+++
=
+++
=M
xExExEM
xxxEEMM
xµ
{ }( ) ( ) ( ) ( ){ } ( ){ } ( ){ }
MMxExExE
MxxxEE
MM
x1
2
222212212 =
+++=
+++=
µ
E{x(i)} = 0
x(i)’s are independent E{x(i)2} = 1
Slide 28
Monte Carlo Analysis Accuracy
E{μx} = 0 ux is an unbiased estimator Otherwise, if the estimator mean is not equal to the actual
mean, it is called a biased estimator
E{μx2} = 1/M
Variance decreases as M increases Distributions of μx for different M values
-5 0 50
0.1
0.2
0.3
0.4
µx
-5 0 50
1
2
3
4
µx
-5 0 50
0.5
1
1.5
µx
1 sample 10 samples 100 samples μx is a Normal distribution N(0, 1/M)
Slide 29
Monte Carlo Analysis Accuracy
“Average” estimation accuracy is better when using larger M
In this μx example If we require that ±3 sigma of μx is within [-0.1, 0.1]
If we require that ±3 sigma of μx is within [-0.01, 0.01]
1.03≤
M900≥M
01.03≤
M90000≥M
Slide 30
Monte Carlo Analysis Accuracy
Accuracy is improved by 10x if the number of samples is increased by 100x
1K ~ 10K sampling points are typically required to achieve reasonable accuracy
However, even if you use 10K sampling points, an accurate result is not guaranteed! Monte Carlo analysis is random, and you can be unlucky (e.g.,
going beyond ±3 sigma range)
Slide 31
Summary
Monte Carlo analysis Random variable Probability distribution Random sampling