AlidN i lApplied Numerical Analysis - Sharifsharif.edu/~fatemizadeh/Courses/ANA/10-PRNG-View.pdfPseudo Random Number GeneratorPseudo Random Number Generator What is random number:

A li d N i lA li d N i lApplied Numerical Applied Numerical AnalysisAnalysisAnalysisAnalysis

Pseudo Random Number GeneratorPseudo Random Number GeneratorLecturer: Emad FatemizadehLecturer: Emad FatemizadehLecturer: Emad FatemizadehLecturer: Emad Fatemizadeh

Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh

Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator

What is random number:What is random number:What is random number:What is random number:•• A sequence in which each term is unpredictableA sequence in which each term is unpredictable

29, 95, 11, 60, 2229, 95, 11, 60, 22

Application:Application:•• Monte Carlo SimulationsMonte Carlo Simulations•• Generation of Cryptographic KeysGeneration of Cryptographic Keys

Password generationPassword generation

•• Many Combinatorial Optimization AlgorithmsMany Combinatorial Optimization Algorithms•• Many Combinatorial Optimization AlgorithmsMany Combinatorial Optimization Algorithms•• GamesGames



RN’s Types:RN’s Types:RN s Types:RN s Types:•• True random:True random:

Generated in nonGenerated in non--deterministic waysdeterministic waysGenerated in nonGenerated in non--deterministic ways.deterministic ways.•• Not predictable Not predictable •• Not repeatable.Not repeatable.

•• Pseudo random:Pseudo random:Numbers that Numbers that appearappear random, but are random, but are bt i d i d t i i ti bt i d i d t i i ti obtained in a deterministic, obtained in a deterministic, •• RepeatableRepeatable•• PredictablePredictable



RN’s Generation:RN’s Generation:RN s Generation:RN s Generation:•• True Random:True Random:

Physical Phenomenon:Physical Phenomenon:Physical Phenomenon:Physical Phenomenon:•• Decay times of radioactive materialDecay times of radioactive material•• Electrical noise from a resistor or semiconductorElectrical noise from a resistor or semiconductor•• Radio channel or audible noiseRadio channel or audible noise•• Keyboard timingsKeyboard timings

•• Pseudo Random:Pseudo Random:•• Pseudo Random:Pseudo Random:Mathematical AlgorithmMathematical Algorithm



Preliminary:Preliminary:Preliminary:Preliminary:•• PDF (Probability Density Function):PDF (Probability Density Function):

( ) ( )Prob X f≤ ≤ +

( )Xf x

( ) ( )

( ) ( )

Prob

0, 1

X

X X

x X x x f x x

f x f x dx+∞

≤ ≤ + =

≥ =∫•• CDF (Cumulative Density Function):CDF (Cumulative Density Function):

( ) ( )−∞∫

( )XF x

( ) ( ) ( )

( ) ( )

Probx

X XX x F x f s ds−∞

−∞ ≤ ≤ = = ∫


( ) ( )1, 0X XF F+∞ = −∞ =


Preliminary:Preliminary:Preliminary:Preliminary:•• Samples:Samples:

Uniform:Uniform:Uniform:Uniform:

( )1

X

a x bf x b a

⎧ ≤ ≤⎪= −⎨

NormalNormal

( )0 O.W.

Xf b a⎨⎪⎩

( ) ( )2

22

1 exp22

X

xf x

μσπσ

⎛ ⎞−= −⎜ ⎟

⎜ ⎟⎝ ⎠


⎝ ⎠


Pseudo Random Number Generator:Pseudo Random Number Generator:Pseudo Random Number Generator:Pseudo Random Number Generator:•• Uniform distributionUniform distribution

Theoretically all others distribution can be be Theoretically all others distribution can be be f ff fgenerate from Uniform.generate from Uniform.

•• Ideal: Between [0 1]Ideal: Between [0 1]In computer we generate random integer [0,2In computer we generate random integer [0,2NN--1] or 1] or In computer we generate random integer [0,2In computer we generate random integer [0,2 1] or 1] or [0,2[0,2NN] then convert to [0,1]] then convert to [0,1]

•• Desired Properties:Desired Properties:L P i dL P i dLong PeriodLong PeriodUncorrelatedUncorrelatedFastFast



Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:

th d l 0 th d l 0( )1 mod 0n nX aX c m n+ = + ≥

m: the modulus m>0m: the modulus m>0a: the multiplier 0≤a<ma: the multiplier 0≤a<mc: the increment 0≤c<mc: the increment 0≤c<mXX00 the seed point 0≤Xthe seed point 0≤X00<m<mXX0:0: the seed point 0≤Xthe seed point 0≤X00<m<mmod: Integer remindermod: Integer reminder



Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:

Three Classes:Three Classes:( )1 mod 0n nX aX c m n+ = + ≥

Three Classes:Three Classes:•• m=2m=2NN, c>0, c>0•• m=2m=2NN c=0 c=0•• m=2m=2 , c=0, c=0•• m=prime, c=0m=prime, c=0Two first are fast and easy Third has Two first are fast and easy Third has Two first are fast and easy. Third has Two first are fast and easy. Third has high randomness.high randomness.Choice of m a and c are critical!Choice of m a and c are critical!


Choice of m, a, and c are critical!Choice of m, a, and c are critical!


Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:•• Let a=1,c=5,m=16 and xLet a=1,c=5,m=16 and x00=1. =1. •• The sequence of pseudoThe sequence of pseudo random random •• The sequence of pseudoThe sequence of pseudo--random random

integers generated by this algorithm is: integers generated by this algorithm is: 1,6,15,12,13,2,11,8,9,14,7,4,5,10,3,01,6,15,12,13,2,11,8,9,14,7,4,5,10,3,0,,111,6,15,12,13,2,11,8,9,14,7,4,5,10,3,01,6,15,12,13,2,11,8,9,14,7,4,5,10,3,0,,11,6,15,12,13,2,11,8,9,14,6,15,12,13,2,11,8,9,14,,……..



Common Examples:Common Examples:Common Examples:Common Examples:

Name m a c period

ANSI CANSI C 223131 11035152451103515245 1234512345 223131

ParkPark--MillerMiller 223131--11 1680716807 00 223131--22

drand48drand48 224848 2521490391725214903917 1111 224848drand48drand48 224848 2521490391725214903917 1111 224848

Hayes 64Hayes 64--bitbit 226464 63641362238467930056364136223846793005 11 226464



Improvement on LCG Method:Improvement on LCG Method:Improvement on LCG Method:Improvement on LCG Method:•• Multiple Recursive GeneratorMultiple Recursive Generator

1 -1 2 -2 -( ) mod i i i k i kX a X a X a X c m= + + + +



Matlab Coding:Matlab Coding:Matlab Coding:Matlab Coding:

x(1) = 1;x(1) 1;

a = 16807;

m = 2^31 – 1;

c = 0;

for n=1:10000,

x(n+1)= mod(a*x(n)+c m); x(n+1)= mod(a*x(n)+c,m);

end;

x = x/(m+1);


hist(x)


Matlab Coding:Matlab Coding:Matlab Coding:Matlab Coding:



Lagged Fibonacci Generators:Lagged Fibonacci Generators:Lagged Fibonacci Generators:Lagged Fibonacci Generators:•• Remember Fibonacci Sequence:Remember Fibonacci Sequence:

XX =X=X +X+X ; X; X = 0; X= 0; X =1=1XXnn=X=Xnn--11+X+Xnn--22; X; X00 = 0; X= 0; X11=1=1

•• General Formulation:General Formulation:( ) modX X X m= ( ) mod

: An Operator: +, -, *r>s>0

n n r n sX X X m− −=

••

N

r>s>0m=2

random seed required!r

•

••


random seed required!Popular Value of (r,s): (17,5), (55,24), (127,97), (607,273), (1279,418)r

•


How to generate other distributions:How to generate other distributions:How to generate other distributions:How to generate other distributions:•• A mathematical theory:A mathematical theory:

If we pass a Uni[0 1] random variables from If we pass a Uni[0 1] random variables from If we pass a Uni[0,1] random variables from If we pass a Uni[0,1] random variables from inverse of desired cumulative density inverse of desired cumulative density function, then output will have distribution function, then output will have distribution f b bilit d it f tif b bilit d it f tiof probability density function:of probability density function:

[ ]0,1u U∼

( ) ( )1X Xx F u f x−= ∼



How to generate other distributions:How to generate other distributions:How to generate other distributions:How to generate other distributions:•• Example:Example:

We want 10000 random variable with We want 10000 random variable with We want 10000 random variable with We want 10000 random variable with distribution of distribution of

( ) 0xe xf

−⎧ ≥⎨

CDF is:CDF is:

( )0 O.W.Xf x = ⎨⎩

1 0x−⎧ ≥( ) ( )

( ) [ ]

11 0ln 1

0 O.W.

l 1 0 1

x

X Xe x

F x F x

U

−−⎧ − ≥

= ⇒ = − −⎨⎩


( ) [ ]ln 1 0,1x x U= − − ∼




Method is good but Method is good but NOTNOT practical:practical:Method is good but Method is good but NOTNOT practical:practical:•• For example consider normal For example consider normal

distribution:distribution:distribution:distribution:

( ) ( )2 2

2 21 1 ??xx s

f x e F x e ds− −

= ⇒ = =∫( ) ( )2 2

2 2??X Xf x e F x e ds

π π −∞

= ⇒ = =∫



BoxBox--Muller Method for Normal:Muller Method for Normal:BoxBox Muller Method for Normal:Muller Method for Normal:•• Suppose uSuppose u11 and uand u2 2 are two independent are two independent

uniform random variable then:uniform random variable then:uniform random variable, then:uniform random variable, then:

π= − 1 22log( ) cos(2 )x u u

are jointly normal random variable;are jointly normal random variable;

π= − 1 22log( ) sin(2 )y u u

are jointly normal random variable;are jointly normal random variable;





Central Limit Theorem for Normal:Central Limit Theorem for Normal:Central Limit Theorem for Normal:Central Limit Theorem for Normal:•• Average of N independent random Average of N independent random

variable (with zero mean) from any variable (with zero mean) from any variable (with zero mean) from any variable (with zero mean) from any distribution converge in limits to distribution converge in limits to normal!normal!

=→∞=∑

∼1lim

N

ii

N

xy Normal

N→∞mNy mN



1 21 2

4 12



Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• rand(m,n): A mrand(m,n): A m××n matrix with Uni[0,1]n matrix with Uni[0,1]•• s = rand('state') : s = rand('state') : Return Current state.Return Current state.s rand( state ) : s rand( state ) : Return Current state.Return Current state.•• rand('state',s): rand('state',s): Resets the state to s.Resets the state to s.•• rand('state',0): rand('state',0): Resets the generator to Resets the generator to ( , )( , ) gg

its initial state. its initial state. •• rand('state',j): rand('state',j): For integer j, resets the For integer j, resets the

t t it jt t it j th t tth t tgenerator to its jgenerator to its j--th state.th state.•• rand('state',sum(100*clock)): rand('state',sum(100*clock)): Resets it Resets it

to a different state each timeto a different state each time


to a different state each time.to a different state each time.


Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• rand(m,n): A mrand(m,n): A m××n matrix with Uni[0,1]n matrix with Uni[0,1]•• How to u[a b]?How to u[a b]?•• How to u[a,b]?How to u[a,b]?

a = 10; b=20;a = 10; b=20;u=rand(1 1000);u=rand(1 1000);u=rand(1,1000);u=rand(1,1000);x = a + (bx = a + (b--a) * u;a) * u;





Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• randrandnn(m,n): A m(m,n): A m××n matrix with N[0,1]n matrix with N[0,1]•• s = rand('state') : s = rand('state') : Return Current state.Return Current state.s rand( state ) : s rand( state ) : Return Current state.Return Current state.•• rand('state',s): rand('state',s): Resets the state to s.Resets the state to s.•• rand('state',0): rand('state',0): Resets the generator to Resets the generator to ( , )( , ) gg

its initial state. its initial state. •• rand('state',j): rand('state',j): For integer j, resets the For integer j, resets the

t t it jt t it j th t tth t tgenerator to its jgenerator to its j--th state.th state.•• rand('state',sum(100*clock)): rand('state',sum(100*clock)): Resets it Resets it

to a different state each timeto a different state each time


to a different state each time.to a different state each time.


Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• rand(m,n): A mrand(m,n): A m××n matrix with Uni[0,1]n matrix with Uni[0,1]•• How to N[mHow to N[m δδ22]?]?•• How to N[m,How to N[m,δδ22]?]?

m=10; s=16;m=10; s=16;u=randn(1 1000);u=randn(1 1000);u=randn(1,1000);u=randn(1,1000);x = m + sqrt(s) * u;x = m + sqrt(s) * u;





Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• p = randperm(n); Random permutation p = randperm(n); Random permutation

of 1:n:of 1:n:of 1:n:of 1:n:p=randperm(5);p=randperm(5);

•• P=[2 5 1 3 4];P=[2 5 1 3 4];



Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• Other distribution:Other distribution:

exprndexprnd Exponential random numbersExponential random numbersexprndexprnd Exponential random numbersExponential random numberstrndtrnd Student's t random numbersStudent's t random numbersraylrndraylrnd Rayleigh random numbersRayleigh random numbersyy y gy gSee See StatisticsStatistics Toolbox for others.Toolbox for others.


Monte Carlo MethodMonte Carlo MethodMonte Carlo MethodMonte Carlo Method

Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration••ππ estimation as a simple example:estimation as a simple example:

l fl fGenerate N couples of RVs Generate N couples of RVs

4insideˆ N

2

inside

TotalNπ =

-4 -2 2 4

-2


-4


MATLAb experiment:MATLAb experiment:MATLAb experiment:MATLAb experiment:N = 1000;AB = 2*(rand(2,N)-0.5);

( )a = AB(1,:);b = AB(2,:);c = sqrt(a.^2+b.^2);PIh = 4*length(find(c<=1))/length(c);PIh 4 length(find(c< 1))/length(c);err = 100*abs(PIh-pi)/pi;

NN 101022 101033 101044 101044NN 1010 1010 1010 1010PihPih 3.32003.3200 3.10803.1080 3.12003.1200 3.13893.1389

%% 5 67895 6789 1 06931 0693 0 68730 6873 0 08510 0851


err%err% 5.67895.6789 1.06931.0693 0.68730.6873 0.08510.0851


Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration••Another ExampleAnother Example::

l fl fGenerate N couples of RVs Generate N couples of RVs

insideˆ N 4inside

TotalNπ =

2

-4 -2 2 4

-2


-4


Simple Approach:Simple Approach:Simple Approach:Simple Approach:•• Random Point Selection and count!Random Point Selection and count!

f N∫( ) :x y

h

( ) White

W t de ehi Ra

Ng x dxN N

≈+∫( )

( )

, :1)2)

x ya x bf x y h≤ ≤

≤ ≤( ), :1)x ya x b≤ ≤

( )2) f x y h≤ ≤

( )1)2)0

a x by f x

≤ ≤

≤ ≤


a b

Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration

Simple Formulation:Simple Formulation:Simple Formulation:Simple Formulation:•• Expected Value:Expected Value:{ } ( ) { } 1b b

UniE x xf x dx E x xdx= → =∫ ∫{ } ( ) { }

( ){ } ( )1

xa a

b

E x xf x dx E x xdxb a

E g x g x dx

= ⎯⎯→ =−

=

∫ ∫

∫•• Integral:Integral:

( ){ } ( )a

E g x g x dxb a

=− ∫

1 b

( ){ } ( )1

1 1a

b bN N

E g x g x dxb a

b a

=− ∫


( ) ( ) ( ) ( )1 1

1 1N N

i ii ia a

b ag x g x dx g x dx g xN b a N= =

−≈ ⇒ ≈

−∑ ∑∫ ∫

Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration

Example and MATLAB Code:Example and MATLAB Code:Example and MATLAB Code:Example and MATLAB Code:

[ ]1

sin sin , Uni 0,N

i ii

I dN

π πθ θ θ θ π= ≈ ∑∫ ∼10

Exact value of I: 2iN =

N = 100;a = 0;b = pi;x = rand(1,N)*(b-a)+a;Ih = sum(sin(x))*(b-a)/N;

NN 101022 101033 101044 101044NN 101022 101033 101044 101044

II 1.99421.9942 2.02042.0204 2.00742.0074 2.00112.0011

err%err% 0.29130.2913 --1.01861.0186 --0.36830.3683 --0.05650.0565


err%err% 0.29130.2913 1.01861.0186 0.36830.3683 0.05650.0565

Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization

Problem Statement:Problem Statement:Problem Statement:Problem Statement:

( )minx

f x

Methods:Methods:

x

•• Pure MathematicalPure Mathematical•• Direct SearchDirect Search

( ) ( )0,f x f′ = ∇ =x 0

•• Steepest DescentSteepest Descent



Example:Example:Example:Example:•• Best Elliptic that match a set of points!Best Elliptic that match a set of points!



Example:Example:Example:Example:•• Try to solve this problem:Try to solve this problem:

( )h fN( )i i 1

22 22 2

we have a set of x ,yN

i

Ni ix ye r

=

⎡ ⎤⎛ ⎞ ⎛ ⎞+⎢ ⎥⎜ ⎟ ⎜ ⎟∑1

2 2 2

0 ???

i i

ie r

a b

e e e

=

= + −⎢ ⎥⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠⎢ ⎥⎣ ⎦

∂ ∂ ∂→

∑

0 ???a b r= = = →

∂ ∂ ∂



Multivariable function F(Multivariable function F(xx):):Multivariable function F(Multivariable function F(xx):):•• Gradient: A vector:Gradient: A vector:

TF F F⎡ ⎤∂ ∂ ∂

Hessian: A Matrix:Hessian: A Matrix:

( )1 2

, , ,n

F F FGx x x

⎡ ⎤∂ ∂ ∂= ⎢ ⎥∂ ∂ ∂⎣ ⎦

x …

•• Hessian: A Matrix:Hessian: A Matrix:

( ) ( )ij n nH H

×⎡ ⎤= ⎣ ⎦x x

( ) ( )2n n

iji j

FH

x x

×⎣ ⎦∂

=∂ ∂

xx


i j


Multivariable function F(Multivariable function F(xx):):Multivariable function F(Multivariable function F(xx):):•• Taylor Series:Taylor Series:

1

Example: Example:

( ) ( ) ( ) ( )12

T TF F G H+ = + + +x h x x h h x h

•• Example: Example:

( ) ( ) ( ) 1 21 2 1 2, cos sin x xF x x x x e= + +

( ) ( ) ( )

( ) ( ) ( )

1 2 1 2

1 2 1 2

1 2 1 2 2 1

21 2 1 2

, sin ,cos

cos 1

x x x x

x x x x

G x x x x e x x e

x x e x x eG

⎡ ⎤= − + +⎣ ⎦⎡ ⎤− + +⎢ ⎥


( ) ( ) ( )( ) ( )1 2 1 2

1 2 1 21 2 2

1 2 2 1

,1 sinx x x xG x x

x x e x x e= ⎢ ⎥+ − +⎣ ⎦


Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:•• Change function in direction of Change function in direction of ––G(G(xx))

Example in 1DExample in 1DExample in 1DExample in 1D

0G∇ >0G∇ < 0G∇ >0G∇ <



Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:•• Very Simple solution:Very Simple solution:

Some methods solve this 1D Some methods solve this 1D

( )( )1 0,k k kGε+ = + −∇x x x x

•• Some methods solve this 1D Some methods solve this 1D minimization:minimization:

( )( )( )

mint

F tG

tG

+

← +

x x

x x x


( )


More Advanced Method:More Advanced Method:More Advanced Method:More Advanced Method:

( ) ( ) ( ) ( )12

T TF F G H+ ≈ + +x h x x h h x h

( ) ( ) 12

n n n

i i i ij ji i i

F F G h h H h= = =

+ = + +∑ ∑∑x h x

( )1

0, 1, 2, ,n

k kj jjk

FG H h k n

h =

∂ += + = =

∂ ∑x h

1

1

, 1, 2, ,n

kj j kj

H h G k n=

= − =∑


1kH G−∴ = −h


Matlab CommandMatlab CommandMatlab CommandMatlab Command•• Fmin(f,x0): 1DFmin(f,x0): 1D•• Fminsearch(f x0): nDFminsearch(f x0): nD•• Fminsearch(f,x0): nDFminsearch(f,x0): nD•• Optimization toolboxOptimization toolbox



Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• p=rand(2,1);p=fminsearch(@ellip,p);p=rand(2,1);p=fminsearch(@ellip,p);

f ti lli ( )function e = ellip(x);a = x(1);b = x(2);t = 0:359;t 0 359;xi = 5*sin(t*pi/180);yi = 40*cos(t*pi/180);e = 0;for i=1:360for i=1:360,

e = e+((xi(i)/a).^2+(yi(i)/b).^2-1).^2;end;


Documents

AlidN i lApplied Numerical Analysis - Sharifsharif.edu/~fatemizadeh/Courses/ANA/10-PRNG-View.pdfPseudo Random Number GeneratorPseudo Random Number Generator What is random number: