Upload
theodora-stokes
View
214
Download
1
Embed Size (px)
Citation preview
Probability Theory Overview and Analysis of Randomized
Algorithms
Prepared by
John Reif, Ph.D.
Analysis of Algorithms
Probability Theory Topics
a) Random Variables: Binomial and Geometric
b) Useful Probabilistic Bounds and Inequalities
Readings
• Main Reading Selections:– CLR, Chapter 5 and Appendix C
Probability Measures
• A probability measure (Prob) is a mapping from a set of events to the reals such that:1) For any event A
0 Prob(A) 12) Prob (all possible events) = 13) If A, B are mutually exclusive
events, thenProb(A B) = Prob (A) + Prob (B)
Conditional Probability
• Define
Prob(A B)Prob(A | B)
Prob(B)
for Prob(B) > 0
Bayes’ Theorem
• If A1, …, An are mutually exclusive and contain all events then
ii n
jj=1
j j j
PProb(A | B)
P
where P = Prob(B | A ) Prob(A )
Random Variable A (Over Real Numbers)
• Density Function
Af (x) = Prob(A=x)
Random Variable A (cont’d)
• Prob Distribution Function
x
A A
-
F (x) = Prob(A x) = f (x) dx
Random Variable A (cont’d)
• If for Random Variables A,B
• Then “A upper bounds B” and “B lower bounds A”
A Bx F (x) F (x)
A
B
F (x) Prob (A x)
F (x) Prob (B x)
Expectation of Random Variable A
• Ā is also called “average of A” and “mean of A” = A
A
-
E(A) = A = x f (x) dx
Variance of Random Variable A
2 2 2 2A
2 2A
-
σ (A A) A (A)
where 2nd momoment A x f (x) dx
Variance of Random Variable A (cont’d)
Discrete Random Variable A
Discrete Random Variable A (cont’d)
Discrete Random Variable A Over Nonnegative Numbers
• Expectation
Ax=0
E(A) = A = x f (x)
Pair-Wise Independent Random Variables
• A,B independent if
Prob(A B) = Prob(A) X Prob(B)
• Equivalent definition of independence
A B A B
A B A B
A B A B
f (x) = f (x) f (x)
M (s) = M (s) M (s)
G (z) = G (z) G (z)
Bounding Numbers of Permutations
• n! = n x (n-1) x 2 x 1= number of permutations of n objects
• Stirling’s formulan! = f(n) x (1+o(1)), where
n -nf(n) = n e 2πn
Bounding Numbers of Permutations (cont’d)
• Note– Tighter bound
= number of permutations of n objects taken k at a time
1 1(12n+1) 12nf(n) e < n! < f(n) e
n!
(n-k)!
Bounding Numbers of Combinations
= number of (unordered)
combinations of n objects
taken k at a time
• Bounds (due to Erdos & Spencer, p. 18)
n n!
k k! (n-k)!
2 3
2
k kk 2n 6nn n e
~ (1 o(1))k k!
3
4for k = o n
Bernoulli Variable
• Ai is 1 with prob P and 0 with prob 1-P
• Binomial Variable• B is sum of n independent Bernoulli
variables Ai each with some probability p
BINOMIAL with parameters n,p
B 0
i=1 n
probability P B B+1
procedure
begin
for to do
with do
output
end
B is Binomial Variable with Parameters n,p
n pmean
2 np (1-p)variance
B is Binomial Variable with Parameters n,p (cont’d)
x n-xn Prob(B=x) = p (1-p)
xdensity fn
xk n-k
k=0
n Prob(B x) = p (1-p)
kdistribution fn
Poisson Trial
• Ai is 1 with prob Pi and 0 with prob 1-Pi
• Suppose B’ is the
sum of n independent Poisson trials
Ai with probability Pi for i > 1, …, n
Hoeffding’s Theorem
• B’ is upper bounded by a Binomial Variable B
• Parameters n,p where
n
ii=1
Pp=
n
Geometric Variable V• parameter p
xx 0 Prob(V=x) = p(1-p)
GEOMETRIC parameter p
V 0
loop with probability 1-p
exit
procedure
begin
goto
Probabilistic Inequalities
• For Random Variable A
2 2 2
A
A (A)
mean
variance
Markov and Chebychev Probabilistic Inequalities
• Markov Inequality (uses only mean)
• Chebychev Inequality (uses mean and variance)
Prob (A x)x
2
2Prob ( A )
Example of use of Markov and Chebychev Probabilistic Inequalities
• If B is a Binomial with parameters n,p
2
npThen Prob (B x)
xnp (1-p)
Prob ( B np )
Gaussian Density Function
2x-
21
(x) = e2π
Normal Distribution
• Bounds x > 0 (Feller, p. 175)
x
-
(X) = (Y) dY
3
1 1 Ψ(x)(x) 1 (x)
x x x
x [0,1]
x 1 xx (1) (x) - (0) =
22πe 2π
Sums of Independently Distributed Variables
• Let Sn be the sum of n independently distributed variables A1, …, An
• Each with mean and variance
• So Sn has mean and variance 2
n
2
n
Strong Law of Large Numbers: Limiting to Normal Distribution• The probability density function of
to normal distribution (x)
• Hence
Prob
nn
(S - )T = as nlimits
n( S - x) (x) as n
Strong Law of Large Numbers (cont’d)
• So
Prob
(since 1- (x) (x)/x)
n( S - x) 2(1 (x))
2 (x)/x
Moment Generating Functions and Chernoff Bounds
Advanced Material
Moments of Random Variable A (cont’d)
• n’th Moments of Random Variable A
• Moment generating function
n nA
-
A x f (x) dx
sxA A
-
sA
M (s) e f (x) dx
= E(e )
Moments of Random Variable A (cont’d)
• NoteS is a formal parameter
n
0
nAAn
d M (s)
dss
Moments of Discrete Random Variable A
• n’th moment
n nA
x=0
A x f (x)
Probability Generating Function of Discrete Random Variable A
2 " ' ' 2Avariance σ (1) (1) ( (1))A A AG G G
x AA A
x=0
G (z) = z f (x) E(z )
" 2A2nd derivative G (1) = A A
A1st derivative G ' (1) = A
• If A1, …, An independent with same distribution
1
1 1
1 2 n
n
B A
n n
B A B A
Then if B = A A ... A
f (x) = f (x)
M (s) = M (s) , G (z) = G (z)
1 iA Af (x) = f (x) for i = 1, ... n
Moments of AND of Independent Random Variables
Generating Function of Binomial Variable B with Parameters n,p
• Interesting fact
xn k k n-k
k=0
nG(z) = (1-p+pz) = z p (1-p)
k
1Prob(B= ) =
n
Generating Function of Geometric Variable with parameter p
k k
k=0
pG(Z) = Z (p(1-p) ) =
1-(1-p)Z
Chernoff Bound of Random Variable A
• Uses all moments• Uses moment generating function
By setting x = ’ (s) 1st derivative minimizes bounds
-sxA
γ(s) - sxA
γ(s) - s γ'(s)
Prob (A x) e M (s) for s 0
= e where γ(s) = ln (M (s))
e
Chernoff Bound of Discrete Random Variable A
• Choose z = z0 to minimize above bound
• Need Probability Generating function
-xAProb (A x) z G (z) for z 1
x AA A
x 0
G (z) = z f (x) = E(z )
Chernoff Bounds for Binomial B with parameters n,p
• Above mean x
n-x x
x xx-μ 1
-x - μ 2
Prob (B x)
n-
n-x x
1 e since 1 e
x x
e for x μe
Chernoff Bounds for Binomial B with parameters n,p (cont’d)
• Below mean x
n-x x
Prob (B x)
n-
n-x x
Anguin-Valiant’s Bounds for Binomial B with Parameters n,p
• Just above mean
• Just below mean
2 μ-ε
2
= np for 0 < < 1
Prob (B (1+ ) ) e
2 μ-ε
3
for 0 < < 1
Prob (B (1- ) ) e
Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d)
Tails are bounded by Normal distributions
Binomial Variable B with Parameters p,N and Expectation = pN• By Chernoff
Bound for p < ½
• Raghavan-Spencer bound for any > 0
nN 2
NProb (B ) 2 p
2
μ
(1+ )Prob (B (1+ ) )
(1+ )
e
Probability Theory
Prepared by
John Reif, Ph.D.
Analysis of Algorithms