Basics on ProbabilityJingrui He

09/11/2007

Coin Flips You flip a coin

Head with probability 0.5

You flip 100 coins How many heads would you expect

Coin Flips cont. You flip a coin

Head with probability p Binary random variable Bernoulli trial with success probability p

You flip k coins How many heads would you expect Number of heads X: discrete random variable Binomial distribution with parameters k and p

Discrete Random Variables Random variables (RVs) which may take on

only a countable number of distinct values E.g. the total number of heads X you get if you

flip 100 coins

X is a RV with arity k if it can take on exactly one value out of E.g. the possible values that X can take on are 0,

1, 2,…, 100

1, , kx x

Probability of Discrete RV Probability mass function (pmf): Easy facts about pmf

if if

P X X 0i jx x

i j

1 2P X X X 1kx x x P X X P X P Xi j i jx x x x

i j

P X 1iix

P X ix

Common Distributions Uniform

X takes values 1, 2, …, N E.g. picking balls of different colors from a box

Binomial X takes values 0, 1, …, n

E.g. coin flips

X 1, ,U N

P X 1i N

X ,Bin n p

P X 1n iin

i p pi

Coin Flips of Two Persons Your friend and you both flip coins

Head with probability 0.5 You flip 50 times; your friend flip 100 times How many heads will both of you get

Joint Distribution Given two discrete RVs X and Y, their joint

distribution is the distribution of X and Y together E.g. P(You get 21 heads AND you friend get 70

heads)

E.g.

P X Y 1x y

x y

50 100

0 0P You get heads AND your friend get heads 1

i ji j

Conditional Probability is the probability of ,

given the occurrence of E.g. you get 0 heads, given that your friend gets

61 heads

P X Yx y

P X YP X Y

P Y

x yx y

y

X xY y

Law of Total Probability Given two discrete RVs X and Y, which take

values in and , We have 1, , mx x 1, , ny y

P X P X Y

P X Y P Y

i i jj

i j jj

x x y

x y y

Marginalization

Marginal Probability Joint Probability

Conditional Probability

P X P X Y

P X Y P Y

i i jj

i j jj

x x y

x y y

Marginal Probability

Bayes Rule X and Y are discrete RVs…

P Y X P XP X Y

P Y X P X

j i ii j

j k kk

y x xx y

y x x

P X YP X Y

P Y

x yx y

y

Independent RVs Intuition: X and Y are independent means that

neither makes it more or less probable that

Definition: X and Y are independent iff P X Y P X P Yx y x y

X xY y

More on Independence

E.g. no matter how many heads you get, your friend will not be affected, and vice versa

P X Y P X P Yx y x y

P X Y P Xx y x P Y X P Yy x y

Conditionally Independent RVs Intuition: X and Y are conditionally

independent given Z means that once Z is known, the value of X does not add any additional information about Y

Definition: X and Y are conditionally independent given Z iff

P X Y Z P X Z P Y Zx y z x z y z

More on Conditional Independence

P X Y Z P X Z P Y Zx y z x z y z

P X Y , Z P X Zx y z x z

P Y X , Z P Y Zy x z y z

Monty Hall Problem You're given the choice of three doors: Behind one

door is a car; behind the others, goats. You pick a door, say No. 1 The host, who knows what's behind the doors, opens

another door, say No. 3, which has a goat. Do you want to pick door No. 2 instead?

                        

                        

Host mustreveal Goat B

                        

                        

Host mustreveal Goat A

                       

                           

  

Host revealsGoat A

orHost reveals

Goat B

           

             

Monty Hall Problem: Bayes Rule : the car is behind door i, i = 1, 2, 3 : the host opens door j after you pick door i

iC

ijH

1 3iP C

0

0

1 2

1 ,

ij k

i j

j kP H C

i k

i k j k

Monty Hall Problem: Bayes Rule cont. WLOG, i=1, j=3

13 1 11 13

13

P H C P CP C H

P H

13 1 11 1 1

2 3 6P H C P C

Monty Hall Problem: Bayes Rule cont.

13 13 1 13 2 13 3

13 1 1 13 2 2

, , ,

1 11

6 31

2

P H P H C P H C P H C

P H C P C P H C P C

1 131 6 1

1 2 3P C H

Monty Hall Problem: Bayes Rule cont.

1 131 6 1

1 2 3P C H

You should switch!

2 13 1 131 2

13 3

P C H P C H

Continuous Random Variables What if X is continuous? Probability density function (pdf) instead of

probability mass function (pmf) A pdf is any function that describes the

probability density in terms of the input variable x.

f x

PDF Properties of pdf

Actual probability can be obtained by taking the integral of pdf E.g. the probability of X being between 0 and 1 is

0,f x x

1f x

1

0P 0 1X f x dx

1 ???f x

Cumulative Distribution Function Discrete RVs

Continuous RVs

X P XF v v

X P Xi

ivF v v

X

vF v f x dx

X

dF x f x

dx

Common Distributions Normal

E.g. the height of the entire population

2X ,N

22

1exp ,

2 2

xf x x

-5 -4 -3 -2 -1 0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

x

f(x)

Common Distributions cont. Beta

: uniform distribution between 0 and 1 E.g. the conjugate prior for the parameter p in

Binomial distribution

X ,Beta

111; , 1 , 0,1

,f x x x x

B

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

x

f(x)

Joint Distribution Given two continuous RVs X and Y, the joint

pdf can be written as

X,Y ,f x y

X,Y , 1x y

f x y dxdy

Multivariate Normal Generalization to higher dimensions of the

one-dimensional normal

1X 1 22

1

1, ,

2

1exp

2

d d

T

f x x

x x

Covariance Matrix

Mean

Moments Mean (Expectation):

Discrete RVs:

Continuous RVs: Variance:

Discrete RVs:

Continuous RVs:

XE X P X

ii iv

E v v XE xf x dx

2X XV E 2X P X

ii iv

V v v 2XV x f x dx

Properties of Moments Mean

If X and Y are independent,

Variance If X and Y are independent,

X Y X YE E E

X XE a aE XY X YE E E

2X XV a b a V

X Y (X) (Y)V V V

Moments of Common Distributions Uniform

Mean ; variance Binomial

Mean ; variance Normal

Mean ; variance Beta

Mean ; variance

X 1, ,U N 1 2N 2 1 12N

X ,Bin n pnp 2np

2X ,N 2

X ,Beta 2

1

Probability of Events X denotes an event that could possibly happen

E.g. X=“you will fail in this course” P(X) denotes the likelihood that X happens,

or X=true What’s the probability that you will fail in this

course? denotes the entire event set

X,X

The Axioms of Probabilities 0 <= P(X) <= 1 , where are

disjoint events Useful rules

P 1

1 2P X X P Xii Xi

1 2 1 2 1 2P X X P X P X P X X

P X 1 P X

Interpreting the Axioms

1X2X