Review of Probability

Definitions (1)

Event space: pretty much any set of things, we’ll usually call it E if we have to refer to it.

Random variable: a variable whose possible values are taken from an event space. Usually denoted with a capital letter.

Probability distribution: a function P with event space E is a probability distribution if it has these properties:

1. 2.3.

Quiz

1. Let’s say I have a random variable X for a coin, with event space {H, T}.

If the probability P(X=H) is 0.5, what is P(X=T)?

2. If P(X=H)=0.25, what is P(X=T)?

Definitions (2)

Joint distribution: A probability distribution of two or more random variables. The event space for this distribution is the cross product of the event space of the individual random variables.

E.g. Let X1 be a random variable for a coin flip. Let X2 be a random variable for a second coin flip.

P(X1, X2) is a joint distribution over all possible values for both coin flips.

Quiz

How many events are in the event space for flipping two coins?

Name two of these events.

Definitions (3)

Marginal distribution: This is just any probability distribution, but people use it to refer to a distribution over one variable when they’ve separately introduced a joint distribution over that variable and a second variable.

E.g., if I have a joint distribution P(X1, X2), then P(X1) is a marginal distribution over X1, and P(X2) is a marginal distribution over X2.

Important Equation

Marginalization rule, or rule of Total Probability:

This rule gives a way of figuring out a marginal distribution from a joint distribution.

Definitions (4)

Conditional distribution: A conditional distribution over random variable X given random variable Y is written as P(X|Y=e), and is defined as:

Notice that in this conditional distribution, the probabilities for X need to sum to one.

Confusingly, people often say that P(X|Y) is a conditional distribution. However, this is actually a family of many different probability distributions, one for each value of Y.

Important Equation (2)

Marginalization rule, or rule of Total Probability,Second Version:

Really just the same as before, but written slightly differently.

Definitions (5)

Independence: Random variables X and Y are independent (denoted ) if:

P(X, Y) = P(X)P(Y), for all possible events

Conditional Independence: Random variables X and Y are conditionally independent given random variable Z (denoted ) if:

P(X, Y|Z) = P(X|Z)P(Y|Z), for all possible events

Quiz

1. Suppose I flip a coin 3 times. Each time has P(H)=0.5. Assume the three coin tosses are independent. What is P(H, H, H)?

2. Suppose I flip the coin 4 times, and let the random variable for the i-th time be Xi. What is P(X1=X2=X3=X4)?

3. What is the probability that, in 4 coin flips, I get at least 3 heads?

Quiz

Suppose I have one normal coin (P(X1=H)=0.5), and one weird coin with the following properties:

P(X2=H|X1=H)=0.9

P(X2=T|X1=T)=0.8

If I flip X1 and then X2, what is P(X2=H)?

Important Equations so farComplementarity:

Conditional Probability:

Marginalization (or total probability):

or:

Common Mistake: The following looks like the complementarity rule, but there is no guarantee that it is true, and quite often it will not be true:

Very bad, don’t do this:

Quiz: DiabetesP(D)=0.01 (called the prior probability)Test for diabetes is either + or –P(+|D)=0.9P(-|D)=0.8

P(-|D) =P(+|D) =

P(+, D) =P(-, D) =P(+, D) =P(-, D) =

P(D|+) =

Bayes Rule

Bayes Rule:

or:

Proof of Bayes Rule:1. (def. of cond. prob.)2. (also def. of cond. prob.)3. (multiply both sides of step 2 by P(Y))4. (substitute LHS of step 3 into numerator of step 1)

Likelihood

Prior

Marginal Likelihood

Posterior

Bayes Rule for Diabetes Test

P(D|+) = ?