Bayesian Models

Agenda

• Project• WebCT• Late HW• Math

– Independence– Conditional Probability– Bayes Formula & Theorem

• Steyvers, et al 2003

Independence

• Two events A and B are independent if the occurrence of A has no influence on the probability of the occurrence of B.– Independent: “It doesn’t matter who is elected

president, the world will still be a mess.”– Not independent: “If candidate B is elected

president, the probability that the world will be a mess is 99%. If candidate A is elected, the probability that the world will be a mess will be lowered to 98%.”

Independence• A and B are independent if P(AB) = P(A) x

P(B).– Independent:

• Pick a card from a deck. • A = “The card is an ace”, P(A) = 1/13.• B = “The card is a spade”, P(B) = 1/4• P(AB) = 1/13 x 1/4 = 1/52.

– Not independent: • Draw two cards from a deck without replacement. • A = “The first card is a space”, P(A) = 1/4• B = “The second card is a spade”, P(B) = 1/4• P(AB) = (13 x 12) / (52 x 51) < 1/4 x 1/4.

Conditional Probability

• Example:– What is the probability that a husband will

vote Democrat given that his wife does?– P(HusbandDemocrat|WifeDemocrat)– This is different from:

• What is the probability that a husband will vote democrat?

• What is the probability that a hustband and wife will vote democrat?

All possible events

Conditional Probability

• P(B|A) is the conditional probability that B will occur given that A has occurred: P(B|A) = P(BA) / P(A).

P(A) = A occurs

P(B) = B occurs

P(BA) = A and B occur

Conditional Probability

• Suppose we roll two dice– A = “The sum is 8”

• A = {(2,6), (3,5), (4,4), (5,3), (6,2)}• P(A) = 5/36

– B = “The first die is 3”• B = {(3,1), (3,2), (3,3), (3,4), (3,5), (3,6)}• P(B) = 6/36

– AB = {(3,5)}– P(B|A) = (1/36)/(5/36) = 1/5.

Conditional Probability36

5 61

P(A) = 5/36P(B) = 6/36P(BA) = 1/36P(B|A) = (1/36)/(5/36) = 1/5

P(B|A) = P(BA)/P(A)

Bayes FormulaSuppose families have 1, 2, or 3 children with 1/3probability each. Bobby has no brothers. What is the probability he is an only child?

Bayes FormulaSuppose families have 1, 2, or 3 children with 1/3probability each. Bobby has no brothers. What is the probability he is an only child?

Let child1, child2, and child3 be the events thatA family has 1, 2, or 3 children, respectively.

Let boy1 be the event that a family has only 1 boy.

Want to compute P(child1|boy1) = P(child1boy1)/P(boy1)

Bayes FormulaSuppose families have 1, 2, or 3 children with 1/3probability each. Bobby has no brothers. What is the probability he is an only child?

Want to compute P(child1|boy1) = P(child1boy1)/P(boy1)

We need to compute P(child1boy1) and P(boy1)

Bayes Formula

We need to compute P(child1boy1) and P(boy1)

Because P(B|C) = P(CB) / P(C), we can write: P(CB) = P(C) P(B|C).

So, P(child1boy1) = P(child1)P(boy1| child1)= 1/3 x 1/2 = 1/6.

Bayes Formula

We need to compute P(child1boy1) and P(boy1)

P(boy1) = P(child1boy1) + P(child2boy1) + P(child3boy1)

We know P(child1boy1) = 1/6.Likewise,

P(child2boy1) = 1/6P(child3boy1) = 1/8

P(boy1) = 1/6 + 1/6 + 1/8

Bayes FormulaSuppose families have 1, 2, or 3 children with 1/3probability each. Bobby has no brothers. What is the probability he is an only child?

P(child1|boy1) = P(child1boy1)/P(boy1) = (1/6) / (1/6 + 1/6 + 1/8) = 4/11

Bayes Formula

1 Child120

2 Children120

3 Children120

Suppose families have 1, 2, or 3 children with 1/3probability each. Bobby has no brothers. What is the probability he is an only child?

1 boy60

1 boy60

1 boy45

Bayes Formula

1 Child120

2 Children120

3 Children120

1 boy60

1 boy60

1 boy45

P(child1|boy1) = P(child1boy1)/P(boy1) = (60/360) / ((60+60+45)/360) = 60/165 = 4/11

Bayes Formula

Event1 Event2 Eventn

Sub-event

Sub-event

Sub-event

…

Known: P(Eventi), P(Eventi) = 1, and P(Subevent|Eventi)

Compute: P(Event1|Subevent)

Bayes Formula

Event1 Event2 Eventn

Sub-event

Sub-event

Sub-event

…

P(Event1|Subevent) = P(Event1Subevent) / P(Subevent)

P(EventiSubevent) = P(Eventi)P(Subevent|Eventi)P(Subevent) = P(EventiSubevent) = P(Eventi)P(Subevent|Eventi)

Bayes Formula

Event1 Event2 Eventn

Sub-event

Sub-event

Sub-event

…

P(Event1)P(Subevent|Event1)P(Event1|Subevent) = -----------------------------------------

P(Eventi)P(Subevent|Eventi)

Bayes Formula

1% of the population has a disease.99% of the people who have the disease have the symptoms.10% who don’t have the disease have the symptoms.

Let D = “A person has the disease”.Let S = “A person has the symptoms”.

P(D) = .01 and so P(~D) = .99P(S|D) = .99 and P(S|~D) = .10

What is P(D|S)?

Bayes Formula P(D) P(S|D)

P(D|S) = -------------------------------------------- P(D) P(S|D) + P(~D) P(S|~D)

= (.01 x .99) / (.01 x .99 + .99 x .10) = .091

Bayes Formula P(Event1)P(Subevent|Event1)

P(Event1|Subevent) = ----------------------------------------- P(Eventi)P(Subevent|Eventi)

If there are a very large portion of events, the denominator may be very hard to calculate.

If, however, you are only interested in relative probabilities…

Bayes Formula P(Event1)P(Subevent|Event1)

P(Event1|Subevent) = ----------------------------------------- P(Eventi)P(Subevent|Eventi)

P(Event1|Subevent) P(Event1)P(Subevent|Event1)--------------------------- = ----------------------------------------P(Event2|Subevent) P(Event2)P(Subevent|Event2)

This is called the “odds”.

Bayes Formula• Let’s say you have 2 hypotheses (or

models), H1 and H2, under consideration.

• The log odds of these two hypotheses given experimental data, D, is:

€

logP(H1 |D)

P(H2 |D)= log

P(H1)

P(H2)

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

= logP(H1)

P(H2)

⎛

⎝ ⎜

⎞

⎠ ⎟+ log

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

Bayes Formula

€

logP(H1 |D)

P(H2 |D)= log

P(H1)

P(H2)

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

= logP(H1)

P(H2)

⎛

⎝ ⎜

⎞

⎠ ⎟+ log

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

Posterior odds =Relative belief in 2 hypotheses given the data. Quantity of interest.

Bayes Formula

€

logP(H1 |D)

P(H2 |D)= log

P(H1)

P(H2)

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

= logP(H1)

P(H2)

⎛

⎝ ⎜

⎞

⎠ ⎟+ log

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

Prior odds =Relative belief in 2 hypotheses Before observing any data. Often assumed to be 1 (0 in log odds).

Bayes Formula

€

logP(H1 |D)

P(H2 |D)= log

P(H1)

P(H2)

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

= logP(H1)

P(H2)

⎛

⎝ ⎜

⎞

⎠ ⎟+ log

P(D |H1)

P(D |H2)

⎛

⎝ ⎜

⎞

⎠ ⎟

Likelihoods =Relative probability of the data,given the two hypotheses. Usuallycomputed from your models…