Lecture 2: Conditional Probability

Lecture 2: Conditional Probability

Ziyu Shao

School of Information Science and TechnologyShanghaiTech University

September 25, 2018

Ziyu Shao (ShanghaiTech) Lecture 2: Conditional Probability September 25, 2018 1 / 58

Outline

1 Definition & Intuition

2 Bayes’ Rule & The Law of Total Probability

3 Conditional Probabilities are Probabilities

4 Independence of Events

5 Conditioning As A Problem-Solving Tool

6 Pitfalls & Paradoxes


Outline








Thinking Conditionally

New data & information may affect our uncertainties

Conditional probability: how to update our belief?

All probabilities are conditional! (explicit/implicit background info orassumption)


Thinking Conditionally

Conditioning :a powerful problem-solving strategy

Reducing a complicated probability problem to a bunch of simplerconditional probability problems

First-step analysis: obtain recursive solution to multi-stage problems

Conditioning is the soul of statistics!


Definition of Conditional Probability

Definition

If Aand B are events with P(B) > 0, then the conditional probability of Agiven B, denoted by P(A|B), is defined as

P(A|B) =P(A ∩ B)

P(B).

P(A): prior probability of A.

P(A|B): posterior probability of A.


Outline








Chain Rule

Theorem

For any events A1, . . . ,An with positive probabilities,

P(A1, . . . ,An) = P(A1)P(A2|A1)P(A3|A1,A2) · · ·P(An|A1, . . . ,An−1).


Bayes’ Rule

Theorem

For any events A and B with positive probabilities,

P(A|B) =P(B|A)P(A)

P(B).


The Law of Total Probability (LOTP)

Theorem

Let A1, . . . ,An be a partition of the sample space S (i.e., the Ai aredisjoint events and their union is S), with P(Ai ) > 0 for all i . Then

P(B) =n∑

i=1

P(B|Ai )P(Ai ).


Inference & Bayes’ Rule

Theorem

Let A1, . . . ,An be a partition of the sample space S (i.e., the Ai aredisjoint events and their union is S), with P(Ai ) > 0 for all i . Then forany event B such that P(B) > 0, we have

P(Ai |B) =P(Ai )P(B|Ai )

P(A1)P(B|A1) + · · ·+ P(An)P(B|An).


Example: Bayes Spam Filter

A spam filter is designed by looking at commonly occurring phrases inspam. Suppose that 80% of email is spam. In 10% of the spam emails,the phrase “free money” is used, whereas this phrase is only used in 1% ofnon-spam emails. A new email has just arrived, which does mention “freemoney”. What is the probability that it is spam?


Example: Bayes Spam Filter

Bayes’ rule is used several times in the context of spam:

Compute the probability that the message is spam, knowing that agiven word appears in this message

Compute the probability that the message is spam, taking intoconsideration all of its words (or a relevant subset of them)

Sometimes a third time, to deal with rare words.

Reference: https:

//en.wikipedia.org/wiki/Naive_Bayes_spam_filtering


Example: Random Coin

You have one fair coin, and one biased coin which lands Heads withprobability 3/4. You pick one of the coins at random and flip it threetimes. It lands Heads all three times. Given this information, what is theprobability that the coin you picked is the fair one?


https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering

https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering

Example: Communication Channel


Example: Communication Channel

Suppose that Alice sends only one bit (a 0 or 1) to Bob, with equalprobabilities. If she sends a 0, there is a 5% chance of an error occurring,resulting in Bob receiving a 1; if she sends a 1, there is a 5% chance of anerror occurring, resulting in Bob receiving a 0. Given that Bob receives a1, what is the probability that Alice actually sent a 1?


Outline








Conditional Probability

Condition on an event E , we update our beliefs to be consistent withthis knowledge.

P(·|E ) is also a probability function with sample space S :I 0 ≤ P(·|E ) ≤ 1 with P(S |E ) = 1 and P(∅|E ) = 0.I If events A1,A2, . . . are disjoint, then P(∪∞j=1Aj |E ) =

∑∞j=1 P(Aj |E ).

I P(Ac |E ) = 1− P(A|E ).I Inclusion-exclusion: P(A ∪ B|E ) = P(A|E ) + P(B|E )− P(A ∩ B|E ).


Bayes’ Rule with Extra Conditioning

Theorem

Provided that P(A ∩ E ) > 0 and P(B ∩ E ) > 0, we have

P(A|B,E ) =P(B|A,E )P(A|E )

P(B|E ).


LOTP with Extra Conditioning

Theorem

Let A1, . . . ,An be a partition of the sample space S (i.e., the Ai aredisjoint events and their union is S), with P(Ai ∩ E ) > 0 for all i . Then

P(B|E ) =n∑

i=1

P(B|Ai ,E )P(Ai |E ).



You have one fair coin, and one biased coin which lands Heads withprobability 3/4. You pick one of the coins at random and flip it threetimes. It lands Heads all three times. If we toss the coin a fourth time,what is the probability that it will land Heads once more?




Approaches for Finding P(A|B ,C )

Think of B,C as the single event B ∩ C .

Use Bayes’ rule with extra conditioning on C .

Use Bayes’ rule with extra conditioning on B.


Outline








Independence of Two Events

Definition

Events A and B are independent if

P(A ∩ B) = P(A)P(B).

If P(A) > 0 and P(B) > 0, then this is equivalent to

P(A|B) = P(A)

P(B|A) = P(B)


Independence vs. Disjointness


Independence of Complementary Set

Theorem

If A and B are independent, then A and Bc are independent, Ac and B areindependent, and Ac and Bc are independent.


Independence of Three Events

Definition

Events A, B and C are independent if all of the following equations hold:

P(A ∩ B) = P(A)P(B)

P(A ∩ C ) = P(A)P(C )

P(B ∩ C ) = P(B)P(C )

P(A ∩ B ∩ C ) = P(A)P(B)P(C ).


Pairwise Independence and Independence

Consider two fair, independent coin tosses.

A: the event that the first is Heads.

B: the event that the second is Heads.

C: the event that both tosses have the same result.


Conditional Independence

Definition

Events A and B are said to be conditionally independent given E if:

P(A ∩ B|E ) = P(A|E )P(B|E ).


Example: Conditional Independence 6=⇒ Independence

We choose either a fair coin or a biased coin (w.p. 3/4 of landingHeads).

But we do not know which one we have chosen and we flip it twice.

Event F : “chosen the fair coin”

Event A1: “the first coin tosses landing Heads”

Event A2: “the second coin tosses landing Heads”


Example: Independence 6=⇒ Conditional Independence

Only Alice and Bob call me.

Each day, they call me independently.

Event R: “Phone rings”.

Event A: “Alice call me”

Event B: “Bob call me”


Example: Conditional Independence Given E vs. E c

There are two classes: good & bad.

Good: students get grade A with working hard.

Bad: students get grades randomly regardless of their efforts.

Event E : “Class is good”.

Event W : “Students working hard”.

Event A: “Students receive grade A”.


Outline








Example: Laplace’s Rule of Succession

There are k + 1 coins in a box. When flipped, the i th coin will turn upheads with probability i/k , i = 0, 1, . . . , k. A coin is randomly selectedfrom the box and is then repeatedly flipped. If the first n flips all result inheads, what is the conditional probability that the (n + 1)st flip will dolikewise?


Solution of Laplace’s Rule of Succession


Example: Branching Process

A single amoeba, Bobo, lives in a pond. After one minute Bobo will eitherdie, split into two amoebas, or stay the same, with equal probability, andin subsequent minutes all living amoebas will behave the same way,independently. What is the probability that the amoeba population willeventually die out?


First-Step Analysis: Branching Process


Example: Gambler’s Ruin

Two gamblers, A and B, make a sequence of dollar 1 bets. In each bet,gambler A has probability p of winning, and gambler B has probabilityq = 1− p of winning. Gambler A starts with i dollars and gambler B startswith N − i dollars; the total wealth between the two remains constantsince every time A loses a dollar, the dollar goes to B, and vice versa. Thegame ends when either A or B is ruined (run out of money). What is theprobability that A wins the game (walking away with all the money)?


First-Step Analysis: Gambler’s Ruin


First-Step Analysis: Gambler’s Ruin


Generating Functions for Recurrence Equations








Outline








Example of Simpson’s Paradox


Math Behind Simpson’s Paradox

If

P(A|B,C ) < P(A|Bc ,C )

,P(A|B,C c) < P(A|Bc ,C c),

then is it possible that

P(A|B) > P(A|Bc)?


Monty Hall Problem

On the game show Let’s Make a Deal, hosted by Monty Hall, a contestantchooses one of three closed doors, two of which have a goat behind themand one of which has a car. Monty, who knows where the car is, thenopens one of the two remaining doors. The door he opens always has agoat behind it (he never reveals the car!). If he has a choice, then he picksa door at random with equal probabilities. Monty then offers thecontestant the option of switching to the other unopened door. If thecontestant’s goal is to get the car, should she switch doors?


Remarks

Even great mathematician Paul Erdos & Persi Diaconis mademistakes.

Originally proposed by Steve Selvin, American Statistician 1975.

Famous when proposed in Marilyn vos Savant’s “Ask Marilyn”column, Parade magazine 1990.

Approximately 10000 readers, including nearly 1000 with PhDs, saidno need to switch.


Solution of Monty Hall Problem


Tree Diagram of Monty Hall Problem


Information Conveyed by Monty’s Action


Setting of Classical Monty Hall Problem

Strategies of the contestant: switching & no switching

Monty knows the location of the car

Homogeneous doors

Three doors


Variant 4: Heterogeneous Doors

Suppose the car is not placed randomly behind the three doors. Instead,the car is behind door one with probability p1, behind door two withprobability p2, and behind door three with probability p3. Herep1 + p2 + p3 = 1 and p1 ≥ p2 ≥ p3 > 0. You are to choose one of thethree doors, after which Monty will open a door he knows to conceal agoat. Monty always chooses randomly from among his options in thosecases where your initial choice is correct. What strategy should you follow?


Solution of Variant 4


Summary : Conditional Probability & Inference


Reading Assignment

Next lecture we will study discrete random variable. Please readcorresponding chapters in required textbooks:

Chapter 3 of BH

Chapter 2 of BT


Documents

Lecture 2: Conditional Probability