Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Lecture 05:IndependenceLisa YanJuly 6, 2018
Announcements
PS1 due today◦ Pain poll◦ Last day to hand in + solutions released next Friday 7/13
PS2 out today! Due next Friday 7/13
2
Goals for today
◦ Properties of probability◦ Independence◦Mutual exclusivity, independence, and DeMorgan’s◦Conditional Independence
3
Summary (conditional probability)
5
P(EF) = P(E) P(F|E) = P(F) P(E|F)
P(E) = P(F)P(E|F) + P(Fc) P(E|Fc)
P(F|E)
P(E|F)P(F)P(E|F)P(F) + P(E|Fc)P(Fc)
Definition ofConditional Probability
Bayes’Theorem
Chain Rule
Law of Total Probability
P(E|F)P(F)P(E)
P(EF)P(E)
= P(E|F)P(F)!
iP Fi P E Fi
=
!i
P Fi P E Fi=
Bayesian Interpretations
6
Define:E: the effect (observation, symptom, test result)H: the hypothesis
Prior: Probability of H before you observe ELikelihood: Probability of E given that H holdsPosterior: Probability of H after you observe E
! " # = ! # " ! "! #
posteriorlikelihood prior
Normalizing constant
!
DNA Paternity testing
Child is born with (A, a) gene pair (event BA,a)
◦ Mother has (A, A) gene pair
◦ Two possible fathers: M1: (a, a) M2: (a, A)
◦ P(M1) = p P(M2) = 1 - p
What is P(M1 | BA,a)?
7
P(M1 | BA,a) = P(M1 BA,a) / P(BA,a)
M1 more likely to be father
than he was before, since
P(M1 | BA,a) > P(M1)
Solution:
P(BA,a | M1)P(M1)
P(BA,a | M1)P(M1) + P(BA,a | M2)P(M2)=
1 (p)
1 (p) + (1/2) (1 – p)= =
2p
1+p> p
!
Properties of probability
8
For an event A:P(A) = 1 – P(Ac) (Complement)
For any events A and B:
P( A B ) = P( B A ) (Commutativity)P( A Bc ) = P( A ) – P( AB ) (Intersection)P( A ) = P( A B ) + P( A Bc ) (Law of Total Probability)
= P(A | B) P(B) + P(A | Bc) P(Bc)
P(A B) = P(B) P(A|B) (Chain Rule)P( A B ) ≥ P( A ) + P( B ) – 1 (Bonferroni)
!
P(•|G) is also a probability space!Formally, P(•|G) satisfies 3 axioms of probability.For example:
Axiom 1 of probability 0 £ P(E|G) £ 1
Corollary 1 (complement) P(Ec|G) = 1 – P(E|G)
9
Chain Rule: P(EF|G) = P(F|G) P(E|FG)
conditioning on G means that G is always given!
P(E|FG)P(F|G)P(E|G)
Bayes Theorem: P(F|EG) =
P(E|G) = P(F|G) P(E|FG) + P(Fc|G) P(E|FcG)Law of Total Probability:
More generally:
And now…
10
!
Two DiceRoll two 6-sided dice, yielding values D1 and D2.
Let E be event: D1 = 1Let F be event: D2 = 1
1. What is P(E), P(F), and P(EF)?
P(E) = 1/6, P(F) = 1/6, P(EF) = 1/36
P(EF) = P(E) P(F) à E and F independent
2. Let G be event: D1 + D2 = 5.What is P(E), P(G) and P(EG)?
P(E) = 1/6, P(G) = 4/36 = 1/9, P(EG) = 1/36
P(EG) ¹ P(E) P(G) à E and G dependent
11
G = {(1,4), (2,3), (3,2), (4,1)}
!
IndependenceTwo events E and F are defined as independent if:
P(EF) = P(E) P(F)Or, equivalently: P(E|F) = P(E)(otherwise E and F are called dependent events).
Three events E, F, and G are independent if:
P(EFG) = P(E) P(F) P(G), andP(EF) = P(E) P(F), andP(EG) = P(E) P(G), andP(FG) = P(F) P(G)
12
Pairwise independence of3 variables is not enoughto prove independence!
Independence?
13
A
B
S
P(AB) = P(A) P(B)
A
B
AB
S
!
When E and F are independent…Given independent events E and F, prove:
P(E|F) = P(E | Fc)
14
Proof:P(E Fc) = P(E) – P(EF) Intersection
= P(E) – P(E) P(F) Independence= P(E) [1 – P(F)] Factoring= P(E) P(Fc) Complement
So, E and Fc independent, implying that:P(E | Fc) = P(E) = P(E | F)
Intuitively, if E and F are independent, knowing whether F holds gives us no information about E
!
Generalized Independence
Events E1, E2, ..., En are independent if for every subset E1’, E2’, ..., Er’ (where r £ n) it holds that:
P(E1E2...Er ) = P(E1 ) P(E2) P(E3)… P(Er)
Example:Outcomes of n separate flips of a coin are all independent of one another.Each flip in this case is a trial of the experiment.
15
!
Two!!!! Dice!!!!Roll two 6-sided dice, yielding values D1 and D2.
Let E be event: D1 = 1Let F be event: D2 = 6Let G be event: D1 + D2 = 7
16
1. Are E and F independent?
2. Are E and G independent?
3. Are F and G independent?
4. Are E, F, and G independent?
Yes!
Yes!
Yes!
No!
P(E) = 1/6, P(F) = 1/6, P(EF) = 1/36
P(E) = 1/6, P(G) = 1/6, P(EG) = 1/36 EG = { (1,6) }
P(F) = 1/6, P(G) = 1/6, P(FG) = 1/36 FG = { (1,6) }
P(EFG) = 1/36 ¹ 1/216 = (1/6)(1/6)(1/6) EFG = { (1,6) }
G = { (1,6), (2,5), (3,4),(4,3), (5,2), (6,1) }
Generating Random BitsA computer produces a series of random bits.• with probability p of producing a 1.• Each bit generated is an independent trial.• E = first n bits are 1’s, followed by a single 0What is P(E)?
17
P(first n 1’s) = P(1st bit=1) P(2nd bit=1) ... P(nth bit=1)
= pn
P(n+1 bit=0) = (1 – p)
P(E) = P(first n 1’s) P(n+1 bit=0) = pn (1– p)
Solution: Independent trials
!
(biased) Coin Flips
Suppose we flip a coin n times.
• A coin comes up heads with probability p.
• Each coin flip is an independent trial.
18
1. P(n heads on n coin flips)
2. P(n tails on n coin flips)
3. P(first k heads, then n – k tails)
4. P(exactly k heads on n coin flips)
!" 1 − ! %&"
1 − ! %!%
'( !" 1 − ! %&"
BreakAttendance: t inyurl.com/cs109summer2018
19
!
Probability in a nutshell
20
Just Add! Inclusion Exclusion
Chain Rule
Just Multiply!
DeMorgan’s
Mutually Exclusive?
ORP(E È F)
ANDP(E F)
Independent?
Level 6 Level 30
Our goal today:
Level 15
Hash Tablesm strings are hashed (equally randomly) into a hash table with n buckets.• Each string hashed is an independent trial.• Let event E = at least one string hashed into first bucket.What is P(E)?
21
Define: Fi = string i not hashed into first bucket (where 1 ≤ i ≤ m)F1F2…Fm = no strings hashed into first bucket
WTF: P(E) = 1 – P(F1F2…Fm)
P(Fi) = 1 – 1/n = (n – 1)/n (for all 1 ≤ i ≤ m)
P(F1F2…Fm) = P(F1) P(F2) … P(Fm) = ((n – 1)/n)m
P(E) = 1 – P(F1F2…Fm) = 1– ((n– 1)/n)mSimilar to ≥ 1 of m people having same birthday as you.
Solution:Find P(Ec)!!!!
!
More Hash Table Funm strings are hashed (unequally) into a hash table with n buckets• Each string hashed is an independent trial, with
probability pi of getting hashed to bucket i• E = At least 1 of buckets 1 to k has ≥ 1 string hashed to it
22
Define: Fi = at least one string hashed into i-th bucket (where 1 ≤ i ≤ n)
WTF: P(E) = P(F1ÈF2È…ÈFk) = 1 – P((F1ÈF2È…ÈFk)c)
= 1 – P(F1c F2
c …Fkc) (DeMorgan’s Law)
P(F1c F2
c …Fkc) = P(no strings hashed to buckets 1 to k)
= (1 – p1 – p2 – … – pk)m
P(E) = 1– (1– p1 – p2 – …– pk)m
Solution:
No seriously, it’s more Hash Table Funm strings are hashed (unequally) into a hash table with n buckets• Each string hashed is an independent trial, with
probability pi of getting hashed to bucket i• E = Each of buckets 1 to k has ≥ 1 string hashed to it
23
Define: Fi = at least one string hashed into i-th bucket
P(E) = P(F1 F2 … Fk) = 1 – P((F1 F2 … Fk)c)
= 1 – P(F1c È F2
c È … È Fkc) (DeMorgan’s Law)
= 1 − # ∪%&'( )&* = 1– ∑.'(% −1 ./( ∑&01⋯1&3 # )&0* )&4* …)&3*
Where # )&0* )&4* …)&3* = (1 − 8&0 − 8&0…– 8&0):
Solution:
General Inclusion-ExclusionPrinciple of Probability
!
Network reliability
Consider the following parallel network:• n independent routers, each with
probability pi of functioning (where 1 ≤ i ≤ n)
• E = functional path from A to B exists.
What is P(E)?
24
P(E) = 1 – P(all routers fail)
= 1 – (1 – p1)(1 – p2)…(1 – pn)
p1p2
pn…
A B
Solution:
= 1 −$%&'
(1 − )%
Find P(Ec)!!!!
Digging deeper on independence
Two events E and F are defined as independent if:P(EF) = P(E) P(F)
If E and F are independent, does that tell us whether the following is true?
P(EF | G) = P(E | G) P(F | G),where G is an arbitrary event?
In general, No!
25
!
Two events E and F are defined as conditionally independent given G if:
P(EF | G) = P(E | G) P(F | G)Or, equivalently: P(E|FG) = P(E | G)(otherwise E and F are called conditionally dependent given G).
Warning: After conditioning on additional information,
1. Independent events can become conditionally dependent.
2. Dependent events can become conditionally independent.
Conditional Independence
26
Two…Dice…Roll two 6-sided dice, yielding values D1 and D2.
Let E be event: D1 = 1Let F be event: D2 = 6Let G be event: D1 + D2 = 7
1. Are E and F independent?
2. Are E and F independent given G?
27
Yes!
Independent events can become conditionally dependent.
No!
P(E) = 1/6, P(F) = 1/6, P(EF) = 1/36
P(E|G) = 1/6, P(F|G) = 1/6, P(EF|G) = 1/6P(EF|G) ¹ P(E|G) P(F|G) à E|G and F|G dependent
Faculty NightIn a dorm with 100 students:
30 students have straight A’s P(A) = 0.3020 students are in CS P(CS) = 0.206 students are in CS with straight A’s P(A,CS) = 0.06
At faculty night F, only CS students and straight A students show up (44 total).
1. Are A and CS independent?
2. Are A and CS independent given F?
Which is correct?
• Being a CS major lowers your probability of getting A’s
• Knowing you went to faculty night because you are a CS majormakes it less likely that you went because you have straight A’s
28
P(A|CS) = 6/20 = 0.30 = P(A) Yes!
P(A|CS,F) = 0.30 ¹ 30/44 = P(A|F) No!
Summary
29
Two events E and F are defined as independent if:P(EF) = P(E) P(F)
Or, equivalently: P(E|F) = P(E)If E and F are independent, then E and Fc are also independent, and
P(E|F) = P(E) = P(E|Fc)Two events E and F are defined as conditionally independent given G if:
P(EF | G) = P(E | G) P(F | G)Or, equivalently: P(E|FG) = P(E | G)Important notes:1. Independence does not imply causality; it’s just math.2. Conditioning on an event can break independence, or
it can make dependent variables conditionally independent.
Summary
30
Just Add! Inclusion Exclusion
Chain Rule
Just Multiply!
DeMorgan’s
Mutually Exclusive?
ORP(E È F)
ANDP(E F)
Independent?