07: Variance, Bernoulli, Binomial - Stanford...

07: Variance, Bernoulli, BinomialJerry CainApril 12, 2021

Lisa Yan, Chris Piech, Mehran Sahami, and Jerry Cain, CS109, Spring 2021

Quick slide reference

3 Variance 07a_variance_i

10 Properties of variance 07b_variance_ii

17 Bernoulli RV 07c_bernoulli

22 Binomial RV 07d_binomial

34 Exercises LIVE

Variance

07a_variance_i

Stanford, CA𝐸 high = 68°F𝐸 low = 52°F

Washington, DC𝐸 high = 67°F𝐸 low = 51°F

Average annual weather

Is 𝐸 𝑋 enough?

Washington, DC𝐸 high = 67°F𝐸 low = 51°F

Average annual weather

𝑃𝑋=𝑥

Stanford high temps Washington high temps

35 50 65 80 90 35 50 65 80 90

68°F 67°F

Normalized histograms are approximations of PMFs.

Variance = “spread”Consider the following three distributions (PMFs):

• Expectation: 𝐸 𝑋 = 3 for all distributions• But the “spread” in the distributions is different!• Variance, Var 𝑋 : a formal quantification of “spread”

Variance

The variance of a random variable 𝑋 with mean 𝐸 𝑋 = 𝜇 is

Var 𝑋 = 𝐸 𝑋 − 𝜇 (

• Also written as: 𝐸 𝑋 − 𝐸 𝑋 !

• Note: Var(X) ≥ 0• Other names: 2nd central moment, or square of the standard deviation

Var 𝑋

def standard deviation SD 𝑋 = Var 𝑋7

Units of 𝑋!

Units of 𝑋

Variance of Stanford weather

Varianceof 𝑋

Var 𝑋 = 𝐸 𝑋 − 𝐸 𝑋 !

𝑃𝑋=𝑥

Stanford high temps

35 50 65 80 90

𝑋 𝑋 − 𝜇 !

57°F 121 (°F)2

𝐸 𝑋 = 𝜇 = 68°F

71°F 9 (°F)2

75°F 49 (°F)2

69°F 1 (°F)2

… …

Variance 𝐸 𝑋 − 𝜇 ! = 39 (°F)2

Standard deviation = 6.2°F

Stanford, CA𝐸 high = 68°F

Washington, DC𝐸 high = 67°F

Comparing variance

𝑃𝑋=𝑥

Stanford high temps Washington high temps

35 50 65 80 90 35 50 65 80 90

Var 𝑋 = 39 °F ! Var 𝑋 = 248 °F !

Varianceof 𝑋

68°F 67°F

Properties of Variance

07b_variance_ii

Properties of varianceDefinition Var 𝑋 = 𝐸 𝑋 − 𝐸 𝑋 !

def standard deviation SD 𝑋 = Var 𝑋

Property 1 Var 𝑋 = 𝐸 𝑋! − 𝐸 𝑋 !

Property 2 Var 𝑎𝑋 + 𝑏 = 𝑎!Var 𝑋

Units of 𝑋!

Units of 𝑋

• Property 1 is often easier to compute than the definition• Unlike expectation, variance is not linear

Units of 𝑋!

Units of 𝑋

Computing variance, a proof

= 𝐸 𝑋! − 𝐸 𝑋 !

= 𝐸 𝑋! − 𝜇!= 𝐸 𝑋! − 2𝜇! + 𝜇!= 𝐸 𝑋! − 2𝜇𝐸 𝑋 + 𝜇!

= 𝐸 𝑋 − 𝜇 ! Let 𝐸 𝑋 = 𝜇

Everyone, please

welcome the second

moment!

Varianceof 𝑋

= 𝐸 𝑋! − 𝐸 𝑋 !

𝑥 − 𝜇 !𝑝 𝑥

𝑥! − 2𝜇𝑥 + 𝜇! 𝑝 𝑥

𝑥!𝑝 𝑥 − 2𝜇,"

𝑥𝑝 𝑥 + 𝜇!,"

𝑝 𝑥

Let Y = outcome of a single die roll. Recall 𝐸 𝑌 = 7/2 .Calculate the variance of Y.

Variance of a 6-sided dieVarianceof 𝑋

= 𝐸 𝑋! − 𝐸 𝑋 !

1. Approach #1: Definition

Var 𝑌 =161 −

!+162 −

+163 −

!+164 −

+165 −

!+166 −

2. Approach #2: A property

𝐸 𝑌! =161! + 2! + 3! + 4! + 5! + 6!

= 91/6

Var 𝑌 = 91/6 − 7/2 !

= 35/12 = 35/12

2nd moment

Units of 𝑋!

Units of 𝑋

Property 2: A proofProperty 2 Var 𝑎𝑋 + 𝑏 = 𝑎!Var 𝑋

Var 𝑎𝑋 + 𝑏= 𝐸 𝑎𝑋 + 𝑏 ! − 𝐸 𝑎𝑋 + 𝑏 ! Property 1

= 𝐸 𝑎!𝑋! + 2𝑎𝑏𝑋 + 𝑏! − 𝑎𝐸 𝑋 + 𝑏 !

= 𝑎!𝐸 𝑋! + 2𝑎𝑏𝐸 𝑋 + 𝑏! − 𝑎! 𝐸[𝑋] ! + 2𝑎𝑏𝐸[𝑋] + 𝑏!

= 𝑎!𝐸 𝑋! − 𝑎! 𝐸[𝑋] !

= 𝑎! 𝐸 𝑋! − 𝐸[𝑋] !

= 𝑎!Var 𝑋 Property 1

Proof:

Factoring/Linearity of Expectation

Bernoulli RV

07c_bernoulli

Consider an experiment with two outcomes: “success” and “failure.”def A Bernoulli random variable 𝑋 maps “success” to 1 and “failure” to 0.

Other names: indicator random variable, boolean random variable

Examples:• Coin flip• Random binary digit• Whether a disk drive crashed

Bernoulli Random Variable

𝑃 𝑋 = 1 = 𝑝 1 = 𝑝𝑃 𝑋 = 0 = 𝑝 0 = 1 − 𝑝𝑋~Ber(𝑝)

Support: {0,1} VarianceExpectation

𝐸 𝑋 = 𝑝Var 𝑋 = 𝑝(1 − 𝑝)

Remember this nice property of expectation. It will come back!

Jacob Bernoulli

Jacob Bernoulli (1654-1705), also known as “James”, was a Swiss mathematician

One of many mathematicians in Bernoulli familyThe Bernoulli Random Variable is named for him

My academic great14 grandfather

Defining Bernoulli RVs

Serve an ad.• User clicks w.p. 0.2• Ignores otherwise

Let 𝑋: 1 if clicked

𝑋~Ber(___)𝑃 𝑋 = 1 = ___𝑃 𝑋 = 0 = ___

Roll two dice.• Success: roll two 6’s• Failure: anything else

Let 𝑋 : 1 if success

𝑋~Ber(___)

𝐸 𝑋 = ___

𝑋~Ber(𝑝) 𝑝" 1 = 𝑝𝑝" 0 = 1 − 𝑝𝐸 𝑋 = 𝑝

Run a program• Crashes w.p. 𝑝• Works w.p. 1 − 𝑝

Let 𝑋: 1 if crash

𝑋~Ber(𝑝)𝑃 𝑋 = 1 = 𝑝𝑃 𝑋 = 0 = 1 − 𝑝 🤔

Defining Bernoulli RVs

Serve an ad.• User clicks w.p. 0.2• Ignores otherwise

Let 𝑋: 1 if clicked

𝑋~Ber(___)𝑃 𝑋 = 1 = ___𝑃 𝑋 = 0 = ___

Roll two dice.• Success: roll two 6’s• Failure: anything else

Let 𝑋 : 1 if success

𝑋~Ber(___)

𝐸 𝑋 = ___

𝑋~Ber(𝑝) 𝑝" 1 = 𝑝𝑝" 0 = 1 − 𝑝𝐸 𝑋 = 𝑝

Run a program• Crashes w.p. 𝑝• Works w.p. 1 − 𝑝

Let 𝑋: 1 if crash

𝑋~Ber(𝑝)𝑃 𝑋 = 1 = 𝑝𝑃 𝑋 = 0 = 1 − 𝑝

Binomial RV

07d_binomial

Consider an experiment: 𝑛 independent trials of Ber(𝑝) random variables.def A Binomial random variable 𝑋 is the number of successes in 𝑛 trials.

Examples:• # heads in n coin flips• # of 1’s in randomly generated length n bit string• # of disk drives crashed in 1000 computer cluster

(assuming disks crash independently)

Binomial Random Variable

𝑘 = 0, 1, … , 𝑛:

𝑃 𝑋 = 𝑘 = 𝑝 𝑘 = 𝑛𝑘 𝑝! 1 − 𝑝 "#!𝑋~Bin(𝑛, 𝑝)

Support: {0,1, … , 𝑛}

𝐸 𝑋 = 𝑛𝑝Var 𝑋 = 𝑛𝑝(1 − 𝑝)Variance

Expectation

Lisa Yan, Chris Piech, Mehran Sahami, and Jerry Cain, CS109, Spring 2021 24

Reiterating notation

The parameters of a Binomial random variable:• 𝑛: number of independent trials• 𝑝: probability of success on each trial

1. The random variable

2. is distributed as a

3. Binomial 4. with parameters

𝑋 ~ Bin(𝑛, 𝑝)

Reiterating notation

If 𝑋 is a binomial with parameters 𝑛 and 𝑝, the PMF of 𝑋 is

𝑋 ~ Bin(𝑛, 𝑝)

𝑃 𝑋 = 𝑘 = 𝑛𝑘 𝑝( 1 − 𝑝 )*(

Probability Mass Function for a BinomialProbability that 𝑋takes on the value 𝑘

Three coin flipsThree fair (“heads” with 𝑝 = 0.5) coins are flipped.• 𝑋 is number of heads• 𝑋~Bin 3, 0.5

Compute the following event probabilities:

𝑋~Bin(𝑛, 𝑝) 𝑝 𝑘 = 𝑛𝑘 𝑝# 1 − 𝑝 $%#

𝑃 𝑋 = 0

𝑃 𝑋 = 1

𝑃 𝑋 = 2

𝑃 𝑋 = 3

𝑃 𝑋 = 7P(event)

Three coin flipsThree fair (“heads” with 𝑝 = 0.5) coins are flipped.• 𝑋 is number of heads• 𝑋~Bin 3, 0.5

Compute the following event probabilities:

𝑋~Bin(𝑛, 𝑝) 𝑝 𝑘 = 𝑛𝑘 𝑝# 1 − 𝑝 $%#

𝑃 𝑋 = 0 = 𝑝 0 = 30 𝑝$ 1 − 𝑝 % = &

𝑃 𝑋 = 1

𝑃 𝑋 = 2

𝑃 𝑋 = 3

𝑃 𝑋 = 7

= 𝑝 1 = 31 𝑝& 1 − 𝑝 ! = %

= 𝑝 2 = 32 𝑝! 1 − 𝑝 & = %

= 𝑝 3 = 33 𝑝% 1 − 𝑝 $ = &

= 𝑝 7 = 0P(event) PMF

Extra math note:By Binomial Theorem,we can prove∑()$* 𝑃 𝑋 = 𝑘 = 1

𝑋~Bin(𝑛, 𝑝)Range: {0,1, … , 𝑛} Variance

Expectation

𝐸 𝑋 = 𝑛𝑝Var 𝑋 = 𝑛𝑝(1 − 𝑝)

𝑘 = 0, 1, … , 𝑛:

𝑃 𝑋 = 𝑘 = 𝑝 𝑘 = 𝑛𝑘 𝑝! 1 − 𝑝 "#!

Ber 𝑝 = Bin(1, 𝑝)

Binomial RV is sum of Bernoulli RVs

Bernoulli• 𝑋~Ber(𝑝)

Binomial• 𝑌~Bin 𝑛, 𝑝• The sum of 𝑛 independent

Bernoulli RVs

𝑌 =,+)&

𝑋+ , 𝑋+ ~Ber(𝑝)

𝑋~Bin(𝑛, 𝑝)Range: {0,1, … , 𝑛}

𝐸 𝑋 = 𝑛𝑝Var 𝑋 = 𝑛𝑝(1 − 𝑝)Variance

Expectation

PMF 𝑘 = 0, 1, … , 𝑛:

𝑃 𝑋 = 𝑘 = 𝑝 𝑘 = 𝑛𝑘 𝑝! 1 − 𝑝 "#!

Proof:

𝑋~Bin(𝑛, 𝑝)Range: {0,1, … , 𝑛}

𝐸 𝑋 = 𝑛𝑝Var 𝑋 = 𝑛𝑝(1 − 𝑝)

We’ll prove this later in the course

VarianceExpectation

𝑘 = 0, 1, … , 𝑛:

𝑃 𝑋 = 𝑘 = 𝑝 𝑘 = 𝑛𝑘 𝑝! 1 − 𝑝 "#!

No, give me the variance proof right now

proofwiki.org

07: Variance, Bernoulli, Binomial - Stanford...

Documents

Probability (10A) - Wikimedia CommonsA single success / failure experiment a Bernoulli trial or Bernoulli experiment a single trial, i.e., n = 1, the binomial distribution is a Bernoulli

Discrete Probability Distributions Chapter66 Discrete Distributions Uniform Distribution Bernoulli Distribution Binomial Distribution Poisson Distribution

07: Variance, Bernoulli, Binomial · 2020. 4. 20. · Bernoulli, Binomial Lisa Yan April 20, 2020 1. Lisa Yan, CS109, 2020 Quick slide reference 2 3 Variance 07a_variance_i 10 Properties

A Generalized Linear Model for Bernoulli Response …...When the response is Bernoulli or more generally, binomial, the logit link function is one natural choice. However, other link

MATH442601j2 Notebook 2 Fall 2018/20192.2.4 Bernoulli Experiments, Bernoulli and Binomial Distributions A Bernoulli experiment is a random experiment with two possible outcomes. The

Unit 4 The Bernoulli and Binomial Distributions

14 Sep STATISTIKA id - istiarto.staff.ugm.ac.id Discrete Probability... · •Distribusi Hipergeometrik •Bernoulli Processes •Distribusi Binomial •Distribusi Geometrik •Distribusi

07: Variance, Bernoulli, Binomial · 2020-01-21 · Bernoulli, Binomial David Varodayan January 22, 2020. Adapted from slides by Lisa Yan. 1. Discrete random variables 2 Discrete

Project Prove that the cumulative binomial distributions ...thinkingwriting.qmul.ac.uk/wishees/collections/... · A Bernoulli trial, which is used in the binomial distribution, is

Bernoulli Experiments, Binomial Distributiondgalvin1/10120/10120_S16/Topic19_8p... · 2016. 3. 18. · Bernoulli Experiments, Binomial Distribution If a person randomly guesses the

6. Bernoulli and Binomial 2018 - UMassbiep540w/pdf/6. Bernoulli and... · 2018. 10. 17. · The Bernoulli and Binomial probability distribution models are often very good models of

Lecture 8 bernoulli binomial poisson

Binomial Random Variables. Binomial experiment A sequence of n trials (called Bernoulli trials), each of which results in either a “success” or a “failure”

1 Week 3.. 2 3 NORMAL DISTRIBUTION BERNOULLI TRIALS BINOMIAL DISTRIBUTION EXPONENTIAL DISTRIBUTION UNIFORM DISTRIBUTION POISSON DISTRIBUTION

Bernoulli and Binomial Random Variables

20: Maximum Likelihood Estimationweb.stanford.edu/.../20_mle_blank-announcements.pdf5. / = 12 +4 11 0 = 1 = Bernoulli model, parameter 0 = 0.2.! Lisa Yan, CS109, 2020 What are parameters?

6. Bernoulli and Binomial 2021

Deriving the Binomial PMF...Bernoulli and binomial probability distributions A Bernoulli random variable models a very simple process. For example, Y = the number of females in one

Some Representations of the Multivariate Bernoulli and Binomial … · 2017-01-27 · Multivariate but vectorized versions for Bernoulli and binomial distributions are established

6. Bernoulli and Binomial 2017 - UMasspeople.umass.edu/biep540w/pdf/6. Bernoulli and Binomial... · 2017. 10. 15. · The Bernoulli and Binomial probability distribution models are