View
7
Download
0
Category
Preview:
Citation preview
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Random Variables and Distributions
Brian Vegetabile
2017 Statistics BootcampDepartment of Statistics
University of California, Irvine
September 15th, 2016
1 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
A Few Definitions for Random Variables
Definition (Random Variable [Casella and Berger, 2002])
A random variable is a function from a sample space intothe real numbers.
2 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
A Few Definitions for Random Variables
Definition (Random Variable [Schervish, 2014])
Consider an experiment for which the sample space isdenoted by S. A real-valued function that is defined on thespace S is called a random variable. In other words, in aparticular experiment a random variable X would be somefunction that assigns a real number X(s) to each possibleoutcome s ∈ S.
3 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
A Few Definitions for Random Variables
Definition (Random Variable [Dudewicz and Mishra,1988])
A random variable X is a real-valued function with domainΩ [i.e. for each ω ∈ Ω, X(ω) ∈ R = y : −∞ < y < +∞].An n-dimensional random variable (or n-dimensionalrandom vector, or vector random variable),X = (X1, . . . , Xn) is a function with domain Ω and range inEuclidean n-space Rn. [i.e. for each ω ∈ Ω,X(ω) ∈ Rn =(y1, . . . , yn) : −∞ < yi < +∞, (1 ≤ i ≤ n)]
4 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Random Variables
I We can consider a random variable X which is the sumobtained from rolling two fair dice.
I Each die is capable of taking on the valuesRolli = 1, 2, 3, 4, 5, 6, thus the sample space Ω is
Ω =
(1, 1) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6)(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
......
......
......
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)
I And thus we can map values from the sample space
into the reals. For example,
X((1, 6)) = 7
5 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Idea of “Induced Probabilities” I
I We now show how probabilities can be created forrandom variables, the following is from [Casella andBerger, 2002]
I This definition will be investigated more in higher levelprobability and statistics courses
6 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Idea of “Induced Probabilities” II
I From Casella and Berger, consider the sample space Ω,with probability function P and we define a randomvariable X with range X .
I We can define a probability function PX on X in thefollowing way.
I We will observe X = xi, xi ∈ X if and only if theoutcome of the random experiment is an ωj ∈ Ω, suchthat X(ωj) = xi.
I Thus,
PX(X = xi) = P (ωj ∈ Ω : X(ωj) = xi)
7 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - ‘Induced’ Probabilities I
I We can consider a random variable X which is the sumobtained from rolling two fair dice.
I The sample space for the experiment isΩ = (i, j) : i, j = 1, 2, 3, 4, 5, 6.
I We have a probability function P , which describes thechance that each outcome in Ω is possible. That is
P ((i, j)) =1
36for any combination of i, j
since we are considering fair dice.
8 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - ‘Induced’ Probabilities II
I Consider the random variable X, such that for eachω ∈ Ω,
X(ω) = X((i, j)) = i+ j
Therefore X can take on the values2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.
I Now using this fact, we can create probabilities for therandom variable X. For example, we can find theprobability that X = 4.
PX(X = 4) = P (ωj ∈ Ω : X(ωj) = 4)= P ((1, 3), (2, 2), (3, 1))= P ((1, 3)) + P ((2, 2)) + P ((3, 1))
=3
36=
1
12
9 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - ‘Induced’ Probabilities III
I Therefore using the original probability function on thesample space we can find probabilities for the randomvariable X.
I PX is an induced probability function on X .
10 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Discrete Random Variables
I Random variables which can only take on at most acountable number of possible values are considered tobe discrete random variables.
I Each one of these possible values will be assigned somenon-negative probability, therefore we can intuitively asplacing ‘mass’ of probability at each possible value.
11 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Discrete Random Variables
Definition (probability mass function)
If a random variable X is a discrete distribution (that is ittakes on only a countable number of different values) thenthe probability mass function of X is defined as the functionf such that for every real number x
f(x) = PX(X = x) for all x
Since f is a probability function it follows that f(x) ≥ 0 forall x and ∑
all x
f(x) = 1
12 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Probability Mass Function for RollingTwo Dice
I Again consider the random variable X which is the sumobtained from rolling two fair dice.
I Clearly the random variable takes on a countablenumber of different values
I Thus we can construct the probability mass function forthis random variable.
2 4 6 8 10 12
0.00
0.05
0.10
0.15
0.20
Probability Mass Function − Rolling Two Dice
Experimental Outcome
Pro
babi
lity
Mas
s F
unct
ion
0.028
0.056
0.083
0.111
0.139
0.167
0.139
0.111
0.083
0.056
0.028
13 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Continuous Random Variables I
I Continuous random variables are concerned withprobability on intervals.
I From Degroot/Schervisch, a random variable X has acontinuous distribution, or is a continuous randomvariable, if there exists a non-negative function f ,defined on the real line, such that for every subset A ofthe real line, the probability that X takes a value in Ais the integral over the set A.
PX(X ∈ A) =
∫x∈A
f(x)dx
14 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Continuous Random Variables II
I When dealing with intervals (a, b] this becomes
PX(a < X ≤ b) =
∫ b
af(x)dx
I The function f is called the probability density functionof the random variable X. Again f(x) ≥ 0 for all x and∫ ∞
−∞f(x)dx = 1
I Additionally recall from calculus that∫ aa f(x)dx = 0 for
any x. This further implies that
PX(X = x) = 0
for a continuous random variable.
15 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Continuous Random Variable I
I Consider a random variable that measures the timebetween events occurring in an interval of time.
I Notice this is a measure of time and therefore we willonly expect probability density on the non-negativereals.
I Such a random variable is called an exponential randomvariable and has the following density
f(x|λ) =
1λ exp
(−xλ
)if x ≥ 0
0 otherwise
16 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Continuous Random Variable II
0 1 2 3 4 5
0.0
0.5
1.0
1.5
2.0
Various Exponential Distributions
x
Pro
babi
lity
Den
sity
λ = 0.5λ = 1λ = 3λ = 5
17 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
A Quick Aside:Observable Data vs. Unobservable Parameters
I Before continuing this is a good time to point out afundamental issue in statistics
I In our dice example, we are able to make assumptions,based on geometry or other things, which allow us toapriori know the probability distribution for eachexperimental outcome.
I In the continuous distribution example, notice theparameter λ in the distribution.
I It is often impossible to know these parametersbeforehand and thus our goal eventually will be toestimate these parameters.
I Often we have a set of “observed data” and we want tohelp us learn about the “unobservable parameters”
18 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Cumulative Distribution Functions I
I Once we understand the probability functions forrandom variables we can begin to talk about otherfunctions of, and properties of, the random variables.
I The first major function that is considered for everyrandom variable is its cumulative distribution. This isone of the first places that integration will come intoplay.
19 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Cumulative Distribution Functions II
Definition (Cumulative Distribution Function)
The distribution function FX of a random variable X is afunction defined for each real number x as follows:
F (x) = PX(X ≤ x) for all x
I Additionally based on our previous definitions of thedensity function this is equivalent to
F (x) = PX(X ≤ x) =
∫ x
−∞f(x)dx
20 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Relationship between Mass/Density FunctionsAnd Distribution Functions
I From the Fundamental Theorem of Calculus if therandom variable is continuous we have that
F ′(x) =d
dxF (x) = f(x)
21 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Properties of the CDF
i) The function FX(x) is nondecreasing as x increases;that is, if x1 < x2, then FX(x1) ≤ FX(x2).
ii) limx→−∞ FX(x) = 0 and limx→+∞ FX(x) = 1
iii) FX(x) is right continuous.
22 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Discrete Random Variable CDF I
I Consider the function
f(x) =
0.1 if x = 00.2 if x = 10.7 if x = 20 otherwise
I Clearly this is a probability mass function sincef(x) ≥ 0 and
∑2i=0 f(x) = 1
I To find the distribution function must have a rightcontinuous function where we find P (X ≤ x) for all x
23 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Discrete Random Variable CDF II
I Therefore, by the mass function we have thatP (X < 0) = 0, and we see that P (X ≤ 0) = 0.1.
I Now since the next ‘mass jump’ is at x = 1 thefunction remains constant up until that point and forexample P (X ≤ 0.5) = 0.1
I Once x = 1 though we add mass and haveP (X ≤ 1) = 0.3 and finally P (X ≤ 2) = 1.
I We can see how this is a ‘cumulative’ function.
24 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Discrete Random Variable CDFI Continuing with our rolling two dice example we can
create the CDF of this distribution by consideredP (X ≤ x) for all x.
I Therefore we obtain the following function,
2 4 6 8 10 12
0.0
0.2
0.4
0.6
0.8
1.0
Cumulative Distribution Function − Rolling Two Dice
Experimental Outcome
Cum
ulat
ive
Dis
trib
utio
n F
unct
ion
0.0280.083
0.167
0.278
0.417
0.583
0.722
0.8330.917
0.972
25 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Continuous Random Variable CDF I
I For the exponential distribution we can integrate theprobability density function to obtain the cumulativedistribution function
I Recall f(x|λ) = 1λ exp
(−xλ
)and F (x) =
∫ x−∞ f(x)dx.
26 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Continuous Random Variable CDF II
I Thus
F (x) =
∫ x
−∞f(x)dx) =
∫ x
0
1
λexp
(−xλ
)dx
= − exp(−xλ
)∣∣∣x0
= − exp(−xλ
)+ 1
= 1− exp(−xλ
)I We can use the CDF is this case to find the probability
that the time between events is less than some value(or greater than).
I This can inform us about rare event times.
27 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Continuous Random Variable CDF III
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
Various Exponential CDFs
x
Cum
ulat
ive
Dis
trib
utio
n F
unct
ions
λ = 0.5λ = 1λ = 3λ = 5
28 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Expectations of Random Variables
I The distribution of a random variable X contains all ofthe probabilistic information about X.
I Often we summarize the random variable by certainmeasures, namely expectations, variances, means,modes, etc.
I We introduce the first such summary, the expectationof the random variable
29 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Defining the Expected Value
I This value is sometimes referred to as the mean of therandom variable.
Definition (Expected Value)
The expected value of a random X, denoted E(X) isdefined
E(X) =
∫ ∞−∞
xf(x)dx
For a discrete random variable this would become
E(X) =∑all x
xf(x)
30 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Discrete Random Variable I
I We continue investigating our created randomvariables.
I As an intuition the expected value can be thought of asa “balancing point” of the distribution.
2 4 6 8 10 12
0.00
0.05
0.10
0.15
0.20
Probability Mass Function − Rolling Two Dice
Experimental Outcome
Pro
babi
lity
Mas
s F
unct
ion
0.028
0.056
0.083
0.111
0.139
0.167
0.139
0.111
0.083
0.056
0.028
31 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Discrete Random Variable II
I Performing our calculation
E(X) =∑x
xf(x) = 2× 0.028 + 3× 0.056 + . . .
= 7
I One thing worth noting is that It is NOT where themost mass is, as we will see in the continuous example.
32 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example Expected Value of a ContinuousRandom Variable I
I For an exponential random variable, this can bethought of as the expected time between two eventsoccurring.
I Calculating...
E(X) =
∫ ∞−∞
xf(x)dx =
∫ ∞0
x
λexp
(−xλ
)
33 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example Expected Value of a ContinuousRandom Variable II
I We can do this by integration by parts, where‘∫udv = uv −
∫vdu′ with the following
u =x
λdv = exp
(−xλ
)du =
1
λv = −λ exp
(−xλ
)
34 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example Expected Value of a ContinuousRandom Variable III
I Thus,
E(X) =
∫ ∞0
x
λexp
(−xλ
)= −x exp
(−xλ
)∣∣∣∞0
+
∫ ∞0
exp(−xλ
)dx
= −λ exp(−xλ
)∣∣∣∞0
= λ
I As we saw from our earlier example, the maximum ofthe distribution occurs at 0, which does not correspondto the expected value of this distribution.
35 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
An Important Property of Expectations I
I We now present a useful relationship for expectations
I Claim: Y = aX + b, then E(Y ) = aE(X) + b.
36 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
An Important Property of Expectations II
I Claim: Y = aX + b, then E(Y ) = aE(X) + b.
I Proof:
E(Y ) = E(aX + b) =
∫(ax+ b)f(x)dx
= a
∫xf(x)dx+ b
∫f(x)dx
= aE(X) + b
37 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Generalizing Expectations of Random Variables
I We are not restricted to only finding the expected valuefor the random variable X, we can also find theexpected value of functions of the random variable X,that is some g(X).
Definition (Law of The Unconcious Statistician)
The expected value of a function of the random value X,g(X), denoted E(g(X)) is defined
E(g(X)) =
∫ ∞−∞
g(x)f(x)dx
For a discrete random variable this would become
E(g(X)) =∑all x
g(x)f(x)
38 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Variance of Random Variables I
I One function that is ultimately of interest is thevariance of a random variable
I The variance allows us to quantify the variability or thespread of a distribution.
Definition (Variance)
The variance of a random variable is defined as follows
V ar(X) = E[(X − E(X))2]
Often the expected value is defined as E(X) = µ andtherefore this is often written in textbooks as
V ar(X) = E[(X − µ)2]
39 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Variance of Random Variables II
The equation for variance can be simplified to the followinguseful relationship.
I Claim : V ar(X) = E(X2)− (E(X))2
40 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Variance of Random Variables III
I Proof:
V ar(X) = E[(X − µ)2]
= E[X2 − 2µX + µ2]
=
∫(x2 − 2µx+ µ2)f(x)dx
=
∫x2f(x)dx−
∫2µxf(x)dx+
∫µ2f(x)dx)
= E(X2)− 2µE(X) + µ2
= E(X2)− µ2
= E(X2)− (E(X))2
41 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
A Simple Property of Variance I
I Claim: V ar(aX + b) = a2V ar(X)
42 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
A Simple Property of Variance II
I Claim: V ar(aX + b) = a2V ar(X)
I Proof: We know E(aX + b) = aE(X) + b, thus
V ar(aX + b) = E((aX + b− E(aX + b))2)
= E((aX + b− aE(X) + b)2)
= E(a2(X − E(X))2
= a2E((X − E(X))2) = a2V ar(X)
43 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Moments of Random Variables
I Related to both the mean and variance of a randomvariable are it’s moments.
I Moments will become useful for simple estimationstrategies
Definition (Moment of a Random Variable)
For each random variable X and every positive integer k,E(Xk) is called the kth moment of X.
I We can also talk about Central Moments
Definition (Central Moment of a Random Variable)
Suppose that X is a RV such that E(X) = µ, then forevery positive integer k, E((X − µ)k) is called the kth
central moment of X.
44 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Moment Generating Functions
I Finally, one of the most important properties of arandom variable is its moment generating function.
I If two random variables have moment generatingfunctions that exist and are equal, then they have thesame distribution and similarly the converse is true.
I MGFs will be useful for showing convergence of randomvariables, they’re used tor finding the distribution of asum of independent and identically distributed randomvariables, and they play important role in the CentralLimit Theorem.
45 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Moment Generating Functions
Definition (Moment Generating Functions)
The moment-generating function of a random variable X isdefined for every real number t by MX(t) = E(etX).
I MGF’s can be used to ‘generate’ the moments of therandom variable in the following way:
EXn = MX(n)(0) ≡ dn
dtnMX(t)
∣∣∣∣t=0
46 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
MGF for the Exponential Distribution I
I Want to find E(etX)
E(etX) =
∫ ∞0
exp(tx)1
λexp
(−xλ
)dx
47 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
MGF for the Exponential Distribution II
E(etX) =
∫ ∞0
exp(tx)1
λexp
(−xλ
)dx
=
∫ ∞0
1
λexp
(tx− x
λ
)dx
=
∫ ∞0
1
λexp
(−x1− λt
λ
)dx
= − 1
λ
λ
1− λtexp
(−x1− λt
λ
)∣∣∣∣∞0
= 0 +1
1− λt
=1
1− λt
48 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Extending to Multiple Random Variables I
I Clearly, we will want to extend these concepts tomultiple random variables.
I For example, we may like to consider the jointdistribution for two random variables X and Y .
I Therefore we require a function f such that such thatfor every subset A of the sample space, the probabilitythat X and Y , the pair (x, y), takes a value in A, isthe integral over the set A.
P (X,Y ∈ A) =
∫∫Af(x, y)dxdy
49 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Extending to Multiple Random Variables II
I When we are considering two random variables, this isconsidered a bivariate distribution.
I When we consider more than two random variables thisis considered a multivariate distribution.
P (X1, X2, . . . Xn ∈ A)
=
∫. . .
∫∫Af(x1, x2, . . . xn)dx1dx2 . . . dxn
I Multivariate random variables have correspondingdistribution functions, expectations, moments, etc.
50 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Marginal Distributions
I We now consider properties of joint distributionfunctions, which we will illustrate mainly using bivariatedistributions.
I First we consider finding the marginal distribution ofrandom variables.
I Consider two random variables X and Y and their jointdistribution f(x, y).
I The marginal distribution of X and Y are given asfollows
fX(x) =
∫y∈R
f(x, y)dy fY (y) =
∫x∈R
f(x, y)dx
51 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Joint Distribution I
I Consider the density
f(x, y) = x2(1 + y) +3
2y2 for 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1
I We first ask is this a proper density?I Need f(x, y) ≥ 0 for any (x, y) ∈ Ω = [0, 1]× [0, 1]I Need
∫∫f(x, y)dxdy = 1
52 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Joint Distribution II
I Taking derivatives(d
dxf(x, y),
d
dyf(x, y)
)=(2x(1 + y), x2 + 3y
)We see that the extreme values can be found at (0, 0)which occurs at a corner.
53 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Joint Distribution III
I Additionally, plotting we see that the distribution isstrictly increasing
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.5
1
1.5 2 2.5 3
54 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Joint Distribution IV
I Now interested in∫∫
f(x, y)dxdy = 1
∫ 1
0
∫ 1
0x2(1 + y) +
3
2y2dydx =
∫ 1
0x2(y +
y2
2
)+
3
2
y3
3
∣∣∣∣10
dx
=
∫ 1
0
3
2x2 +
1
2dx
=3
2
x3
3+
1
2x
∣∣∣∣10
=1
2+
1
2= 1
55 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Example - Joint Distribution V
I Now let us find the marginal distributions
f(y) =
∫ 1
0x2(1 + y) +
3
2y2dx
=x3
3(1 + y) +
3
2y2x
∣∣∣∣10
=1
3(1 + y) +
3
2y2
f(x) =
∫ 1
0x2(1 + y) +
3
2y2dy
= x2(y +
y2
2
)+y3
2
∣∣∣∣10
=3
2x2 +
1
2
56 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Conditioning on Random Variables
I Once we have a way to calculate the marginaldistribution of a random variable, we can begin to thinkof conditioning on random variables.
I This is similar to conditioning in probability that welearned about.
Definition (Conditional Density Function)
Consider two random variables X and Y , we define theconditional density of X given Y = y denoted X|Y = y tobe
fX|Y=y(x|y) =f(x, y)
fY (y)
I Conditional distributions will become increasinglyimportant in Bayesian statistics
57 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Independence between Random Variables
I Consider random variables X1, X2, . . . , Xn, we say therandom variables are independent if
f(x1, x2, . . . xn) = fX1(x1)fX2(x2) . . . fXn(xn)
I That is random variables are independent if we canfactorize the joint distribution into a product of theindividual marginal distributions.
I If each fXi(·) is the same, we say the they areidentically distributed.
I Independence and identical distributions will come intouse for describing random samples
58 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Expectations of Joint Distributions
I Similar to the univariate case, we can discussexpectations in terms of the joint distribution.
I For a bivariate distribution we can obtain the followingrelationships
E(X) =
∫ ∫xf(x, y)dxdy =
∫xf(x)dx
E(Y ) =
∫ ∫yf(x, y)dxdy =
∫yf(y)dy
E(XY ) =
∫ ∫xyf(x, y)dxdy
I Additionally, if X and Y are independent we can seethat E(XY ) = E(X)E(Y )
59 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Covariance & Correlation between RandomVariables
I Summaries of the random variables allow us tounderstand how they variables behave (expectations),but additionally we would like to know how theirbehavior is related
I This is especially true if the random variables are notindependent.
I Both the covariance and correlation between randomvariables attempts to measure the dependence betweenrandom variables
60 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Defining Covariance between Random Variables I
I Covariance allows us to begin to talk about thetendency of two variables to vary together rather thanindependently.
I If greater values of one random variable correspond togreater values of the other random variable (similar forsmaller values), then the covariance will be positive.
I If greater values of one random variable correspond tolesser values of another random variable, then thecovariance will be negative.
I The magnitude of covariance is often uninterpretable
61 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Defining Covariance between Random VariablesII
Definition
Covariance Consider the random variables X and Y and letE(X) = µX and E(Y ) = µY we define the covariancebetween X and Y to be
Cov(X,Y ) = E((X − µX)(Y − µY ))
I Similar to the univariate case, it can be shown that theabove reduces to
Cov(X,Y ) = E(XY )− E(X)E(Y )
which is much easier to work with in application.
62 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Correlation between Random Variables I
I As stated in the previous slides, the magnitude ofcovariance is not helpful in understanding how torandom variables values are related.
I This is due to the fact that covariance is sensitive tothe relative magnitudes of X and Y .
I To remedy this, the notion of correlation is not drivenby arbitrary changes in the scales of the randomvariables.
63 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
Correlation between Random Variables II
Definition (Correlation)
Consider the random variables X and Y , with finite positivevariances V ar(X) and V ar(Y ), then the correlationbetween X and Y is given as follows
ρ(X,Y ) =Cov(X,Y )
V ar(X)V ar(Y )
I It can be shown that the correlation is restrictedbetween −1 and 1 thus
−1 ≤ ρ(X,Y ) ≤ 1
64 / 65
03 - RandomVariables
Random Variables
Probability andRandom Variables
Discrete RVs
Continuous RVs
CDFs
Expectations
Moments
MGFs
Multiple RandomVariables
Independence
Covariance
References
References
References
George Casella and Roger L Berger. Statistical inference.Second edition, 2002.
EJ Dudewicz and SN Mishra. Modern mathematical statis-tics. john wilsey & sons. Inc., West Sussex, 1988.
Mark J Schervish. Probability and Statistics. 2014.
65 / 65
Recommended