summary of stt

8/7/2019 summary of stt

http://slidepdf.com/reader/full/summary-of-stt 1/26



Chapter 1: Sets and Events

∫ 2.1 : Basic Set Theory

SET THEORY

The sample space, Ω , is the set of all possible outcomes of an experiment.

e.g. if you roll a die, the possible outcomes are Ω =1,2,3,4,5,6.

A set is finite/infinitely countable if it has finitely many points.

A set is countable/denumerable if there exists a bijection (i.e. 1-1 mapping) to the naturalnumbers, N=1,2,3,…

e.g. odd numbers, integers, rational numbers (Q = m/n where m and are integers)The opposite of countable is uncountable.

e.g. ℜ= real numbers, any interval (a,b)

If you take any two points in a set and there are infinitely many points between them, then the setis considered to be dense.

e.g. Q,Note: A set can be dense and be countable OR uncountable.

The power set of Ω is the class of all subsets of Ω , denoted 2Ω

(or P[Ω ]). (i.e. if Ω has “n”

elements, P[Ω ] has 2n elements.)

Note: if Ω is infinite and countable, then P[Ω ] is uncountable.

SET OPERATIONS

• Complement: Α∉Ω∈=Α ω ω :

C

• Union: bothor or Β∈Α∈Ω∈=ΒΑ ω ω ω :

• Intersection: Β∈Α∈Ω∈=ΒΑ ω ω ω and :

SET LAWS

• Associativity: ( ) ( )C C ΒΑ=ΒΑ

• Distributivity: ( )

T t

t

T t

t

∈∈

ΑΒ=

ΑΒ

• De Morgan’s: ( )T t

c

t

c

T t

t

∈∈

Α=

Α

* The opposite is true for all of the above.

∫ 3.1 : Limits of Sets



•

∞

=

Α=Α≥ nk

k k nk

inf

• ∞=

Α=Α≥ nk

k k nk

sup

•

∞

=

∞

=

∞

= Α=Α≥=Α 11

inf

inf lim n nk k

nk nn nk = nΑ∈Ω∈ ω ω : for all but finitely many n

o An occurs eventually

•

∞

=

∞

=

∞

=

Α=Α≥

=Α11

supsuplim

n nk

k k

n

nnnk

= nΑ∈Ω∈ ω ω : for infinitely many n

o An occurs infinitely often

• nnnn Α⊆Α supliminf lim

• ( ) c

nn

c

nn Α=Α supliminf lim (and vice versa)

If liminf = limsup = An, then the limit exists and is An.

∫ 5.1 : Set Operations and Closure

A is called a field or algebra if:

1. ∈Ω A

2. ∈Α A ∈Α⇒ cA

3. ∈ΒΑ , A ∈ΒΑ⇒ A

A field is closed under complements, finite unions, and intersections.

β is called a σ -field or σ -algebra if:1. β ∈Ω

2. β β ∈Β⇒∈Β c

3. β ∈Β⇒ΒΒ∞

=

1

21 ,...,i

i

A σ -field is closed under complements, countable unions, and intersections.

Note: Ω,φ is the smallest possible σ -field and Ω2 is the largest possible σ -field.

The sets in a σ -field are called measurable and the space ( )β ,Ω is called a measurable space.

∫ 6.1 : The σ -field Generated by a Given Class C

Corollary: The intersection of σ -fields is a σ -field.





Chapter 2: Probability Spaces

2.1: Basic Definitions and Properties

Fatou’s Lemma (probabilities): ( ) ( )

Α∞→Ρ ≤ΑΡ ∞→≤ΑΡ ∞→≤

Α∞→Ρ nnnn nnnn

suplimsupliminf liminf lim

A function [ ]1,0: →ℜ F satisfying

(1) F is right continuous(2) F is monotone non-decreasing(3) F has limits at ∞±

is called a probability distribution function.

2.2: More on Closure

P is a π -system if it is closed under finite intersections. That is,., P P ∈ΒΑ⇒∈ΒΑ

A class of subsets L of Ω that satisfies

(1) ∈Ω L

(2) ∈Α L ∈Α⇒ c L

(3) ∈ΑΦ=ΑΑ≠nmn

mn ,, L n n ∈Α⇒ L

is called a λ -system .

Dynkin’s Theorem: (a) If P is a π -system and L is a λ -system such that P⊂ L, then σ (P)⊂ L.

(b) If P is a π -system, σ (P)=L(P). That is, the minimal σ -field over P equals the minimal

λ -system over P.

Proposition 2.2.4: If a class C is both a λ -system and a π -system, then it is a σ -field.

A class S of subsets Ω is a semialgebra if the following hold:(i) S ∈ΩΦ,

(ii) S is a p-system

(iii)If ,S ∈Α

then there exist some finite n and disjoint sets,,,

1 n

C C

with each∑ ==Α∋∈

n

i i

c

i C S C 1

.



Chapter 3: Convergence Concepts

3.1: Inverse Maps

Let Ω and 'Ω be two spaces and let ': ΩΩ Χ .

For a set '' Ω⊆Β the set ( ''1 )(: Β∈ ΧΩ∈=Β Χ− ω ω is called the inverse map of 'Β .

( )'1 Β Χ− is also called the σ -field generated by Χ, denoted ( ) Χσ .

Proposition 3.1.1: If 'Β is a σ -field on 'Ω , then ( )'1 Β Χ−is a σ -field on Ω.

3.2: Measurable Maps, Random Elements, Induced Probability Measures

A pair ( ) field set −σ , (e.g. ( )β ,Ω ) is called a measurable space.

Χ is a random variable if ( ) ( )( )ℜΩ Χ β β β ,,: .

Χ is measurable if )'1 Β Χ−is a set in β

'' β ∈Β∀ . (i.e. ( β β ⊆ Χ− '1)

Measurable sets are “events”.Constant functions are always measurable.

Proposition 3.2.2: Let ( ) ( )2211 ,,: ΒΩΒΩ Χ and ( ) ( )3322 ,,: ΒΩΒΩ Υ be two measurable

maps. Then ( ) ( ) ( )3311 ,,:)( ΒΩΒΩ Χ Υ= Χ Υ=Ζ ω is also measurable.

The Borel σ -field in k ℜ is the class of open (or half-open or closed) rectangles.

Χ is a random vector ,, 21 Χ Χ⇔ are random variables.

Corollary 3.2.2: If Χ is a random vector and ℜℜ k g : is measurable, then ( ) Χ g is a

random variable.

The distribution of X (a.k.a. probability measure induced by X) is denoted 1− Χ ΧΡ =Ρ .




4.1: Basic Definitions

Independence events: ( )

∏∈∈

ΑΡ =

ΑΡ

I i

i

I i

i

for all I in 1,2,…,n.Independent classes: Classes iC are independent if for any choice ii C ∈Α , i=1,…,n, the events

nΑΑ ,,1 are independent.

Theorem 4.1.1: If nC C ,,1 are independent π -systems, then ( ) ( )nC C σ σ ,,1 are

independent.

4.2: Independent Random Variables

Independent random variables: Random variables are independent if their σ -fields are

independent.

Corollary 4.2.2: The discrete random variables k Χ Χ ,,1 with countable range R are

independent iff [ ] [ ]∏=

= ΧΡ === ΧΡ k

i

iiii xk i x1

,,1, for all ∈i x R, i=1,…,k.

4.5: Independence, Zero-One Laws, Borel-Cantelli Lemma

Borel-Cantelli Lemma I: Let An be any sequence of events such that ( ) ∞<ΑΡ ∑n

n (i.e.

converges), then P(An i.o.) = 0.

Borel-Cantelli Lemma II: If A1,…, An are independent events such that ( ) ∞=ΑΡ ∑n

n (i.e.

diverges), then P(An i.o.) = 1.

A property that holds with probability = 1 is said to hold almost surely (a.s.).

NOTE:

• ∑n

1is divergent

• ∑2

1

n is convergent

Let n Χ be a sequence of random variables and define ( ),, 21 ++ Χ Χ= nnn F σ , n=1,2,…

The tail σ -field, τ , is n

n F =τ . Events in τ are called tail events. A tail event is

NOT affected by changing the values of finitely many Xn.

A σ -field all of whose events have probabilities equal to 0 or 1 are called almost trivial (a.t.).



Kolmogorov’s 0-1 Law: If n Χ are independent random variables with tail σ -field τ , then( ) 10 or =ΛΡ ⇒∈Λ τ . (This, in turn, implies that the tail σ -field is a.t.).

• Note: if we assume iid, then more events have the “0-1” property.

Lemma 4.5.1: If g is an a.t. σ -field and X is measurable w.r.t g, this implies that X is constanta.s.

An event is called symmetric if its occurrence is not affected by any permutation of finitely manyXk .

Notes:

• Tail events are not affected by the VALUES and symmetric events are not affected bythe ORDER!

• All tail events are symmetric (converse is false)

Hewitt-Savage 0-1 Law: If n Χ are iid, then all symmetric events have probability 0 or 1.



Chapter 5: Integration and Expectation

5.1: Preparation for Integration

A simple function is a random variable of the form ∑=

ΑΙ= Χn

i

i ia

1

where

nia ii ,,2,1,, =∈Αℜ∈ β and nΑΑ ,,1 form a partition of Ω.

Properties of Simple Functions:

1. If X is simple, then ∑=

ΑΙ⋅= Χn

i

i ia

1

α α , where α is constant.

2. If X,Y are simple, then .)(,

∑ Β∩ΑΙ+= Υ+ Χ ji

ji jiba

3. If X,Y are simple, then ∑ Β∩ΑΙ= ΧΥ ji

ji jiba

,

.)(

4. If X,Y are simple, then ∑ Β∩ΑΙ= Υ Χ ji

ji jibaMinMin,

),(),( or

∑ Β∩ΑΙ= Υ Χ ji

ji jibaMaxMax

,

),(),( .

Theorem 5.1.1 - Measurability Theorem: Let X be a nonnegative random variable. Then there

exists a sequence of simple functions Xn Χ↑ Χ≤∋ n0 .

5.2: Expectation and Integration

Expectation of X with respect to P: Let X be a simple function, then

( ) ( )∑∑ ∫ ∑ ∫ ∫ ==Α

=Α ΑΡ =Ρ =Ρ Ι=Ρ Χ= ΧΕ

n

i

ii

n

i

i

n

i

i ad ad ad i

i

111.

Note: ( ) ( )ΑΡ =ΙΕ Α .

Steps to show expectation:

1. Indicators: Let ΑΙ= Χ ( ) ( ) ( )ΑΡ =ΙΕ = ΧΕ Α

2. Simple Functions: Let ∑=

ΑΙ= Χn

ii i

a1

3. Nonnegative Random Variables: Suppose 0≥ Χ , Xn a sequence of simple functions and Χ↑ Χ≤ n0 .

4. General Functions:−+

Χ− Χ= Χ

If Χ↑ Χn and Χ↑ Υm , then E(X) is well-defined if [ ] [ ]mnmn

ΥΕ ∞→

= ΧΕ ∞→

limlim.

X is integrable if E[|X|]< ∞ , where |X| = −+ Χ+ Χ . Therefore X is integrable ⇒

[ ] [ ] ∞< ΧΕ ∞< ΧΕ −+ , (then E[X] will be finite).



Monotone Convergence Theorem (for expectation): [ ] [ ] ΧΕ ↑ ΧΕ ⇒ Χ↑ Χ≤ nn0 or

( )nnnn

ΧΕ ↑∞→

=

Χ↑

∞→Ε

limlim.

Inequalities:

(a) Modulus: ( ) ( ) ΧΕ ≤ ΧΕ .

(b) Markov: Suppose 1 L∈ Χ . For any λ >0, [ ] ( )λ

λ ΧΕ

≤≥ ΧΡ .

(c) Chebychev: Assume ( ) ∞< Χ∞< ΧΕ Var , . ( )[ ] ( )2λ

λ Χ

≤≥ ΧΕ − ΧΡ Var

.

(d) Triangle: 2121 Χ+ Χ≤ Χ+ Χ

(e) Weak Law of Large Numbers (WLLN): The sample average of an iid sequence

approximates the mean. Let 1, ≥ Χ nn be iid with finite mean and variance and

suppose [ ] µ = ΧΕ n and Var(X)=σ 2 is finite. Then for any

0lim

,0 1 =

>− Χ

Ρ ∞→

> ∑= ε µ ε nn

n

ii

.

5.3: Limits and Integrals

Monotone Convergence Theorem (for series): If 0≥ Χ j are nonnegative random variables

for 1≥n , then [ ]∑∑∞

=

∞

=

ΧΕ =

ΧΕ

11 j j

j j . (i.e. the expectation and infinite sum can be

interchanged)

Fatou’s Lemma (expectations):

( ) ( )

Χ∞→

Ε ≤ ΧΕ ∞→

≤ ΧΕ ∞→

≤

Χ∞→

Ε nnnnnnnn


Dominated Convergence Theorem (for expectations): If Χ→ Χn and there exists a

dominating random variable 1 L∈Ζ ,Ζ ≤ Χ∋ n then ( ) ( ) 0→ Χ− ΧΕ ΧΕ → ΧΕ nn and .

# NOT IN BOOK – General Measure and Integration Theory #

A function ℜ→β µ : is called a measure if



(i) 0≥ µ

(ii) ( ) 0=φ µ

(iii) β ∈ΑΑ ,, 21 disjoint ( )∑ Α=

Α⇒

i

i

i

i µ µ PPN: ( ) ( )Α→Α⇒Α↑Α µ µ nn

If ∞=

=Ω1n

nC where C C n = and ( ) ,nC n ∀∞< µ then µ is called σ -finite on the class C .

THM: If µ 1= µ 2 on a π− system P and µ 1, µ 2 are σ -finite on P, then µ 1= µ 2 on σ (P).

THM: A σ -finite measure on a semialgebra S, has a unique extension to σ (P).

5.5: Densities

Let X be a random variable. If there exists a function ( ) ( )∫ Β Χ =ΒΡ ∋ dx x f f for all Borel sets

B, then f is called the density of X.

THM: If f exists, then ( )( ) ∫ =Ε dx x f x g x g )()( .

Note: dF(x) = f(x)dx

LEMMA: If ..ea g f gd fd −=→∈Α∀= ∫ ∫ ΑΑµ β µ µ

5.6: The Riemann vs. Lebesgue Integral

Lebesgue Measure (λ ):

• λ (singleton) = 0

• λ (interval) = length of interval

Steps to show integration:

1. Indicators: Let ΑΙ= f ( )Α==Ι= ∫ ∫ ∫ ΑΩ ΑΩµ µ µ µ d d fd

2. Simple Functions: Let ∑=

ΑΙ=n

i

i ia f

1

3. Nonnegative Random Variables: Suppose 0≥ f , n f simple and f f n ↑≤0 . Define

∫ ∫ = µ µ d f fd nnlim .



4. General Functions: −+ −= f f f and define ∫ ∫ ∫ −+ −= µ µ µ d f d f fd

(4) DNE if “ ∞−∞ ”.

Types of Integration:

• Riemann – approximate area under a curve via rectangles

• Lebesgue – approximates via a measure. This is the limit of simple functions.

Note: Riemann divides the domain and Lebesgue divides the range of a function.

THM: If Riemann exists, then Lebesgue integral exists. The converse is not necessarily true.

A property which holds everywhere except on a set A with µ (A)=0 is said to hold µ -almost

everywhere (µ -a.e.).THM: g is Riemann Integrable ⇔ g is continuous a.e.

Fatou’s Lemma (integrals):

( ) ( ) ∫ ∫ ∫ ∫

∞→

≤∞→

≤∞→

≤

∞→

µ µ µ µ d f n

d f n

d f n

d f n

nnnn


Dominated Convergence Theorem (for integrals): Let ,,,21

f f f be a sequence of

functions f f n →∋ . If there exists a function n g f g ∀≤∋ and ∫ ∞< µ gd , then

∫ ∫ → µ µ fd d f n .

Monotone Convergence Theorem (for integrals): Let ,,,21

f f f be a sequence of

nonnegative functions ∞→↑∋ n f f n , . Then ∫ ∫ → µ µ fd d f n .

Identity: ( ) ( )nnnn f f −−= inf limsuplim

5.7: Product Spaces

( ) 22112121 ,:, Ω∈Ω∈=Ω×Ω ω ω ω ω is called the product space.

If 1β ∈Α and 2β ∈Β then the set ( ) Β∈Α∈=Β×Α 2121 ,:, ω ω ω ω is called a (measurable)

rectangle.

The σ -field generated by the rectangles is called the product σ -field, denoted 21 β β ⊗ .

For 21 β β ⊗∈C we define the sections as ( ) C y x yC x ∈= ,: and ( ) C y x xC y ∈= ,: .

• Sections (of functions and sets) are measurable.

Define measures 21 ,π π on ( )2121 , β β ⊗Ω×Ω :



• ( ) ( ) ( )∫ Ω= 11 xd C C x µ υ π

• ( ) ( ) ( )∫ Ω= 22 yd C C y υ µ π

• Both measures equal ( ) ( )ΒΑυ µ , when Β×Α=C .

• If µ, ν are probability measures, then 21 ,π π are probability measures.

THM: If µ, ν are σ -finite, then there exists a unique measure on 21 β β ⊗ such that( ) ( ) ( )Β⋅Α=Β×Α υ µ π .

NOTE: υ µ π ×=

5.9: Fubini’s Theorem

Fubini’s Theorem: If ∫ ⋅ π d g exists, then

( ) ∫ ∫ ∫ ∫ ∫ ∫ Ω ΩΩ ΩΩ×Ω

⋅=

⋅=×⋅=⋅

2 11 221

υ µ µ υ υ µ π d d g d d g d g d g .

Special Cases when Theorem Holds:1. 0≥ g (Tonelli’s Theorem)

2. ∞<⋅∫ π d g (i.e. g integrable) (often called Fubini’s Theorem)




Suppose (Ω ,F,P) is a probability space, then ),(),(: B F X ℜ→Ω is a random variable.

The distribution of X is denoted X P X P =−1

If µ is a measure on (Ω ,F), then1− f µ is the induced measure on (Λ ,G).

Almost Sure

• ..1)( sa X X X X P nn

→⇒=→

• ..0)||(1

sa X X X X P nn

n

→⇒>∀∞<>−∑∞

=

ε ε

• PPN: Y X Y X Y Y sa X X nnnn+→+⇒→→ &.. (this holds for L p and therefore in

probability and in distribution.

In Probability

• X X X X P P

n

n

n → ⇒>∀ → >− ∞→ 00)|(| ε ε

• THM 6.3.1(b): Convergence in probability can be characterized by convergence a.s of subsequences:

⇔ → X X P

n Eachk n X has a further subsequence ( ) ( )

.. sa X X X ik ik nn →∋

• COR 6.3.1( ) ( )

( ) ( ) X g X g continuous g X X

sa X g X g continuous g sa X X

P

n

P

n

nn

→ ⇒ →

→⇒→

&

..&..

• COR 6.3.2: Version of DCT [ ] [ ] [ ] X E X E Y E Y X sa X X

nnn

→⇒∞<≤→ &||.,.

• PPN: ( ) 1& ==⇒ → → Y X P Y X X X P

n

P

n , therefore cannot have different limits in

probability. Same holds for a.s and L p.

In Lp

• Let p>0. X X X X E P L

n

P

n → ⇒→− 0||

• PPN: ( ) ( ) X g X g bounded continuous g X X P P L

n

L

n → ⇒ → &,

• PPN: If X X P L

n → , for some p and r<p, then X X r L

n → .

In Distribution

• Let X (X1, X2, …) with distribution function F (F1, F2, …) where F(x)=P(X ≤x),

F1(x)=P(X1 ≤x), etc… ( ) ( ) X X continuous F x x F x F d

nn→ ⇒∀→ , .

Relationships between types of convergence:

• X X sa X X P

nn → ⇒→ ..



• X X X X P

n

L

n P → ⇒ →

• X X X X d

n

P

n → ⇒ →

Inequalities

• Chebyshev’s: ( ) [ ] P

P Y E Y P

ε ε

|||| ≤≥ for any p>0 & ε >0.

• Holder’s: If p>1, q>1, & 111

=+q p

, then [ ] [ ] [ ] q p q pY E X E X Y E

11

|||||| ⋅≤ .

Special case of Holder’s is when p=q=2 (i.e. Cauchy-Schwartz ).

• Minkowski’s: For p ≥>1, [ ] [ ] [ ] p p p p p p Y E X E Y X E 111

|||||| +≤+ .

For p=1, this is the Triangle Inequality.



Chapter 7: Law of Large Numbers & Sums of Independent Variables

Weak Law of Large Numbers (WLLN) - THM 7.2.1: (where bn=n & ∑=

=n

j

jn X S 1

)

• Let X1, X2, … independent & [ ]∑= ≤=

n

j

j jn n X X E a1

||; , then if

(1) ( )∑=

∞→ → >n

j

n

j n X P 1

0||

(2) [ ] 0||;1

1

2

2 → ≤ ∞→

=∑ nn

k

j j n X X E n

⇒ 0 → − P nn

n

aS .

• Let X1, X2, … i.i.d. & n X X E na j jn ≤⋅= ||; , then if

(1) ( ) 0|| → >⋅ ∞→n j n X P n

(2) [ ] 0||;1 2 → ≤ →∞n

j j n X X E n

⇒ 0 → − P nn

n

aS .

LEMMA: 0 → − P nn

n

aS & a

n

S a

n

a P nn → ⇒→

Kolmogorov’s Convergence Criterion – THM 7.3.3:

• Let X1, X2, … independent . [ ] [ ]( ) ..,11

saconverges X E X then X Var If j

j j

j

j ∑∑∞

=

∞

=−∞<

• ∴The convergence of ∑∞

=1 j j X is determined by [ ]∑

∞

=1 j

j X E .

Kronecker’s Lemma – 7.4.1:

• Let Xk and bn be sequences of real numbers ∞↑∋ nb . If ∑∞

=1 j j

j

b

X converges, then

01

1 → ∞→

=∑n

n

j

j

n X b .

• COR 7.4.1:

Suppose [ ] n X E n ∀∞<2.

∞↑nb &[ ]

..01

2sa

b

S E S

b

X Var

n

nn

j j

j →−

⇒∞<∑∞

=



Kolmogorov’s Strong Law of Large Numbers (SLLN) – THM 7.5.1:

• Let X1, X2, … i.i.d. with finite mean m (i.e. [ ] ∞<|| 1 X E ).

Then µ

→n

S n

a.s.



Chapter 8: Convergence in Distribution

CDF of a random variable is defined as F(x) = P[X ≤ x].

F is a possible cdf iff:

1. ( ) 10 ≤≤ x F 2. F is nondecreasing3. F is right-continuous

If ( ) 0=∞− F and ( ) 1=∞ F we call F proper (or non-defective). It is assumed that all cdf’s are

proper.

Convergence in Distribution: Χ → Χ d

n if ( ) ( ) x x F x F n ∀→ where F is continuous.

This is also called weak convergence and can be denoted ( ) ( ) x F x F w

n → .

Lemma 8.1.1: If two cdf’s agree on a dense set, then they agree everywhere.

A dense set has no gaps.

LEMMA: A cdf has at most countably many discontinuities.

PPN: If F F w

n → , then F is unique.

Skorohod’s Theorem: Suppose Χ → Χ d

n . Then there exists a random variable ,, #

2

#

1

# Χ Χ∋ Χ

on the probability space [ ]( ) ,2,1,,,,1,0 ## = Χ= Χ Χ= Χ∋ n Lebesgue Borel n

d

n

d

and

..## san Χ→ Χ

Continuous Mapping Theorem: If Χ → Χ d

n and g continuous, then ( ) ( ) Χ → Χ g g d

n .

Theorem 8.4.1: ( )[ ] ( )[ ] ΧΕ→ ΧΕ⇔ Χ→ Χ hh nn for all bounded and continuous functions h.



Chapter 9: Characteristic Functions and the Central Limit Theorem

Moment Generating Function: ( ) ΧΕ =Μ t et

Characteristic Function: ( ) ( ) ( )( ) ( )( ) ΧΕ + ΧΕ =Ε =Φ Χ t it et it sincos ; ℜ∈t • Maps from the Real Numbers to the Complex Plane

• Chf always exists, whereas mgf does not.

Properties of CHF:

• ( ) 10 =Φ

• Υ Χ Υ+ Χ ΦΦ=Φ⇒ Υ Χ t independen,

• ( ) [ ] ΧΕ =Φ i0'

o In Generality: ( ) ( ) [ ]nnn i ΧΕ =Φ 0

Theorem 9.5.1: Φ uniquely determines the distribution of X.

Theorem 9.5.2: ( ) ( ) t t t n

d

n ∀Φ→Φ⇒ Χ → Χ Χ Χ .

The chf of a standard normal distribution is ( ) 2

2t

et −

Χ =Φ

Theorem 9.7.1 (Central Limit Theorem, iid case):

Let ,, 21 Χ Χ be iid with mean µ and ∞<2σ .

Let ∑=

Χ=n

k

k nS 1

and let ( )1,0~ ΝΝ .

Then Ν → − d n

nnS

σ

µ .

[PROOF IS SHOWN ON NEXT PAGE]

Proof of CLT (iid case )



Suppose µ =0 and 12 =σ .

Let Φ be the chf of Xk and nΦ be the chf of n

S n .

Then ( )nn

k

nit indep

nit

n

S it

nn

t eeet k k n

Φ=

Ε =

∑Ε =

Ε =Φ ∏

=

Χ Χ

1

.

Want to show that et

n

nt 2

2−→

Φ .

By Taylor’s Formula: ( ) ( ) ( )∑∞

=

Φ Χ

= ΧΦ0

0!k

k k

k (assume after k=2, all functions are negligible.

i.e. converge to 0 as n goes to infinity)

Withn

t = Χ , ( ) ( ) ( ) ( ) ( )02

002

21 Φ+Φ+Φ=

Φ

n

t

n

t

nt

• ( )0Φ =1

• ( ) ( ) µ i=Φ 01 = 0 (since we ‘supposed’ µ = 0)

•( ) ( ) 102 −=Φ

n

t

nt

21

2

−=

Φ⇒

( ) et

n

nn

nn

t

nt t 2

2 2

21

−∞→ →

−=

Φ=Φ .

.Ν → ∴ d n

n

S

Delta Method

Let ,, 21 Χ Χ be iid. By CLT withn

S n= Χ , Ν →

− Χ=

− d n nn

nS µ

σ

µ . Hence, for large n,

n

2

,~ σ µ Ν Χ .

This method is used to make inferences about µ .

PPN: If ( ) ,0' ≠ µ g then( ) ( )

( )( ) ( ) ( )[ ]( )2'

',~ µ σ µ

µ σ

µ g g g

g

g g n d ⋅Ν Χ⇒Ν →

⋅

− Χ.

The k Χ are said to satisfy the Lindeberg condition if [ ] 0,0;1

1

2 >∀ → > Χ ΧΕ ∞→

=∑ t tS

S

nn

k

nk k

n

.

Theorem 9.8.1 (Lindeberg-Feller CLT):



Let ( )n

n

k

k

n

k

k n S Var Var s =

Χ== ∑∑

== 11

2σ . Then the Lindeberg condition implies ( )1,0Ν → d

n

n

s

S .

Liapunov condition: For some 0>δ ,

[ ]0

21

2

→

ΧΕ

+=

+∑δ

δ

n

n

k

k

S .

PPN: Lindeberg Liapunov ⇒

Important relationship: ( )nk

n

k

log~1

1

∑=



Chapter 10: Martingales

10.1: Prelude to Conditional Expectation: The Radon-Nikodym Theorem

DFN: If µ (B)=0 ⇒ ν (B)=0, then ν is absolutely continuous with respect to µ (i.e. ν

<< µ ).

Radon-Nikodym Theorem – THM 10.1.2:

• If ν << µ & µ is σ -finite, then ∃ a measurable function( ) Borel Bd f B f

B∈∀=∋ℜ→Ω ∫ µ ν : .

• R-N Derivative: µ

ν

d

d f = .

• Chain Rule: If ν << µ & µ < < ρ , then ν<<ρ & ρ

µ

µ

ν

ρ

ν

d

d

d

d

d

d ⋅= .

10.2: Definition of Conditional Expectation

Conditional Probability:

• Let ( )Ρ ΒΩ ,, be a probability space & let Β∈ g be a sub-σ -field of B.Take .Β∈ A ∃ a unique random variable z ∋

(1) z is measurable with respect to g

(2) ( ) ∫ ∈∀Ρ =∩Ρ G

g Gd z G A .

• In general:( )

( )

( )

( )( )

( )

=

Ρ

∩Ρ

=Ρ

y f

y x f y x f C o n t i n u o u s

B

B A

B A D i s c r e te

,|

|

Conditional Expectation:

• Let z=E[X|g] be the conditional expectation of X with respect to g. Then it must fulfillthe following:

(1) z is measurable with respect to g(2) ∫ ∫ ∈∀Ρ =Ρ

GG g Gd z d X .

• Generally:o If X is g-measurable, then E[X|g]=X.

o If X is independent of g, then E[X|g]=E[X].



o [ ] ( ) ( )( )∫ ∫ ⋅=⋅= dxY f

Y x f xdxY x f xY X E

,,|

10.3: Properties of Conditional Expectation

1. Linearity: If 1, L∈ Υ Χ and ℜ∈β α , , we have

( )( ) ( ) ( )GGG sa

ΥΕ ⋅+ ΧΕ ⋅= Υ+ ΧΕ β α β α ..

.

2. If G∈ Χ ; 1 L∈ Χ , then ( ) Χ= ΧΕ .. sa

G .

3. ( ) ( ) ΧΕ =Ω ΧΕ ,φ

4. Monotonicity: If 0≥ Χ and 1 L∈ Χ , then ( ) ..0 saG ≥ ΧΕ

5. Modulus Inequality: If 1 L∈ Χ , then ( ) ( )G G ΧΕ ≤ ΧΕ .

6. Monotone Convergence Theorem: If 1 L∈ Χ , Χ↑ Χ≤ n0 , then ( ) ( )GGn

ΧΕ ↑ ΧΕ .

7. Monotone Convergence implies the Fatou Lemma:

a. ( )GnGnnn ΧΕ ∞→≤

Χ∞→Ε

inf liminf lim

b. ( )Gn

Gn

nn ΧΕ ∞→

≥

Χ∞→

Ε suplimsuplim

8. Fatou implies Dominated Convergence: If 1 Ln ∈ Χ , 1 Ln ∈Ζ ≤ Χ and ∞ Χ→ Χn , then

( )Gn

Gn

n

sa

n ΧΕ ∞→

=

Χ∞→

Ε limlim ..

9. Product Rule: Let X,Y be random variables satisfying X, YX 1 L∈ . If G∈ Υ , then

( ) ( )GG sa

ΧΕ ⋅ Υ = ΧΥ Ε ..

.

10. Smoothing: If Β⊂⊂ 21 GG , then for 1 L∈ Χ ,

a. ( )( ) ( )112 GGG ΧΕ = ΧΕ Ε

b. ( )( ) ( )121 GGG ΧΕ = ΧΕ Ε

• NOTE: The smallest σ -field always wins.

10.4: Martingales

Martingales:

• DFN: Sn is called a martingale with respect to Bn if (a) Sn is measurable with respect to Bn n(b) E[Sn+1 | Bn] = Sn

• If (b) is “ ≤”, then it is a supermartingale. If “ ≥”, then it is a submartingale.o Submartingales tend to increase

o Supermartingales tend to decrease

• If Sn is a martingale, then E[Sn+k | Bn] = Sn ∀k=1,2,…

• If Sn is a martingale, then E[Sn] = constant



10.5: Martingales

10.6: Connections between Martingales and Submartingales

Any submartingale ( ) 0,, ≥Β Χ nnn can be written in a unique way as the sum of a martingale

( ) 0,, ≥ΒΜ nnn and an increasing process 0, ≥Α nn . i.e. nnn Α+Μ= Χ

10.7: Stopping Times

A mapping ∞Ω ,,2,1,0: υ is a stopping time if [ ] ,2,1,0, =Ν∈∀Β∈= nn nυ .

To understand this concept, consider a sequence of gambles. Then υ is the rule for when to

stop and nΒ is the information accumulated up to time n. You decide whether or not to stop

after the nth gamble based on information available up to and including the nth gamble.

Optional Stopping Theorem: If Sn is a martingale and <<< 321 υ υ υ are well-behaved

stopping times, then the sequence ,,,321 υ υ υ S S S is a martingale with respect to

,,,321 υ υ υ ΒΒΒ .

Consequences:

• =Ε=Ε=Ε321 υ υ υ S S S

• If we let υ υ υ ==21

0 and , then [ ] [ ]υ S S Ε =Ε 0 .

υ is a well-behaved if:

(a) ( ) 1=∞<Ρ υ

(b) [ ] ∞<Ε υ S

(c) [ ] 0; → >Ε ∞→nnS υ υ

10.9: Examples

Gambler’s Ruin

Basic concept: Imagine you initially have K dollars and you place $1 bet. You continue to bet$1 after each win/loss. Playing this “game” called Gambler’s ruin, the ultimate question is, willyou hit $0 or $N first?

Formally written, suppose nΖ are iid Bernoulli random variables (i.e. win/lose $1) satisfying

[ ]2

11 =±=Ζ Ρ n , and let 00 j= Χ (i.e. initial balance), 1,0

1

≥+Ζ = Χ ∑=

n jn

i

in (i.e. balance after

nth bet). This is a simple random walk starting at 0 j . Assume Ν≤≤ 00 j . Will the random

walk hit 0 or N first?



Ν=−Ν

Ν−=

= Χ0

0

1,

1,0

j p y probabilit with

j p y probabilit with

υ

Branching Process

Basic concept: Consider a population of individuals who reproduce independently. Let X be a

random variable with offspring distribution [ ] ,2,1,0, == ΧΡ = k k pk and let m = E[X]. Start

with one individual (easiest to think of a single-celled organism) who has children according tothe offspring distribution. The children then have children of their own, independently, based on

the same distribution. Let nΖ be the number of individuals in the nth generation ( )10 ≡Ζ . The

process nΖ is called a simple branching process or Galton-Watson process.

NOTE: ∑−Ζ

= Χ=Ζ 1

1

n

k

k n , where k Χ are the number of children of the k th individual in generation

(n-1). k Χ are iid.

( ) [ ]( ) nn

n m ΧΕ==Ζ Ε

Hence [ ] growth

stagnation

extinction

m

m

m

n ⇒

>∞→=≡<→

=Ζ Ε `,

1,1

1,0

Extinction: ( ) ( )0lim0 =Ζ Ρ =

=Ζ Ρ =Ε Ρ nn

n

n

Let [ ] ,2,1,0, == ΧΡ = k k pk . The function ( ) ∑∞

=

=Φ0k

k

k p s s , is called the probability

generating function (pgf) of X.

Let q = P(E), then you can solve for q via the equation ( ) s s Φ= .

( )

( )

( )

⇒

=Ε Ρ ⇒>

=Ε Ρ ⇒==Ε Ρ ⇒<

ercritical

critical

l subcritica

m

m

m

sup01

11

11

Useful identity: Let 0≥ Υ with mean µ and [ ] ∞< ΥΕ 2(finite 2nd moment). Then

( )[ ]2

2

0 ΥΕ

≥> ΥΡ µ

.



THM: Let[ ] size populationected

size populationactual

mW

n

n

n

nn

exp=

Ζ =

Ζ Ε Ζ

= . Then nW is a martingale with

respect to the σ -field ( )nn Ζ Ζ =Β ,,0 σ .

• P(W=0) is either 1 or q.

• If variance is finite, then P(W=0)=q.

COR: There exists an integrable random variable W such that .. saW W n →

PPN: In a martingale, if 2nd moments are bounded, then S S L

n → 2 .

10.10: Martingale and Submartingale Convergence

Doob’s Theorem: If nS is a submartingale such that ( ) ∞<Ε +nn S sup , then there exists an

integrable random variable S such that .. saS S n →

PPN: Nonnegative martingales always converge.

Documents

summary of stt