Probability Theory Presentation 10

8/8/2019 Probability Theory Presentation 10

1/51

BST 401 Probability Theory

Xing Qiu Ha Youn Lee

Department of Biostatistics and Computational BiologyUniversity of Rochester

October 7, 2010

Qiu, Lee BST 401
http://find/http://goback/


2/51

Outline

1 Introduction to functional analysis

2 Convergence of Sequence of Measurable Functions

Qiu, Lee BST 401


3/51

Motivation (I)

Functional analysis is in some sense the linear algebra ofmeasurable functions/random variables. Youve already

seen that linear combinations of r.v.s are r.v.s.

The usual linear algebra deals with finite dimensional

vectors. In general, random variables are inherently infinite

dimensional.

For an Euclidean space, all linear transformations can be

expressed as matrix multiplications in a basis system.

There is also a way to define a (infinite) basis system (and

coordinates) for a functional space. So lineartransformations of r.v.s can be expressed in this basis

system explicitly.

It turns out, all linear transformations are integrals in a

basis system.

Qiu, Lee BST 401


4/51

Motivation (I)





dimensional.For an Euclidean space, all linear transformations can be




system explicitly.


basis system.

Qiu, Lee BST 401


5/51

Motivation (I)









system explicitly.


basis system.

Qiu, Lee BST 401


6/51

Motivation (I)









system explicitly.


basis system.

Qiu, Lee BST 401


7/51

Motivation (I)









system explicitly.


basis system.

Qiu, Lee BST 401


8/51

Motivation (II)

The functional norm will act as vector length, andsometimes we can even define an inner product between

two vectors. Consequently two r.v.s may have an angle

between them; they may be orthogonal to each other.

Many important mathematical concepts, such as continuity,

convergence, and completeness, can be derived from thenorm of a functional space.

Unlike n-dim Euclidean vector spaces, norms defined on

an infinite functional space are not equivalent. Depending

on different norms, we have different functional spaces.

Lp() spaces, 1 p are the most importantfunctional spaces for studying probability theory.

Other spaces, such as the Sobolev spaces are useful for

nonparametric regression, functional analysis, SDE, etc.

Qiu, Lee BST 401


9/51

Motivation (II)

The functional norm will act as vector length, and

sometimes we can even define an inner product between











Qiu, Lee BST 401


10/51

Motivation (II)













Qiu, Lee BST 401


11/51

Motivation (II)













Qiu, Lee BST 401


12/51

Motivation (II)













Qiu, Lee BST 401


13/51

Lp-space

(,F, ) is a measurable space.

For p 1, we define Lp(,F, ) (in short, Lp) to be thespace of -measurable functions such that

fp =

|f|

p

d

1p

< .

Special case: random variables with finite mean (L1);

random variables with finite variance (L2).

Another special case: L(), the space of all almostsurely bounded r.v.s:

f = limp

fp = ess sup

|f(x)|.

Qiu, Lee BST 401


14/51

Lp-space

(,F, ) is a measurable space.

For p 1, we define Lp(,F, ) (in short, Lp) to be thespace of -measurable functions such that

f

p =

|f

|

pd1p

N.

Completeness. A functional space X is complete if every

Cauchy sequence converges to a member in X.

Lp spaces are complete.

Implication: if a sequence of r.v.s X1, X2, . . . satisfies

limn,mE|Xn Xm|p = 0, then there must be a r.v. X to

which Xn converges, and X Lp() as well. So say if Xn

have finite variances, X must have finite variance as well.

Qiu, Lee BST 401


24/51

Basic properties (II)

A norm induces a distance: distp(f,g) = f gp. Withdistance we can define Cauchy sequence. f1, f2, . . . is aCauchy sequence (relative to the given distance) if > 0,there exists N N, such that

distp(fn, fm) < , n, m> N.








Qiu, Lee BST 401


25/51











Qiu, Lee BST 401


26/51











Qiu, Lee BST 401


27/51

Dense subset/approximation

For simplicity, assume = R.

Recall Q is dense in R. Dense subsets in Lp:

set of simple functions;set of continuous functions;set of smooth functions (functions with arbitraryderivatives).set of polynomials. (checkout the Bernstein polynomialsfrom Wikipedia)

Qiu, Lee BST 401


28/51





Qiu, Lee BST 401


29/51





Qiu, Lee BST 401


30/51





Qiu, Lee BST 401


31/51





Qiu, Lee BST 401


32/51





Qiu, Lee BST 401


33/51

Basis

A basis (e1,e2, . . . , en) of n-dim linear space (notnecessarily orthogonal):

1 ei are linearly independent;2 every X X can be written as a linear combination of

(e1,e2, . . . ,en). X =n

i=1 xiei.

For a Banach space:1 ei are linearly independent;2 every X X can be written as

X =

i=1

xiei,

this summation is understood as a limit.

Example: Taylor expansion + smooth function

approximation of an Lp([0, 1],B,L) function.

Qiu, Lee BST 401


34/51

Basis

A basis (e1,e2, . . . , en) of n-dim linear space (notnecessarily orthogonal):

1 ei are linearly independent;2 every X X can be written as a linear combination of

(e1,e2, . . . ,en). X =n

i=1 xiei.


X =

i=1

xiei,




Qiu, Lee BST 401


35/51

Basis

A basis (e1,e2, . . . , en) of n-dim linear space (not

necessarily orthogonal):1 ei are linearly independent;2 every X X can be written as a linear combination of

(e1,e2, . . . ,en). X =n

i=1 xiei.


X =

i=

1

xiei,




Qiu, Lee BST 401

B i


36/51

Basis



(e1,e2, . . . ,en). X =n

i=1 xiei.


X =

i=

1

xiei,




Qiu, Lee BST 401

B i


37/51

Basis



(e1,e2, . . . ,en). X =n

i=1 xiei.


X =

i=

1

xiei,




Qiu, Lee BST 401

B i


38/51

Basis



(e1,e2, . . . ,en). X =n

i=1 xiei.


X =

i=

1

xiei,




Qiu, Lee BST 401

B i


39/51

Basis



(e1,e2, . . . ,en). X =n

i=1 xiei.


X =

i=

1

xiei,




Qiu, Lee BST 401

Inner product and Hilbert space


40/51


A complete normed linear space such as Lp

is called aBanach space.

A Hilbert space H is a Banach space with a inner productf, g : H H R which satisfies1

1 Bilinearity: aX+ bY, Z = aX, Z + bY, Z.

2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.

An inner product induces a norm: X :=

X, X. But anorm in general can not be extended to an inner product.

L2 is a Hilbert space and the only Hilbert space among Lp

spaces. Its inner product: X, Y2 = EXY =XYd.

1R should be replaced by C for spaces of complex valued functions.2For complex Hilbert spaces, X, Y = Y, X, where is complex

conjugate.

Qiu, Lee BST 401



41/51


A complete normed linear space such as L

p




2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.






conjugate.

Qiu, Lee BST 401



42/51



p




2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.






conjugate.

Qiu, Lee BST 401



43/51



p




2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.






conjugate.

Qiu, Lee BST 401



44/51



p




2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.






conjugate.

Qiu, Lee BST 401



45/51



p




2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.






conjugate.

Qiu, Lee BST 401



46/51



p




2 X, Y = Y, X.2

3 X, X 0 and X, X = 0 iff X = 0.






conjugate.

Qiu, Lee BST 401

Properties of a Hilbert space


47/51


With an inner product, we can define orthogonality. X isorthogonal to Y if X, Y = 0.

Also the angel between two vectors: cos := X, YXY .

A Hilbert space is a Banach space, so it has a basis. We

can go one step further: a separable Hilbert spaces has anorthonormal basis (e1,e2, . . .) such that: a) (ei) is a basis;b) ei = 1; c) ei, ej = 0. Given an orthonormal basis,every X X can be expressed as:

X =i=1

X, eiei.

Qiu, Lee BST 401



48/51






X =i=1

X, eiei.

Qiu, Lee BST 401



49/51






X =i=1

X, eiei.

Qiu, Lee BST 401

Applications


50/51

Applications

The first n-terms provides a good approximation of X:

Xn

i=1

X, eiei =

i=n+1

qX, eiei =

i=n+1

X, ei 0.

This approximation is the foundation of nonparametricregression (splines are n-term approximations of an

unknown regression function in an abstract Hilbert space),

Fourier analysis, wavelet analysis, PDE, and much more.

We can define projections in a Hilbert space. A projection

to a Hilbert subspace M X breaks X into two parts,X = ProjMX+ X

. ProjMX Mhas the smallest distancewith X. This is the theoretic foundation of regression

theory.

Qiu, Lee BST 401

Applications


51/51

pp cat o s

The first n-terms provides a good approximation of X:

Xn

i=1

X, eiei =

i=n+1

qX, eiei =

i=n+1

X, ei 0.

This approximation is the foundation of nonparametricregression (splines are n-term approximations of an

unknown regression function in an abstract Hilbert space),

Fourier analysis, wavelet analysis, PDE, and much more.

We can define projections in a Hilbert space. A projection

to a Hilbert subspace M X breaks X into two parts,X = ProjMX+ X

. ProjMX Mhas the smallest distancewith X. This is the theoretic foundation of regression

theory.

Qiu, Lee BST 401

Documents

Probability Theory Presentation 10