SSPre-PFIEV-Chapter4-2012

8/2/2019 SSPre-PFIEV-Chapter4-2012

1/38

Dept. of Telecomm. Eng.Faculty of EEE

SSPre2012DHT, HCMUT1

References

[1] Athanasios Papoulis,Probability, Random Variables, andStochastic Processes, McGraw-Hill, 1991 (3rd Ed.), 2001 (4th Ed.).

[2] Steven M. Kay, Fundamentals of Statistical Signal Processing:

Estimation Theory, Prentice Hall, 1993.

[3] Alan V. Oppenheim, Ronald W. Schafer,Discrete-Time Signal

Processing, Prentice Hall, 1989.

[4] Dimitris G. Manolakis, Vinay K. Ingle, Stephen M. Kogon,

Statistical and Adaptive Signal Processing, Artech House, 2005.


2/38


SSPre2012DHT, HCMUT

Chapter 4:

Mean Square Estimation


3/38


SSPre2012DHT, HCMUT3

4. Minimum Mean Square Estimation (1)

Given some information that is related to an unknown quantity of

interest, the problem is to obtain a good estimate for the unknown in

terms of the observed data.

Suppose represent a sequence of random

variables about whom one set of observations are available, and Y

represents an unknown random variable. The problem is to obtain a

good estimate for Yin terms of the observations

Let

represent such an estimate for Y.

Note that can be a linear or a nonlinear function of the observation

. Clearly,

represents the error in the above estimate, and the square of the error.

nXXX ,,, 21

nXXX ,,, 21

)(),,,( 21 XXXXY n == (4-1)

)(

nXXX ,,, 21

)()( XYYYX == (4-2)2||


4/38


SSPre2012DHT, HCMUT


Since is a random variable, represents the mean square error.

One strategy to obtain a good estimator would be to minimize the mean

square error by varying over all possible forms of and this proceduregives rise to the Minimization of the Mean Square Error (MMSE)

criterion for estimation. Thus under MMSE criterion, the estimator is

chosen such that the mean square error is at its minimum.

Next we show that the conditional mean ofYgivenXis the best estimatorin the above sense.

Theorem 1: Under MMSE criterion, the best estimator for the unknown Y

in terms of is given by the conditional mean ofYgivesX.

Thus

4

}||{2E

),(

),(}||{

2E

nXXX ,,, 21

}.|{)( XYEXY == (4-3)


5/38


SSPre2012DHT, HCMUT


Proof : Let represent an estimate ofYin terms of

Then the error and the mean square

error is given by

Since

we can rewrite (4-4) as

where the inner expectation is with respect to Y, and the outer one is with

respect to Thus

5

)( XY =

).,,,( 21 nXXXX = ,YY=

}|)(|{}||{}||{ 2222 XYEYYEE === (4-4)

}]|{[][ XzEEzEzX

= (4-5)

}]|)(|{[}|)(|{

z

2

z

22XXYEEXYE YX

==

.X

+

=

=

.)(}|)(|{

}]|)(|{[

2

22

dxXfXXYE

XXYEE

X

(4-6)


6/38


SSPre2012DHT, HCMUT


To obtain the best estimator we need to minimize in (4-6)

with respect to . In (4-6), since

and the variable appears only in the integrand term, minimization

of the mean square error in (4-6) with respect to is

equivalent to minimization of with respect to .

Since X is fixed at some value, is no longer random,and hence minimization of is equivalent to

This gives

or

6

2

,0)( XfX ,0}|)(|{2 XXYE

2

}|)(|{ 2 XXYE

)(X}|)(|{

2XXYE

.0}|)(|{ 2 =

XXYE

(4-7)

0}|)({| = XXYE

.0}|)({}|{ = XXEXYE (4-8)


7/38


SSPre2012DHT, HCMUT


But

since when is a fixed number Using (4-9) in (4-8) we

get the desired estimator to be

Thus, the conditional mean ofYgiven represents the bestestimator for Ythat minimizes the mean square error.

The minimum value of the mean square error is given by

7

),(}|)({ XXXE = (4-9)

)(, XxX = ).(x

}.,,,|{}|{)( 21 nXXXYEXYEXY === (4-10)

nXXX ,,, 21

.0)}|{var(

]}|)|(|{[}|)|(|{

)var(

222

min

=

==

XYE

XXYEYEEXYEYE

XY

(4-11)


8/38


SSPre2012DHT, HCMUT


As an example, suppose is the unknown. Then the best MMSE

estimator is given by

Clearly if then indeed is the best estimator for Yin terms

ofX. Thus the best estimator can be nonlinear.

8

3XY =

.}|{}|{ 33 XXXEXYEY === (4-12)

3XY = 3 XY =


9/38


SSPre2012DHT, HCMUT


Example: Let

where k> 0 is a suitable normalization constant. To determine the best

estimate for Yin terms ofX, we need

Thus

9


10/38

Dept. of Telecomm. Eng.Faculty of EEE SSPre2012DHT, HCMUT


Hence the best MMSE estimator is given by

Once again the best estimator is nonlinear. In general the best estimator

is difficult to evaluate, and hence next we will examine the

special subclass of best linear estimators.

10

.1

)1(

3

2

1

1

3

2

13

2

)|(}|{)(

2

2

2

31

2

3

1

2

1

21

1

2

1

22

|

x

xx

x

x

x

y

dyydyy

dyxyfyXYEXY

x

xxx x

y

x XY

++=

=

=

=====

}|{ XYE

(4-14)


11/38


Best Linear Estimator (1)

In this case the estimator is a linear function of the observations

Thus

where are unknown quantities to be determined. The mean

square error is given by

and under the MMSE criterion should be chosen so that the

mean square error is at its minimum possible value. Let

represent that minimum possible value. Then

To minimize (4-16), we can equate:

11

Y

.,,, 21 nXXX

=

=+++=n

iiinnl XaXaXaXaY

12211 .

(4-15)

naaa ,,, 21 )( lYY=

}||{}|{|}||{222

==iil XaYEYYEE (4-16)

naaa ,,, 21

}||{2E

2

n

==

n

i

iiaaa

n XaYEn 1

2

,,,

2

}.|{|min21 (4-17)

,n.,,kEa

k

21,0}|{| 2 ==

(4-18)


12/38



This gives

But

Substituting (4-19) in to (4-18), we get

or the best linear estimator must satisfy

12

.02

||

}|{|

*22

=

=

=

kkk aEaEEa

(4-19)

.

)()(11

k

k

n

i

ii

kk

n

i

ii

k

Xa

Xa

a

Y

a

XaY

a=

=

=

==(4-20)

,0}{2}|{| *

2

==

k

k

XEa

E

.,,2,1,0}{ * nkXEk

== (4-21)


13/38



Notice that in (4-21), represents the estimation error

and represents the data. Thus from (4-21), the error is

orthogonal to the data for the best linear estimator. This isthe orthogonality principle.

In other words, in the linear estimator (4-15), the unknown constants

must be selected such that the error is

orthogonal to every data for the best linear estimator thatminimizes the mean square error.

Interestingly a general form of the orthogonality principle holds good in

the case of nonlinear estimators also.

Nonlinear orthogonality rule: Let represent any functionalform of the data and the best estimator for Ygiven With

, we shall show that

13

=n

i iiXaY

1),(

nkXk =1,

nkXk =1,

naaa ,,, 21 ==

n

i iiXaY

1

nXXX ,,, 21

)(Xh

}|{ XYE .X

}|{ XYEYe =


14/38



implying that

This follows since

Thus in the nonlinear version of the orthogonality rule the error is

orthogonal to any functional form of the data.

14

,0)}({ =XehE (4-22)

).(}|{ XhXYEYe =

.0)}({)}({

]}|)([{)}({})(]|[{)}({

)}(])|[{()}({

==

=

=

=

XYhEXYhE

XXYhEEXYhEXhXYEEXYhE

XhXYEYEXehE


15/38



The orthogonality principle in (4-21) can be used to obtain the unknowns

in the linear case.

For example suppose n = 2, and we need to estimate Yin terms ofX1 and

X2 linearly. Thus

From (4-21), the orthogonality rule gives

Thus

or

Solving (4-23) to obtain in terms of the cross-correlations.

15

naaa ,,, 21

2211 XaXaYl +=

0}){(}X{

0}){(}X{*

22211

*

2

*

12211

*

1

====

XXaXaYEE

XXaXaYEE

}{}|{|}{

}{}{}|{|

*

22

2

21

*

21

*

12

*

121

2

1

YXEaXEaXXE

YXEaXXEaXE

=+

=+

=

}{

}{

}|{|}{

}{}|{|

*

2

*

1

2

1

2

2

*

21

*

12

2

1

YXE

YXE

a

a

XEXXE

XXEXE(4-23)

21 and aa


16/38



The minimum value of the mean square error in (4-17) is given by

But using (4-21), the second term in (4-24) is zero, since the error is

orthogonal to the dataXi where are chosen to be optimum.

Thus the minimum value of the mean square error is given by

where are the optimum values from (4-21).

16

2

n

}.{min}{min

})({min}{min

}|{|min

*

1,,,

*

,,,

1

*

,,,

*

,,,

2

,,,

2

2121

2121

21

l

n

ii

aaaaaa

n

iii

aaaaaa

aaan

XEaYE

XaYEE

E

nn

nn

n

=

=

=

==

=

(4-24)

naaa ,,, 21

}{}|{|

}){(}{

1

*2

1

**2

=

=

=

==

n

iii

n

iiin

YXEaYE

YXaYEYE

naaa ,,, 21

(4-25)


17/38



Since the linear estimate in (4-15) is only a special case of the general

estimator in (4-1), the best linear estimator that satisfies (4-21)

cannot be superior to the best nonlinear estimator Often the bestlinear estimator will be inferior to the best estimator in (4-3).

This raises the following question: Are there situations in which the best

estimator in (4-3) also turns out to be linear? In those situations it is

enough to use (4-21) and obtain the best linear estimators, since they alsorepresent the best global estimators. Such is the case ifYand

are distributed asjointly Gaussian.

We summarize this in the next theorem and prove that result.

Theorem 2: If and Yare jointly Gaussian zero meanrandom variables, then the best estimate for Yin terms of is

always linear.

17

)(X

}.|{ XYE

nXXX ,,, 21

nXXX ,,, 21 nXXX ,,, 21


18/38



Proof : Let

represent the best (possibly nonlinear) estimate ofY, and

the best linear estimate ofY. Then from (4-21)

is orthogonal to the data Thus

Also from (4-28),

Using (4-29)-(4-30), we get

18

}|{),,,( 21 XYEXXXY n == (4-26)

=

=n

i

iilXaY

1

(4-27)

1

n

l i ii

Y Y Y a X =

= = (4-28)

.1, nkXk =

.1,0}{ nkXE*

k == (4-29)

.0}{}{}{1

== =

n

iii XEaYEE (4-30)

.1,0}{}{}{ ** nkXEEXE kk === (4-31)


19/38



From (4-31), we obtain that andXk are zero mean uncorrelated random

variables for But itself represents a Gaussian random

variable, since from (4-28) it represents a linear combination of a set ofjointly Gaussian random variables. Thus and X are jointly Gaussian

and uncorrelated random variables. As a result, andX are independent

random variables. Thus from their independence

But from (4-30), and hence from (4-32)

Substituting (4-28) into (4-33), we get

or

19

.1 nk =

}.{}|{ EXE = (4-32)

,0}{ =E

.0}|{ =XE (4-33)

0}|{}|{ 1

== = XXaYEXE

n

i

ii

(4-34).}|{}|{1

1

l

n

i

ii

n

i

iiYXaXXaEXYE ===

==


20/38

Dept. of Telecomm. Eng.

Faculty of EEE

SSPre2012

DHT, HCMUT


From (4-26), represents the best possible estimator, and

from (4-34), represents the best linear estimator. Thus the best

linear estimator is also the best possible overall estimator in the Gaussian

case. Next we turn our attention to prediction problems using linear

estimators.

20

)(}|{ xXYE =

=n

i iiXa1


21/38


Faculty of EEE

SSPre2012

DHT, HCMUT

Linear Prediction (1)

Suppose are known and is unknown. Thus

and this represents a one-step prediction problem. If the unknown is

then it represents ak-step ahead prediction problem. Returning back tothe one-step predictor, let represent the best linear predictor. Then

where the error

is orthogonal to the data, i.e.,

21

nXXX ,,, 21 1+nX ,1+= nXY

1

+nX

,knX +

11

= ,n

n i i

i

X a X+=

(4-35)

,1,

1

1

1

12211

1111

==

++++=+==

+

+

=

+

=+++

n

n

iii

nnn

n

iiinnnn

aXa

XXaXaXa

XaXXX

(4-36)

.1,0}{ * nkXEkn

== (4-37)


22/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Using (4-36) in (4-37), we get

Suppose represents the sample of a wide sense stationary stochastic

processX(t) so that

Thus (4-38) becomes

Expanding (4-40) for we get the following set of linear

equations

22

+

=

===1

1

**

.1,0}{}{

n

ikiikn nkXXEaXE (4-38)

iX

** )(}{ikkiki

rrkiRXXE === (4-39)

.1,1,0}{ 1

1

1

*nkaraXE

n

n

i

kiikn==== +

+

= (4-40)

,,,2,1 nk =

.0

20

10

10

*

33

*

22

*

11

121302

*

11

1231201

nkrrararara

krrararara

krrararara

nnnn

nnn

nnn

==+++++

==+++++

==+++++

(4-41)


23/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Similarly using (4-25), the minimum mean square error is given by

The n equations in (4-41) together with (4-42) can be represented as

23

.

}){(

}{}{}|{|

01

*

23

*

12

*

1

1

1

*

1

1

1

*

1

*

1

*22

rrararara

raXXaE

XEYEE

nnnn

n

i

ini

n

i

nii

nnnn

+++++=

==

===

+

=+

+

=+

+

(4-42)

.

0

0

0

0

1

2

3

2

1

0

*

1

*

1

*

10

*

2

*

1

20

*

1

*

2

110

*

1

210

=

n

n

nn

nn

n

n

n

a

a

a

a

rrrr

rrrr

rrrr

rrrr

rrrr

(4-43)


24/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Let

Notice that Tn is Hermitian Toeplitz and positive definite. Using (4-44),the unknowns in (4-43) can be represented as

24

.

0*

1

*

1

*

110*

1

210

=

rrrr

rrrr

rrrr

T

nn

n

n

n

(4-44)

=

=

1

2

2

13

2

1

of

column

Last

0

0

0

0

1

n

n

n

n

n T

T

a

a

a

a

(4-45)


25/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Let

Then from (4-45),

Thus

25

.

1,12,11,1

1,22221

1,11211

1

=

++++

+

+

nn

n

n

n

n

n

n

nnn

n

nnn

n

TTT

TTT

TTT

T

(4-46)

.

1

1,1

1,2

1,1

2

2

1

=

++

+

+

nn

n

n

n

n

n

n

n

T

T

T

a

a

a

,01

1,1

2 >=++ nn

n

nT

(4-47)

(4-48)


26/38


Faculty of EEE

SSPre2012

DHT, HCMUT


and

Eq. (4-49) represents the best linear predictor coefficients, and they can beevaluated from the last column ofTn in (4-45). Using these, the best one-

step ahead predictor in (4-35) taken the form

and from (4-48), the minimum mean square error is given by the

(n +1, n +1) entry of

26

.

1

1,1

1,2

1,1

1,1

2

1

=

++

+

+

++

nn

n

n

n

n

n

nn

n

n T

T

T

T

a

a

a

(4-49)

.)(1

1

1,

1,11 =

+

+++

=

n

ii

ni

nnn

n

n XTT

X (4-50)

.1

nT


27/38


Faculty of EEE

SSPre2012

DHT, HCMUT


From (4-36), since the one-step linear prediction error

we can represent (4-51) formally as follows

Thus, let

them from the above figure, we also have the representation

27

,11111 XaXaXaX nnnnnn ++++= + (4-51)

n

n

nnn zazazaX ++++

+ 1 12

1

1

1

,1)( 12

1

1 n

nnn zazazazA

++++= (4-52)

.)(

1

1+

nn

n

XzA


28/38


Faculty of EEE

SSPre2012

DHT, HCMUT


The filter

represents anAR(n) filter, and this shows that linear prediction leads to

an auto regressive (AR) model.

The polynomial in (4-52)-(4-53) can be simplified using

(4-43)-(4-44). To see this, we rewrite as

28

n

nnn zazazazA

zH

++++

==12111

1

)(

1)(

(4-53)

)(zAn

)(zAn

=

=

+++++=

2

n

11)1(

2

1

1)1(

12

1

)1(

21

0

0

0

]1,,,,[

1

]1,,,,[

1)(

n

nn

n

nn

nn

nn

n

Tzzza

a

a

zzz

zazazazazA

(4-54)


29/38


Faculty of EEE

SSPre2012

DHT, HCMUT


To simplify (4-54), we can make use of the following matrix identity

Taking determinants, we get

In particular if we get

Using (4-57) in (4-54), with

29

.

0

0

1

=

BCADC

A

I

ABI

DC

BA

(4-55)

.

1BCADADC

BA = (4-56)

,0D

.0

)1(

1

C

BA

ABCA

n= (4-57)

===

2

1)1(

0

,],1,,,,[

n

n

nnBTAzzzC


30/38


Faculty of EEE

SSPre2012

DHT, HCMUT


we get

Referring back to (4-43), using Cramers rule to solve for

we get

30

.

1

||

01,,,

0

0

0

||

)1()(

1)1(

10

*

2

*

1

110

*

1

210

2

1

2

=

=

zzz

rrrr

rrrr

rrrr

T

zz

T

TzA

nn

nn

n

n

n

n

n

n

n

n

n

n

(4-58)

),1(1 =+na

1||

||

||1201

10

2

1 ===

+n

n

n

n

n

n

n

nT

T

T

rr

rr

a


31/38


Faculty of EEE

SSPre2012

DHT, HCMUT


or

Thus the polynomial (4-58) reduces to

31

.0

||

||

1

2 >=n

n

n

T

T (4-59)

.1

1

||1)(

1

2

1

1

1)1(

10

*

2

*

1

110

*

1

210

1

n

nn

nn

nn

n

n

n

n

zazaza

zzz

rrrr

rrrr

rrrr

TzA

++++=

=

(4-60)


32/38


Faculty of EEE

SSPre2012

DHT, HCMUT


The polynomial in (4-53) can be alternatively represented as

in (4-60), and in fact represents a stableAR filter of

order n, whose input error signal is white noise of constant spectral

height equal to and output is

It can be shown that has all its zeros in provided

thus establishing stability.

32

)(zAn

)(~)(

1

)( nARzAzH n=

n

||/|| 1nn TT .1+nX

)(zAn 1|| >z 0|| >nT


33/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Linear prediction error

From (4-59), the mean square error using n samples is given by

Suppose one more sample from the past is available to evaluate

( i.e., are available). Proceeding as above the new

coefficients and the mean square error can be determined.

From (4-59)-(4-61),

Using another matrix identity it is easy to show that

33

.0||

||

1

2 >=n

nn

T

T (4-61)

1+nX

011 ,,,, XXXX nn

21+n

.||

|| 121

n

nn

T

T++ = (4-62)

).||1(||

||||

2

1

1

2

1 +

+ = nn

n

ns

T

TT (4-63)


34/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Since we must have or for every n.

From (4-63), we have

or

since Thus the mean square error decreases as more and

more samples are used from the past in the linear predictor. In general

from (4-64), the mean square errors for the one-step predictor form a

monotonic non-increasing sequence

whose limiting value

34

,0|| >kT 0)||1(2

1 > +ns 1|| 1


35/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Clearly, corresponds to the irreducible error in linear

prediction using the entire past samples, and it is related to the power

spectrum of the underlying process through the relation

where represents the power spectrum of

For any finite power process, we have

and since Thus

35

02

)(nTX2

1exp ln ( ) 0.

2XX

S d

+

= (4-66)

( ) 0XX

S

( ) ,XX

S d

+

<

)(nTX

( ( ) 0), ln ( ) ( ).XX XX XX

S S S

ln ( ) ( ) .XX XXS d S d + +

< (4-67)


36/38


Faculty of EEE

SSPre2012

DHT, HCMUT


Moreover, if the power spectrum is strictly positive at every frequency,

i.e.,

then from (4-66)

and hence

i.e., for processes that satisfy the strict positivity condition in (4-68)

almost everywhere in the interval the final minimum mean square

error is strictly positive (see (4-70)). i.e., such processes are not

completely predictable even using their entire set of past samples, or they

are inherently stochastic, since the next output contains information that is

not contained in the past samples. Such processes are known as regular

stochastic processes, and their power spectrum is strictly positive.

36

( ) 0, in - ,XXS > < < (4-68)

ln ( ) .XX

S d

+

>

2

1exp ln ( ) 0

2XX

S d e

+

= > =

(4-69)

(4-70)

),,(


37/38


Faculty of EEE

SSPre2012

DHT, HCMUT


37

)(XX

S

Power spectrum of a regular stochastic process:

Conversely, if a process has the following power spectrum,)(

XXS

1 2

such that in then from (4-70),( ) 0XX

S = 21


38/38


Faculty of EEE

SSPre2012

DHT HCMUT


Such processes are completely predictable from their past data samples. In

particular

is completely predictable from its past samples, since consists of

line spectrum.

in (4-71) is a shape deterministic stochastic process.

38

+= k kkk tanTX )cos()( (4-71)

( )XX

S

)(XX

S

1

k

)(nTX

Documents

SSPre-PFIEV-Chapter4-2012