8/15/2019 05_fall_exam1_soln

1/2

Solutions to Exam 1

Problem 1  (12 points)  Let  A1, A2, . . . be a sequence of independent random variables, withP [Ai = 1] = P [Ai =

  1

2] =   1

2  for all  i. Let  Bk  = A1 · · ·Ak.

(a) Does limk→∞Bk  exist in the m.s. sense? Justify your anwswer.(b) Does limk→∞Bk  exist in the a.s. sense? Justify your anwswer.

(c) Let S n =  B1+. . .+Bn. You can use without proof (time is short!) the fact that limm,n→∞E [S mS n] =35

3 ,  which implies that limn→∞   S n  exists in the m.s. sense. Find the mean and variance of the limit

random variable.(d) Does limn→∞   a.s. S n   exist? Justify your anwswer.

(a)  E [(Bk − 0)2] = E [A21]

k = ( 58

)k → 0 as  k →∞. Thus, limk→∞   m.s. Bk  = 0.(b) Each sample path of the sequence   Bk  is monotone nonincreasing and bounded below by zero, and is hence conver-gent. Thus, limk→∞   a.s. Bk exists. (The limit has to be the same as the m.s. limit, so Bk converges to zero almost surely.)(c) Mean square convergence implies convergence of the mean. Thus, the mean of the limit is limn→∞E [S n] = limn→∞

Pnk=1

 E [Bk] P∞

k=1(34

)k = 3. By the property given in the problem statement, the second moment of the limit is   353

 , so the variance of the limit is   35

3  − 32 =   8

3.

(d) Each sample path of the sequence   S n   is monotone nondecreasing and is hence convergent. Thus, limk→∞   a.s. Bkexists. (The limit has to be the same as the m.s. limit.)

Here is a proof of the claim given in part (c) of the problem statement, although the proof was not asked for on the exam.If  j  ≤ k, then  E [BjBk] = E [A

21 · · ·A

2jAj+1 · · ·Ak] = (

5

8)j(3

4)k−j , and a similar expression holds for  j  ≥ k. Therefore,

E [S nS m] =   E [nXj=1

Bj

nXk=1

Bk] =nXj=1

nXk=1

E [BjBk]

→∞Xj=1

∞Xk=1

E [BjBk]

= 2∞Xj=1

∞Xk=j+1

„5

8

«j „3

4

«k−j+∞Xj=1

„5

8

«j

= 2∞

Xj=1∞

Xl=1 „5

8«j

„3

4«l

+∞

Xj=1„5

8«j

= (∞Xj=1

„5

8

«j)(2

∞Xl=1

„3

4

«l+ 1)

=  5

3(2 · 3 + 1) =

  35

3

Problem 2 (12 points)  Let  X, Y, and  Z  be random variables with finite second moments and supposeX  is to be estimated. For each of the following, if true, give a brief explanation. If false, give a counterexample.(a) TRUE or FALSE:  E [|X  − E [X |Y ]|2] ≤  E [|X  −    E [X |Y, Y 2]|2].(b) TRUE or FALSE:  E [|X  − E [X |Y ]|2] = E [|X  −    E [X |Y, Y 2]|2] if  X   and  Y   are jointly Gaussian.

(c) TRUE or FALSE?  E [  |X  −   E [E [X |Z ]  |Y ]|2] ≤  E [|X  − E [X |Y ]|2].(d) TRUE or FALSE? If  E [|X  − E [X |Y ]|2] = Var(X ) then  X   and  Y   are independent.

(a) TRUE. The estimator  E [X |Y  ] yields a smaller MSE than any other function of  Y  , including bE [X |Y, Y  2 ].(b) TRUE. Equality holds because the unconstrained estimator with the smallest mean squre error,   E [X |Y   ], is linear,

and the MSE for bE [X |Y, Y  2 ] is less than or equal to the MSE of any linear estimator.(c) FALSE. For example if   X   is a nonconstant random variable,   Y   =   X , and   Z   ≡   0, then, on one hand,   E [X |Z ] =

E [X ], so   E [E [X |Z ]|Y  ] =   E [X ], and thus   E [|X  − E [E [X |Z ]|Y  ]|2] = Var(X ).  On the other hand,   E [X |Y  ] =   X   so that

E [|X − E [X |Y  ]|2] = 0.

(d) FALSE. For example, suppose X  =  Y W , where W  is a random variable independent of  Y  with mean zero and variance

one. Then given  Y  , the conditional distribution of  X  has mean zero and variance  Y  2 . In particular,  E [X |Y  ] = 0, so that

E [|X − E [X |Y  ]|2] = Var(X ), but  X   and  Y    are not independent.

8/15/2019 05_fall_exam1_soln

2/2

Problem 3   (6 points)  Recall from a homework problem that if 0  < f <  1 and if  S n   is the sum of   nindependent random variables, such that a fraction  f  of the random variables have a CDF  F Y    and afraction 1 − f  have a CDF  F Z , then the large deviations exponent for

  S nn

  is given by:

l(a) = maxθ

{θa − f M Y   (θ) − (1 − f )M Z (θ)}

where  M Y   (θ) and  M Z (θ) are the log moment generating functions for  F Y    and  F Z   respectively.

Consider the following variation. Let  X 1, X 2, . . . , X  n  be independent, and identically distributed, eachwith CDF given by F X (c) = f F Y   (c)+(1−f )F Z (c). Equivalently, each X i can be generated by flippinga biased coin with probability of heads equal to   f , and generating  X i  using CDF   F Y    if heads showsand generating   X i   with CDF   F Z   if tails shows. Let   S n   =  X 1 +  · · · +  X n, and let   l  denote the large

deviations exponent for  eS nn

  .

(a) Express the function   l   in terms of  f ,  M Y   , and  M Z .(b) Determine which is true and give a proof:   l(a) ≤  l(a) for all  a, or   l(a) ≥  l(a) for all  a.

(a) el(a) = maxθ {θa − M X(θ)}  whereM X(θ) = log E [exp(θX )]

= log{f E [exp(θY  )] + (1 − f )E [exp(θZ )]}

= log{f  exp(M Y  (θ)) + (1 − f ) exp(M Z(θ))}

(b) View   f  exp(M Y  (θ)) + (1 − f ) exp(M Z(θ)) as an average of exp(M Y  (θ)) and exp(M Z(θ)). The definition of con-cavity (or Jensen’s inequality) applied to the concave function log u   implies that log(average)  ≥   average(log), so thatlog{f  exp(M Y  (θ))+(1−f ) exp(M Z(θ)) ≥ f M Y  (θ)+(1−f )M Z(θ), where we also used the fact that log exp M Y  (θ) =  M Y  (θ).

Therefore, el(a) ≤ l(a) for all  a.Remark: This means that

  eSnn

  is more likely to have large deviations than   Snn

  . That is reasonable, because  eSnn

  has

randomness due not only to  F Y   and  F Z, but also due to the random coin flips. This point is particularly clear in case the

Y  ’s and  Z ’s are constant, or nearly constant, random variables.