Lecture 2 - TU Berlin · TU Berlin| Sekr. HFT 6|Einsteinufer 25|10587Berlin Faculty of Electrical Engineering and Computer Systems Department of Telecommunication

TU Berlin | Sekr. HFT 6 | Einsteinufer 25 | 10587 Berlin

www.mk.tu-berlin.de

Faculty of Electrical Engineering and Computer Systems Department of Telecommunication Systems Information and Communication Theory Prof. Dr. Giuseppe Caire Einsteinufer 25 10587 Berlin Telefon +49 (0)30 314-29668 Telefax +49 (0)30 314-28320 [email protected] Sekretariat HFT6 Patrycja Chudzik Telefon +49 (0)30 314-28459 Telefax +49 (0)30 314-28320 [email protected]

Firma xy Herrn Mustermann Beispielstraße 11 12345 Musterstadt

Berlin, 1. Month 2014

Subject: Text…& Prof. Dr. Giuseppe Caire

Lecture 2:Typicality

Copyright G. Caire (Sample Lectures) 55


www.mk.tu-berlin.de





Typical Sequences

• Let x 2 X n. The empirical pmf of x is defined as

⇡(x|x) =

|{i : xi = x}|n

, for x 2 X

This is also referred to as the “type” of x.

• Let Xn denote an i.i.d. random vector with Xi ⇠ PX. By the (weak) law oflarge numbers

lim

n!1⇡(x|Xn

)

p

= PX(x), for x 2 X

Definition 7. Typical set: For a given pmf PX on X and ✏ > 0, the ✏-typical setof sequences x 2 X n is defined as

T (n)

✏ (X) = {x 2 X n: |⇡(x|x) � PX(x)| ✏PX(x), 8 x 2 X}

⌃



www.mk.tu-berlin.de





Elementary Properties of Typical Sequences

Lemma 6. Typical Average Lemma: Let x 2 T (n)

✏ (X) and let g(·) denote afunction on X for which E[g(X)] is well-defined (i.e.,

P

x2X PX(x)|g(x)| 1).Then,

(1 � ✏)E[g(X)] 1

n

nX

i=1

g(xi) (1 + ✏)E[g(X)]

⇤

(Proof: HOMEWORK.)

Lemma 7. Asymptotic Equipartition Property (AEP): All typical sequenceshave roughly the same probability. For each x 2 T (n)

✏ (X) we have:

2

�n(H(X)+�(✏)) PXn(x) 2

�n(H(X)��(✏))

where �(✏) # 0 as ✏ ! 0. In short, we write PXn(x)

.= 2

�nH(X). ⇤

(Proof: HOMEWORK.).



www.mk.tu-berlin.de





Properties of the Typical Set

• Typical set cardinality upper bound:�

�

�

T (n)

✏ (X)

�

�

�

2

n(H(X)+�(✏))

(Proof: HOMEWORK.)

• Law of Large Numbers (LLN): if Xn is an i.i.d. sequence with Xi ⇠ PX(x)

thenlim

n!1P

⇣

Xn 2 T (n)

✏ (X)

⌘

= 1

• Typical set cardinality lower bound:�

�

�

T (n)

✏ (X)

�

�

�

� (1 � ✏)2n(H(X)��(✏))

for sufficiently large n.(Proof: HOMEWORK.)



www.mk.tu-berlin.de





Intuitive Representation

• Typical Average Lemma: Let xn 2 T (n)

✏ (X). Then for any nonnegative functiong(x) on X ,

(1 � ✏) E(g(X)) 1

n

nX

i=1

g(xi) (1 + ✏) E(g(X))

Proof: From the definition of the typical set,�

�

�

�

�

1

n

nX

i=1

g(xi) � E(g(x))

�

�

�

�

�

=

�

�

�

�

�

X

x

⇡(x|xn)g(x) �

X

x

p(x)g(x)

�

�

�

�

�

X

x

✏ p(x)g(x)

= ✏ · E(g(X))

• Properties of typical sequences:

1. Let p(xn) =

Qni=1

pX(xi). Then, for each xn 2 T (n)

✏ (X)

2

�n(H(X)+�(✏)) p(xn) 2

�n(H(X)��(✏)),

where �(✏) = ✏ · H(X) ! 0 as ✏ ! 0. This follows from the typical averagelemma by taking g(x) = � log pX(x)

LNIT: Information Measures and Typical Sequences (2010-06-22 08:45) Page 2 – 15

2. The cardinality of the typical set�

�T (n)

✏

�

� 2

n(H(X)+�(✏)). This can be shownby summing the lower bound in the previous property over the typical set

3. If X1

,X2

, . . . are i.i.d. with Xi ⇠ pX(xi), then by the LLN

P�

Xn 2 T (n)

✏

! 1

4. The cardinality of the typical set�

�T (n)

✏

�

� � (1 � ✏)2n(H(X)��(✏)) for nsu�ciently large. This follows by property 3 and the upper bound in property1

• The above properties are illustrated in the following figure

X n

T (n)

✏ (X)

p(xn)

.= 2

�nH(X)

|T (n)

✏ | .= 2

nH(X)

P(T (n)

✏ ) � 1 � ✏




www.mk.tu-berlin.de





Jointly Typical Sequences

• Let x,y 2 X n ⇥ Yn. The empirical joint pmf of (x,y) is defined as

⇡(x, y|x,y) =

|{i : (xi, yi) = (x, y)}|n

, for (x, y) 2 X ⇥ Y

Definition 8. Jointly typical set: For a joint pmf PX,Y (x, y) and ✏ > 0, the jointly✏-typical set of sequence pairs (x,y) 2 X n ⇥ Yn is defined as

T (n)

✏ (X, Y ) = {(x,y) 2 X n ⇥ Yn: |⇡(x, y|x,y) � PX,Y (x, y)| ✏PX,Y (x, y),

8 (x, y) 2 X ⇥ Y}

⌃



www.mk.tu-berlin.de





Properties of the Jointly Typical Set

• Let (Xn, Y n) be a jointly distributed, componentwise i.i.d., pair of random

vectors with (Xi, Yi) ⇠ PX,Y (x, y), and let (x,y) 2 T (n)

✏ (X, Y ), then thefollowing properties hold:

1. x 2 T (n)

✏ (X) and y 2 T (n)

✏ (Y ).2. PXn,Y n

(x,y)

.= 2

�nH(X,Y ).3. PXn

(x)

.= 2

�nH(X) and PY n(y)

.= 2

�nH(Y ).4. PXn|Y n

(x|y)

.= 2

�nH(X|Y ) and PY n|Xn(y|x)

.= 2

�nH(Y |X).

(Proof: HOMEWORK.)



www.mk.tu-berlin.de





Size of Conditional Typical Set

• Let

T (n)

✏ (Y |x) =

n

y 2 Yn: (x,y) 2 T (n)

✏ (X, Y )

o

Then

�

�

�

T (n)

✏ (Y |x)

�

�

�

2

n(H(Y |X)+�(✏))

(Proof: HOMEWORK.)



www.mk.tu-berlin.de





Conditional Typicality

Lemma 8. Conditional typicality lemma: Let ✏ > ✏0 > 0. For x 2 T (n)

✏0 (X), letY n ⇠ PY n|Xn

(y|x) =

Qni=1

PY |X(yi|xi). Then

lim

n!1P

⇣

(x, Y n) 2 T (n)

✏ (X, Y )

�

�

�

Xn= x

⌘

= 1

Proof:

In the proof below, the probability measure P(·) denotes the measure of Y n

conditioned on Xn= x (conditioning is omitted for brevity). This is because by

assumption we have Y n ⇠ PY n|Xn(y|x) =

Qni=1

PY |X(yi|xi). The statement isproved if we show that

lim

n!1P (|⇡(x, y|x, Y n

) � PX,Y (x, y)| > ✏PX,Y (x, y) for some (x, y)) = 0



www.mk.tu-berlin.de





For x 2 X such that PX(x) > 0, consider

P (|⇡(x, y|x, Y n) � PX,Y (x, y)| > ✏PX,Y (x, y)) =

= P✓

�

�

�

�

⇡(x, y|x, Y n)

PX(x)

� PY |X(y|x)

�

�

�

�

> ✏PY |X(y|x)

◆

= P✓

�

�

�

�

⇡(x, y|x, Y n)⇡(x|x)

PX(x)⇡(x|x)

� PY |X(y|x)

�

�

�

�

> ✏PY |X(y|x)

◆

= P✓

�

�

�

�

⇡(x, y|x, Y n)

⇡(x|x)PY |X(y|x)

· ⇡(x|x)

PX(x)

� 1

�

�

�

�

> ✏

◆

P✓

⇡(x, y|x, Y n)

⇡(x|x)PY |X(y|x)

· ⇡(x|x)

PX(x)

> 1 + ✏

◆

+ P✓

⇡(x, y|x, Y n)

⇡(x|x)PY |X(y|x)

· ⇡(x|x)

PX(x)

< 1 � ✏

◆



www.mk.tu-berlin.de





Since x 2 T (n)

✏0 (X), then

1 � ✏0 ⇡(x|x)

PX(x)

1 + ✏0

Hence

P✓

⇡(x, y|x, Y n)

⇡(x|x)PY |X(y|x)

· ⇡(x|x)

PX(x)

> 1 + ✏

◆

P✓

⇡(x, y|x, Y n)

⇡(x|x)

>1 + ✏

1 + ✏0PY |X(y|x)

◆

Similarly

P✓

⇡(x, y|x, Y n)

⇡(x|x)PY |X(y|x)

· ⇡(x|x)

PX(x)

< 1 � ✏

◆

P✓

⇡(x, y|x, Y n)

⇡(x|x)

<1 � ✏

1 � ✏0PY |X(y|x)

◆

Since by assumption we have ✏0 < ✏, then

1 � ✏

1 � ✏0 < 1 <1 + ✏

1 + ✏0



www.mk.tu-berlin.de





Furthermore, since Y n ⇠ Qni=1

PY |X(yi|xi), by the law of large numbers wehave

⇡(x, y|x, Y n)

⇡(x|x)

p! PY |X(y|x)

Hence, both the above upper bounds tend to 0 as n ! 1. Taking the unionover all (x, y) 2 X ⇥ Y and using the union bound yields the final result.



www.mk.tu-berlin.de





Consequences of Conditional Typicality

Corollary 7. Let ✏ > ✏0 > 0. For all x 2 T (n)

✏0 (X) and sufficiently large n, wehave �

�

�

T (n)

✏ (Y |x)

�

�

�

� (1 � ✏)2n(H(Y |X)��(✏))

(Proof: HOMEWORK.) Hint: notice that, by definition,

P⇣

(x, Y n) 2 T (n)

✏ (X, Y )

�

�

�

Xn= x

⌘

= P⇣

Y n 2 T (n)

✏ (Y |x)

�

�

�

Xn= x

⌘

and notice that the Conditional Typicality Lemma implies that, for sufficientlylarge n,

P⇣

(x, Y n) 2 T (n)

✏ (X, Y )

�

�

�

Xn= x

⌘

� 1 � ✏



www.mk.tu-berlin.de





Intuitive Visualization of Joint Typicality

• Conditional Typicality Lemma: Let xn 2 T (n)

✏0 (X) and Y n ⇠ Qni=1

pY |X(yi|xi).Then for every ✏ > ✏0,

P{(xn, Y n) 2 T (n)

✏ (X,Y )} ! 1 as n ! 1This follows by the LLN. Note that the condition ✏ > ✏0 is crucial to apply theLLN (why?)

The conditional typicality lemma implies that for all xn 2 T (n)

✏0 (X)

|T (n)

✏ (Y |xn)| � (1 � ✏)2n(H(Y |X)��(✏)) for n su�ciently large

• In fact, a stronger statement holds: For every xn 2 T (n)

✏ (X) and n su�cientlylarge,

|T (n)

✏ (Y |xn)| � 2

n(H(Y |X)��0(✏)),

for some �0(✏) ! 0 as ✏ ! 0

This can be proved by counting jointly typical yn sequences (the method oftypes [12]) as shown in the Appendix


Useful Picture

xn

yn

T (n)

✏ (Y )�| · | .

= 2

nH(Y )

�

T (n)

✏ (X)

�| · | .

= 2

nH(X)

�

T (n)

✏ (X, Y )�| · | .

= 2

nH(X,Y )

�

T (n)

✏ (Y |xn)�

| · | .= 2

nH(Y |X)

� T (n)

✏ (X|yn)�

| · | .= 2

nH(X|Y )

�

LNIT: Information Measures and Typical Sequences (2010-06-22 08:45) Page 2 – 20Copyright G. Caire (Sample Lectures) 68


www.mk.tu-berlin.de





Intuitive Visualization of Typical Fan-Out

Another Useful Picture

T (n)

✏ (X)

xn

X n Yn T (n)

✏ (Y )

T (n)

✏ (Y |xn)

| · | .= 2

nH(Y |X)


Joint Typicality for Random Triples

• Let (X,Y, Z) ⇠ p(x, y, z). The set T (n)

✏ (X1

,X2

, X3

) of ✏-typical n-sequencesis defined by

{(xn, yn, zn) :|⇡(x, y, z |xn, yn, zn

) � p(x, y, z)| ✏ · p(x, y, z)

for all (x, y, z) 2 X ⇥ Y ⇥ Z}• Since this is equivalent to the typical set of a single “large” random variable

(X,Y, Z) or a pair of random variables ((X,Y ), Z), the properties of jointtypical sequences continue to hold

• For example, if p(xn, yn, zn) =

Qni=1

pX,Y,Z(xi, yi, zi) and

(xn, yn, zn) 2 T (n)

✏ (X,Y, Z), then

1. xn 2 T (n)

✏ (X) and (yn, zn) 2 T (n)

✏ (Y, Z)

2. p(xn, yn, zn)

.= 2

�nH(X,Y,Z)

3. p(xn, yn|zn)

.= 2

�nH(X,Y |Z)

4. |T (n)

✏ (X|yn, zn)| .

= 2

nH(X|Y,Z) for n su�ciently large




www.mk.tu-berlin.de





End of Lecture 2


Documents

Lecture 2 - TU Berlin · TU Berlin| Sekr. HFT 6|Einsteinufer 25|10587Berlin Faculty of Electrical Engineering and Computer Systems Department of Telecommunication