70
A Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback Vincent Y. F. Tan (NUS) Joint work with Silas L. Fong (Toronto) 2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 1 / 22

A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

A Tight Upper Bound on the Second-Order CodingRate of Parallel Gaussian Channels with Feedback

Vincent Y. F. Tan (NUS)

Joint work with Silas L. Fong (Toronto)

2017 Information Theory Workshop, Kaohsiung, Taiwan

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 1 / 22

Page 2: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Channel Model: The Parallel Gaussian Channel

Channel Law (g1 = g2 = . . . = gd = 1)

Yk = Xk + Zk,

Y1,kY2,k

...Yd,k

=

X1,kX2,k

...Xd,k

+

Z1,kZ2,k

...Zd,k

, k ∈ [1 : n]

and Zl,ki.i.d.∼ N (0,Nl) where l ∈ [1 : d].

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 2 / 22

Page 3: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Channel Model: The Parallel Gaussian Channel

Channel Law (g1 = g2 = . . . = gd = 1)

Yk = Xk + Zk,

Y1,kY2,k

...Yd,k

=

X1,kX2,k

...Xd,k

+

Z1,kZ2,k

...Zd,k

, k ∈ [1 : n]

and Zl,ki.i.d.∼ N (0,Nl) where l ∈ [1 : d].

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 2 / 22

Page 4: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Capacity of the Parallel Gaussian Channel

Consider a peak power constraint

Pr

1n

d∑l=1

n∑k=1

X2l,k ≤ P

= 1

Define the capacity functions

C(s) =

d∑l=1

C(

sl

Nl

), where C(P) :=

12

log(1 + P).

“Vanishing-error” capacity is given by the water-filling solution

C(P∗), where P∗ = (P∗1,P∗2, . . . ,P

∗d)

and

P∗l = (λ− Nl)+ and λ > 0 satisfies

d∑l=1

P∗l = P.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 3 / 22

Page 5: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Capacity of the Parallel Gaussian Channel

Consider a peak power constraint

Pr

1n

d∑l=1

n∑k=1

X2l,k ≤ P

= 1

Define the capacity functions

C(s) =

d∑l=1

C(

sl

Nl

), where C(P) :=

12

log(1 + P).

“Vanishing-error” capacity is given by the water-filling solution

C(P∗), where P∗ = (P∗1,P∗2, . . . ,P

∗d)

and

P∗l = (λ− Nl)+ and λ > 0 satisfies

d∑l=1

P∗l = P.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 3 / 22

Page 6: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Capacity of the Parallel Gaussian Channel

Consider a peak power constraint

Pr

1n

d∑l=1

n∑k=1

X2l,k ≤ P

= 1

Define the capacity functions

C(s) =

d∑l=1

C(

sl

Nl

), where C(P) :=

12

log(1 + P).

“Vanishing-error” capacity is given by the water-filling solution

C(P∗), where P∗ = (P∗1,P∗2, . . . ,P

∗d)

and

P∗l = (λ− Nl)+ and λ > 0 satisfies

d∑l=1

P∗l = P.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 3 / 22

Page 7: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Capacity vs. Non-Asymptotic Fundamental Limits

Define the non-asymptotic fundamental limit

M∗(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-code.

Maximum number of messages that can be supported by alength-n code with peak power P and average error prob. ε.

Capacity result can be stated as

limε↓0

lim infn→∞

1n

log M∗(n, ε,P) = C(P∗).

In fact, the strong converse holds, and we have

lim infn→∞

1n

log M∗(n, ε,P) = C(P∗), ∀ ε ∈ (0, 1).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 4 / 22

Page 8: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Capacity vs. Non-Asymptotic Fundamental Limits

Define the non-asymptotic fundamental limit

M∗(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-code.

Maximum number of messages that can be supported by alength-n code with peak power P and average error prob. ε.Capacity result can be stated as

limε↓0

lim infn→∞

1n

log M∗(n, ε,P) = C(P∗).

In fact, the strong converse holds, and we have

lim infn→∞

1n

log M∗(n, ε,P) = C(P∗), ∀ ε ∈ (0, 1).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 4 / 22

Page 9: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Capacity vs. Non-Asymptotic Fundamental Limits

Define the non-asymptotic fundamental limit

M∗(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-code.

Maximum number of messages that can be supported by alength-n code with peak power P and average error prob. ε.Capacity result can be stated as

limε↓0

lim infn→∞

1n

log M∗(n, ε,P) = C(P∗).

In fact, the strong converse holds, and we have

lim infn→∞

1n

log M∗(n, ε,P) = C(P∗), ∀ ε ∈ (0, 1).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 4 / 22

Page 10: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Strong Converse

ε = lim supn→∞

P(n)e , R = lim inf

n→∞

1n

log M∗(n, ε,P)

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Peak

R

ε

C(P )

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 5 / 22

Page 11: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Second-Order Asymptotics

LetV(P) :=

P(P + 2)

2(P + 1)2

be the dispersion of the point-to-point AWGN channel.

Define the sum dispersion function to be

V(s) :=d∑

l=1

V(

sl

Nl

).

Then Polyanskiy (2010) and Tomamichel-T. (2015) showed that

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

where P∗ = (P∗1,P∗2, . . . ,P

∗d) is the optimal power allocation and

Φ(b) =

∫ b

−∞

1√2π

exp(− u2

2

)du.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 6 / 22

Page 12: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Second-Order Asymptotics

LetV(P) :=

P(P + 2)

2(P + 1)2

be the dispersion of the point-to-point AWGN channel.Define the sum dispersion function to be

V(s) :=

d∑l=1

V(

sl

Nl

).

Then Polyanskiy (2010) and Tomamichel-T. (2015) showed that

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

where P∗ = (P∗1,P∗2, . . . ,P

∗d) is the optimal power allocation and

Φ(b) =

∫ b

−∞

1√2π

exp(− u2

2

)du.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 6 / 22

Page 13: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Second-Order Asymptotics

LetV(P) :=

P(P + 2)

2(P + 1)2

be the dispersion of the point-to-point AWGN channel.Define the sum dispersion function to be

V(s) :=

d∑l=1

V(

sl

Nl

).

Then Polyanskiy (2010) and Tomamichel-T. (2015) showed that

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

where P∗ = (P∗1,P∗2, . . . ,P

∗d) is the optimal power allocation and

Φ(b) =

∫ b

−∞

1√2π

exp(− u2

2

)du.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 6 / 22

Page 14: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Feedback

If feedback is present, then the encoder at time k has access tomessage W and previous channel outputs (Y1,Y2, . . . ,Yk−1), i.e.,

Xk = Xk(W,Yk−1), ∀ k ∈ [1 : n].

Define the non-asymptotic fundamental limit

M∗FB(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-feedback code.

Since feedback does not increase capacity of point-to-pointmemoryless channels [Shannon (1956)],

limε↓0

lim infn→∞

1n

log M∗FB(n, ε,P) = C(P∗).

Natural question: What happens to the second-order terms?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 7 / 22

Page 15: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Feedback

If feedback is present, then the encoder at time k has access tomessage W and previous channel outputs (Y1,Y2, . . . ,Yk−1), i.e.,

Xk = Xk(W,Yk−1), ∀ k ∈ [1 : n].

Define the non-asymptotic fundamental limit

M∗FB(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-feedback code.

Since feedback does not increase capacity of point-to-pointmemoryless channels [Shannon (1956)],

limε↓0

lim infn→∞

1n

log M∗FB(n, ε,P) = C(P∗).

Natural question: What happens to the second-order terms?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 7 / 22

Page 16: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Feedback

If feedback is present, then the encoder at time k has access tomessage W and previous channel outputs (Y1,Y2, . . . ,Yk−1), i.e.,

Xk = Xk(W,Yk−1), ∀ k ∈ [1 : n].

Define the non-asymptotic fundamental limit

M∗FB(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-feedback code.

Since feedback does not increase capacity of point-to-pointmemoryless channels [Shannon (1956)],

limε↓0

lim infn→∞

1n

log M∗FB(n, ε,P) = C(P∗).

Natural question: What happens to the second-order terms?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 7 / 22

Page 17: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Feedback

If feedback is present, then the encoder at time k has access tomessage W and previous channel outputs (Y1,Y2, . . . ,Yk−1), i.e.,

Xk = Xk(W,Yk−1), ∀ k ∈ [1 : n].

Define the non-asymptotic fundamental limit

M∗FB(n, ε,P) := maxM ∈ N : ∃ (n,M, ε,P)-feedback code.

Since feedback does not increase capacity of point-to-pointmemoryless channels [Shannon (1956)],

limε↓0

lim infn→∞

1n

log M∗FB(n, ε,P) = C(P∗).

Natural question: What happens to the second-order terms?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 7 / 22

Page 18: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Main Result

Theorem (Fong-T. (2017))

Feedback does not affect the second-order term for parallel Gaussianchannels with feedback, i.e.,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Recall that without feedback

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

No guarantees on third-order term.For achievability, encoder can ignore feedback.Only need to prove converse part.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 8 / 22

Page 19: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Main Result

Theorem (Fong-T. (2017))

Feedback does not affect the second-order term for parallel Gaussianchannels with feedback, i.e.,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Recall that without feedback

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

No guarantees on third-order term.For achievability, encoder can ignore feedback.Only need to prove converse part.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 8 / 22

Page 20: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Main Result

Theorem (Fong-T. (2017))

Feedback does not affect the second-order term for parallel Gaussianchannels with feedback, i.e.,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Recall that without feedback

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

No guarantees on third-order term.

For achievability, encoder can ignore feedback.Only need to prove converse part.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 8 / 22

Page 21: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Main Result

Theorem (Fong-T. (2017))

Feedback does not affect the second-order term for parallel Gaussianchannels with feedback, i.e.,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Recall that without feedback

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

No guarantees on third-order term.For achievability, encoder can ignore feedback.

Only need to prove converse part.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 8 / 22

Page 22: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Main Result

Theorem (Fong-T. (2017))

Feedback does not affect the second-order term for parallel Gaussianchannels with feedback, i.e.,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Recall that without feedback

log M∗(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + O(log n).

No guarantees on third-order term.For achievability, encoder can ignore feedback.Only need to prove converse part.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 8 / 22

Page 23: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Comparison to AWGN channel with feedback

For the AWGN channel with feedback (d = 1), information density

logpn

Y|X(Yn|xn)

pnY∗(Yn)

= −‖Yn − xn‖2

2N+ c

=〈xn + Zn, xn〉

N+ c

=1N

n∑k=1

Zkxk + c

has the same distribution for all xn such that ‖xn‖2 = nP.

This spherical symmetry argument [Polyanskiy (2011, 2013) andFong-T. (2015)] yields

log M∗FB(n, ε,P) = nC(P) +√

nV(P)Φ−1(ε) + O(log n).

Symmetry argument doesn’t work for parallel Gaussians.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 9 / 22

Page 24: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Comparison to AWGN channel with feedback

For the AWGN channel with feedback (d = 1), information density

logpn

Y|X(Yn|xn)

pnY∗(Yn)

= −‖Yn − xn‖2

2N+ c

=〈xn + Zn, xn〉

N+ c

=1N

n∑k=1

Zkxk + c

has the same distribution for all xn such that ‖xn‖2 = nP.

This spherical symmetry argument [Polyanskiy (2011, 2013) andFong-T. (2015)] yields

log M∗FB(n, ε,P) = nC(P) +√

nV(P)Φ−1(ε) + O(log n).

Symmetry argument doesn’t work for parallel Gaussians.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 9 / 22

Page 25: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Comparison to AWGN channel with feedback

For the AWGN channel with feedback (d = 1), information density

logpn

Y|X(Yn|xn)

pnY∗(Yn)

= −‖Yn − xn‖2

2N+ c

=〈xn + Zn, xn〉

N+ c

=1N

n∑k=1

Zkxk + c

has the same distribution for all xn such that ‖xn‖2 = nP.

This spherical symmetry argument [Polyanskiy (2011, 2013) andFong-T. (2015)] yields

log M∗FB(n, ε,P) = nC(P) +√

nV(P)Φ−1(ε) + O(log n).

Symmetry argument doesn’t work for parallel Gaussians.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 9 / 22

Page 26: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Comparison to AWGN channel with feedback

For the AWGN channel with feedback (d = 1), information density

logpn

Y|X(Yn|xn)

pnY∗(Yn)

= −‖Yn − xn‖2

2N+ c

=〈xn + Zn, xn〉

N+ c

=1N

n∑k=1

Zkxk + c

has the same distribution for all xn such that ‖xn‖2 = nP.

This spherical symmetry argument [Polyanskiy (2011, 2013) andFong-T. (2015)] yields

log M∗FB(n, ε,P) = nC(P) +√

nV(P)Φ−1(ε) + O(log n).

Symmetry argument doesn’t work for parallel Gaussians.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 9 / 22

Page 27: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Comparison to AWGN channel with feedback

For the AWGN channel with feedback (d = 1), information density

logpn

Y|X(Yn|xn)

pnY∗(Yn)

= −‖Yn − xn‖2

2N+ c

=〈xn + Zn, xn〉

N+ c

=1N

n∑k=1

Zkxk + c

has the same distribution for all xn such that ‖xn‖2 = nP.This spherical symmetry argument [Polyanskiy (2011, 2013) andFong-T. (2015)] yields

log M∗FB(n, ε,P) = nC(P) +√

nV(P)Φ−1(ε) + O(log n).

Symmetry argument doesn’t work for parallel Gaussians.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 9 / 22

Page 28: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Comparison to AWGN channel with feedback

For the AWGN channel with feedback (d = 1), information density

logpn

Y|X(Yn|xn)

pnY∗(Yn)

= −‖Yn − xn‖2

2N+ c

=〈xn + Zn, xn〉

N+ c

=1N

n∑k=1

Zkxk + c

has the same distribution for all xn such that ‖xn‖2 = nP.This spherical symmetry argument [Polyanskiy (2011, 2013) andFong-T. (2015)] yields

log M∗FB(n, ε,P) = nC(P) +√

nV(P)Φ−1(ε) + O(log n).

Symmetry argument doesn’t work for parallel Gaussians.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 9 / 22

Page 29: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Elements of Converse Proof

Want to show

lim supn→∞

log M∗FB(n, ε,P)− nC(P∗)√n

≤√

V(P∗)Φ−1(ε).

Discretizing the power allocation: Turn power allocations intopower types

Apply the meta-converse [Polyanskiy-Poor-Verdú (2010)]

Apply Curtiss’ theorem to the information spectrum term forclose-to-optimal power types

Apply large deviations bounds to far-from-optimal power types

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 10 / 22

Page 30: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Elements of Converse Proof

Want to show

lim supn→∞

log M∗FB(n, ε,P)− nC(P∗)√n

≤√

V(P∗)Φ−1(ε).

Discretizing the power allocation: Turn power allocations intopower types

Apply the meta-converse [Polyanskiy-Poor-Verdú (2010)]

Apply Curtiss’ theorem to the information spectrum term forclose-to-optimal power types

Apply large deviations bounds to far-from-optimal power types

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 10 / 22

Page 31: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Elements of Converse Proof

Want to show

lim supn→∞

log M∗FB(n, ε,P)− nC(P∗)√n

≤√

V(P∗)Φ−1(ε).

Discretizing the power allocation: Turn power allocations intopower types

Apply the meta-converse [Polyanskiy-Poor-Verdú (2010)]

Apply Curtiss’ theorem to the information spectrum term forclose-to-optimal power types

Apply large deviations bounds to far-from-optimal power types

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 10 / 22

Page 32: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Elements of Converse Proof

Want to show

lim supn→∞

log M∗FB(n, ε,P)− nC(P∗)√n

≤√

V(P∗)Φ−1(ε).

Discretizing the power allocation: Turn power allocations intopower types

Apply the meta-converse [Polyanskiy-Poor-Verdú (2010)]

Apply Curtiss’ theorem to the information spectrum term forclose-to-optimal power types

Apply large deviations bounds to far-from-optimal power types

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 10 / 22

Page 33: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Elements of Converse Proof

Want to show

lim supn→∞

log M∗FB(n, ε,P)− nC(P∗)√n

≤√

V(P∗)Φ−1(ε).

Discretizing the power allocation: Turn power allocations intopower types

Apply the meta-converse [Polyanskiy-Poor-Verdú (2010)]

Apply Curtiss’ theorem to the information spectrum term forclose-to-optimal power types

Apply large deviations bounds to far-from-optimal power types

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 10 / 22

Page 34: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part I

Given a channel input xn ∈ Rd×n, define its power type to be

φ(xn) =1n

[‖xn

1‖2, ‖xn2‖2, . . . , ‖xn

d‖2]

=1n

[n∑

k=1

x21,k,

n∑k=1

x22,k, . . . ,

n∑k=1

x2d,k

]∈ Rd

+.

Create a new code such that almost surely,

n∑k=1

X2l,k = mlP, for some ml ∈ [1 : n] and ∀ l ∈ [1 : d]

andd∑

l=1

n∑k=1

X2l,k = nP, i.e.,

d∑l=1

ml = n.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 11 / 22

Page 35: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part I

Given a channel input xn ∈ Rd×n, define its power type to be

φ(xn) =1n

[‖xn

1‖2, ‖xn2‖2, . . . , ‖xn

d‖2]

=1n

[n∑

k=1

x21,k,

n∑k=1

x22,k, . . . ,

n∑k=1

x2d,k

]∈ Rd

+.

Create a new code such that almost surely,

n∑k=1

X2l,k = mlP, for some ml ∈ [1 : n] and ∀ l ∈ [1 : d]

andd∑

l=1

n∑k=1

X2l,k = nP, i.e.,

d∑l=1

ml = n.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 11 / 22

Page 36: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part II

Power allocation vector set

S(n) :=

Pn· a

∣∣∣∣∣ a = (a1, a2, . . . , ad) ∈ Zd+,

d∑l=1

al = n

-

6

a1

a2

u u uu

uuu

P/n

P/n

(n, 0)

(0, n) @@@@@

@@@

@@

@@@

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 12 / 22

Page 37: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part II

Power allocation vector set

S(n) :=

Pn· a

∣∣∣∣∣ a = (a1, a2, . . . , ad) ∈ Zd+,

d∑l=1

al = n

-

6

a1

a2

u u uu

uuu

P/n

P/n

(n, 0)

(0, n) @@@@@@@@@@@@@

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 12 / 22

Page 38: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part III

Set of quantized power allocation vectors s with quantization levelP/n that satisfy the equality power constraint

d∑l=1

sl = P.

With the above construction, almost surely,

φ(Xn) ∈ S(n).

So we can pretend that the codewords are discrete and leveragesome existing theory from second-order analysis for DMCs.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 13 / 22

Page 39: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part III

Set of quantized power allocation vectors s with quantization levelP/n that satisfy the equality power constraint

d∑l=1

sl = P.

With the above construction, almost surely,

φ(Xn) ∈ S(n).

So we can pretend that the codewords are discrete and leveragesome existing theory from second-order analysis for DMCs.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 13 / 22

Page 40: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Turn Power Allocation into Types: Part III

Set of quantized power allocation vectors s with quantization levelP/n that satisfy the equality power constraint

d∑l=1

sl = P.

With the above construction, almost surely,

φ(Xn) ∈ S(n).

So we can pretend that the codewords are discrete and leveragesome existing theory from second-order analysis for DMCs.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 13 / 22

Page 41: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Application of the Meta-Converse (PPV’10)

By a relaxation of the meta-converse, for any ξ > 0,

log M∗FB(n, ε,P) ≤ log ξ−log

(1− ε−Pr

log

pnY|X(Yn|Xn)

qYn(Yn)≥ log ξ

)

Choose auxiliary output distribution qYn carefully.Inspired by Hayashi (2009), we choose

qYn(yn)=12· 1|S(n)|

∑s∈S(n)

∏l,k

N (yl,k; 0, sl+Nl)︸ ︷︷ ︸q(1)

Yn (yn)

+12·∏l,k

N (yl,k; 0,Pl+Nl)︸ ︷︷ ︸q(2)

Yn (yn)

q(1)Yn (yn): Unif. mixture of output dist. induced by input power types

q(2)Yn (yn): Capacity-achieving output dist.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 14 / 22

Page 42: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Application of the Meta-Converse (PPV’10)

By a relaxation of the meta-converse, for any ξ > 0,

log M∗FB(n, ε,P) ≤ log ξ−log

(1− ε−Pr

log

pnY|X(Yn|Xn)

qYn(Yn)≥ log ξ

)

Choose auxiliary output distribution qYn carefully.

Inspired by Hayashi (2009), we choose

qYn(yn)=12· 1|S(n)|

∑s∈S(n)

∏l,k

N (yl,k; 0, sl+Nl)︸ ︷︷ ︸q(1)

Yn (yn)

+12·∏l,k

N (yl,k; 0,Pl+Nl)︸ ︷︷ ︸q(2)

Yn (yn)

q(1)Yn (yn): Unif. mixture of output dist. induced by input power types

q(2)Yn (yn): Capacity-achieving output dist.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 14 / 22

Page 43: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Application of the Meta-Converse (PPV’10)

By a relaxation of the meta-converse, for any ξ > 0,

log M∗FB(n, ε,P) ≤ log ξ−log

(1− ε−Pr

log

pnY|X(Yn|Xn)

qYn(Yn)≥ log ξ

)

Choose auxiliary output distribution qYn carefully.Inspired by Hayashi (2009), we choose

qYn(yn)=12· 1|S(n)|

∑s∈S(n)

∏l,k

N (yl,k; 0, sl+Nl)︸ ︷︷ ︸q(1)

Yn (yn)

+12·∏l,k

N (yl,k; 0,Pl+Nl)︸ ︷︷ ︸q(2)

Yn (yn)

q(1)Yn (yn): Unif. mixture of output dist. induced by input power types

q(2)Yn (yn): Capacity-achieving output dist.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 14 / 22

Page 44: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Application of the Meta-Converse (PPV’10)

By a relaxation of the meta-converse, for any ξ > 0,

log M∗FB(n, ε,P) ≤ log ξ−log

(1− ε−Pr

log

pnY|X(Yn|Xn)

qYn(Yn)≥ log ξ

)

Choose auxiliary output distribution qYn carefully.Inspired by Hayashi (2009), we choose

qYn(yn)=12· 1|S(n)|

∑s∈S(n)

∏l,k

N (yl,k; 0, sl+Nl)︸ ︷︷ ︸q(1)

Yn (yn)

+12·∏l,k

N (yl,k; 0,Pl+Nl)︸ ︷︷ ︸q(2)

Yn (yn)

q(1)Yn (yn): Unif. mixture of output dist. induced by input power types

q(2)Yn (yn): Capacity-achieving output dist.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 14 / 22

Page 45: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Application of the Meta-Converse (PPV’10)

By a relaxation of the meta-converse, for any ξ > 0,

log M∗FB(n, ε,P) ≤ log ξ−log

(1− ε−Pr

log

pnY|X(Yn|Xn)

qYn(Yn)≥ log ξ

)

Choose auxiliary output distribution qYn carefully.Inspired by Hayashi (2009), we choose

qYn(yn)=12· 1|S(n)|

∑s∈S(n)

∏l,k

N (yl,k; 0, sl+Nl)︸ ︷︷ ︸q(1)

Yn (yn)

+12·∏l,k

N (yl,k; 0,Pl+Nl)︸ ︷︷ ︸q(2)

Yn (yn)

q(1)Yn (yn): Unif. mixture of output dist. induced by input power types

q(2)Yn (yn): Capacity-achieving output dist.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 14 / 22

Page 46: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part I

Define the set of almost-optimal power types

Π(n) :=

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 ≤1

n1/6

AAAAAAAAAAAAAAA

S(n)

q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q qq q q q q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q q q q qq q q q q q q q q q qq q q q q q q q q qq q q q q q q q qq q q q q q q qq q q q q q qq q q q q qq q q q qq q q qq q qq qq

XXXXXXz

P∗ t

CCCCCCCCW

Π(n)

q qq qq qq

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 15 / 22

Page 47: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part I

Define the set of almost-optimal power types

Π(n) :=

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 ≤1

n1/6

AAAAAAAAAAAAAAA

S(n)

q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q qq q q q q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q q q q qq q q q q q q q q q qq q q q q q q q q qq q q q q q q q qq q q q q q q qq q q q q q qq q q q q qq q q q qq q q qq q qq qq

XXXXXXz

P∗ t

CCCCCCCCW

Π(n)

q qq qq qq

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 15 / 22

Page 48: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part I

Define the set of almost-optimal power types

Π(n) :=

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 ≤1

n1/6

AAAAAAAAAAAAAAA

S(n)

q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q qq q q q q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q q q q qq q q q q q q q q q qq q q q q q q q q qq q q q q q q q qq q q q q q q qq q q q q q qq q q q q qq q q q qq q q qq q qq qq

XXXXXXz

P∗ t

CCCCCCCCW

Π(n)

q qq qq qq

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 15 / 22

Page 49: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part I

Define the set of almost-optimal power types

Π(n) :=

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 ≤1

n1/6

AAAAAAAAAAAAAAA

S(n)

q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q qq q q q q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q q q q qq q q q q q q q q q qq q q q q q q q q qq q q q q q q q qq q q q q q q qq q q q q q qq q q q q qq q q q qq q q qq q qq qq

XXXXXXz

P∗ t

CCCCCCCCW

Π(n)

q qq qq qq

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 15 / 22

Page 50: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part II

Define threshold

log ξ := nC(P∗) +√

n(√

V(P∗)Φ−1(ε+ τ))

+ log(

2|S(n)|)

Contribution of information spectrum term on Π(n) is

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

where

U(P∗)k :=

d∑l=1

−(

P∗l

Nl

)Z2

l,k + 2Xl,kZl,k + P∗l2(P∗l + Nl)

k ∈ [1 : n]

and φ(Xn) ∈ Π(n).

Because of feedback, U(P∗)k is not independent across time!

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 16 / 22

Page 51: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part II

Define threshold

log ξ := nC(P∗) +√

n(√

V(P∗)Φ−1(ε+ τ))

+ log(

2|S(n)|)

Contribution of information spectrum term on Π(n) is

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

where

U(P∗)k :=

d∑l=1

−(

P∗l

Nl

)Z2

l,k + 2Xl,kZl,k + P∗l2(P∗l + Nl)

k ∈ [1 : n]

and φ(Xn) ∈ Π(n).

Because of feedback, U(P∗)k is not independent across time!

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 16 / 22

Page 52: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part II

Define threshold

log ξ := nC(P∗) +√

n(√

V(P∗)Φ−1(ε+ τ))

+ log(

2|S(n)|)

Contribution of information spectrum term on Π(n) is

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

where

U(P∗)k :=

d∑l=1

−(

P∗l

Nl

)Z2

l,k + 2Xl,kZl,k + P∗l2(P∗l + Nl)

k ∈ [1 : n]

and φ(Xn) ∈ Π(n).

Because of feedback, U(P∗)k is not independent across time!

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 16 / 22

Page 53: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part III

Would like to approximate the nasty

U(P∗)k | k ∈ [1 : n] with the

independent and identically distributed random variables

V(P∗)k :=

d∑l=1

−(

P∗l

Nl

)Z2

l,k + 2√

PlZl,k + P∗l2(P∗l + Nl)

k ∈ [1 : n]

Due to our discretization procedure and the fact that φ(Xn) ∈ Π(n)

limn→∞

E

[exp

( t√n

n∑k=1

U(P∗)k

)]= lim

n→∞E

[exp

( t√n

n∑k=1

V(P∗)k

)]

for all t ∈ R.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 17 / 22

Page 54: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part III

Would like to approximate the nasty

U(P∗)k | k ∈ [1 : n] with the

independent and identically distributed random variables

V(P∗)k :=

d∑l=1

−(

P∗l

Nl

)Z2

l,k + 2√

PlZl,k + P∗l2(P∗l + Nl)

k ∈ [1 : n]

Due to our discretization procedure and the fact that φ(Xn) ∈ Π(n)

limn→∞

E

[exp

( t√n

n∑k=1

U(P∗)k

)]= lim

n→∞E

[exp

( t√n

n∑k=1

V(P∗)k

)]

for all t ∈ R.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 17 / 22

Page 55: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part IV

Curtiss’ (aka Lévy’s continuity) theorem:

limn→∞

E[exp(tXn)] = limn→∞

E[exp(tYn)], ∀ t ∈ R,

implies that

limn→∞

PrXn ≤ a = limn→∞

PrYn ≤ a, ∀ a ∈ R.

Thus,

limn→∞

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

= limn→∞

Pr

1√

nV(P∗)

n∑k=1

V(P∗)k ≥ Φ−1(ε+ τ)

CLT= 1− (ε+ τ).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 18 / 22

Page 56: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part IV

Curtiss’ (aka Lévy’s continuity) theorem:

limn→∞

E[exp(tXn)] = limn→∞

E[exp(tYn)], ∀ t ∈ R,

implies that

limn→∞

PrXn ≤ a = limn→∞

PrYn ≤ a, ∀ a ∈ R.

Thus,

limn→∞

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

= limn→∞

Pr

1√

nV(P∗)

n∑k=1

V(P∗)k ≥ Φ−1(ε+ τ)

CLT= 1− (ε+ τ).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 18 / 22

Page 57: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part IV

Curtiss’ (aka Lévy’s continuity) theorem:

limn→∞

E[exp(tXn)] = limn→∞

E[exp(tYn)], ∀ t ∈ R,

implies that

limn→∞

PrXn ≤ a = limn→∞

PrYn ≤ a, ∀ a ∈ R.

Thus,

limn→∞

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

= limn→∞

Pr

1√

nV(P∗)

n∑k=1

V(P∗)k ≥ Φ−1(ε+ τ)

CLT= 1− (ε+ τ).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 18 / 22

Page 58: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Almost-Optimal Types: Part IV

Curtiss’ (aka Lévy’s continuity) theorem:

limn→∞

E[exp(tXn)] = limn→∞

E[exp(tYn)], ∀ t ∈ R,

implies that

limn→∞

PrXn ≤ a = limn→∞

PrYn ≤ a, ∀ a ∈ R.

Thus,

limn→∞

Pr

1√

nV(P∗)

n∑k=1

U(P∗)k ≥ Φ−1(ε+ τ)

= limn→∞

Pr

1√

nV(P∗)

n∑k=1

V(P∗)k ≥ Φ−1(ε+ τ)

CLT= 1− (ε+ τ).

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 18 / 22

Page 59: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Far-From-Optimal Types

Contribution of information spectrum term on S(n) \Π(n) is

≈ Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

, ∀ s ∈ S(n) \Π(n)

Recall

S(n) \Π(n) =

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 >1

n1/6

On this set,

C(P∗)− C(s) = Ω

(1

n1/3

).

Thus, by a standard exponential Markov inequality argument

maxs∈S(n)\Π(n)

Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

≤ exp

(− O(n1/3)

).

Polynomially many power types.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 19 / 22

Page 60: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Far-From-Optimal Types

Contribution of information spectrum term on S(n) \Π(n) is

≈ Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

, ∀ s ∈ S(n) \Π(n)

Recall

S(n) \Π(n) =

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 >1

n1/6

On this set,

C(P∗)− C(s) = Ω

(1

n1/3

).

Thus, by a standard exponential Markov inequality argument

maxs∈S(n)\Π(n)

Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

≤ exp

(− O(n1/3)

).

Polynomially many power types.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 19 / 22

Page 61: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Far-From-Optimal Types

Contribution of information spectrum term on S(n) \Π(n) is

≈ Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

, ∀ s ∈ S(n) \Π(n)

Recall

S(n) \Π(n) =

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 >1

n1/6

On this set,

C(P∗)− C(s) = Ω

(1

n1/3

).

Thus, by a standard exponential Markov inequality argument

maxs∈S(n)\Π(n)

Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

≤ exp

(− O(n1/3)

).

Polynomially many power types.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 19 / 22

Page 62: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Far-From-Optimal Types

Contribution of information spectrum term on S(n) \Π(n) is

≈ Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

, ∀ s ∈ S(n) \Π(n)

Recall

S(n) \Π(n) =

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 >1

n1/6

On this set,

C(P∗)− C(s) = Ω

(1

n1/3

).

Thus, by a standard exponential Markov inequality argument

maxs∈S(n)\Π(n)

Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

≤ exp

(− O(n1/3)

).

Polynomially many power types.

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 19 / 22

Page 63: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Far-From-Optimal Types

Contribution of information spectrum term on S(n) \Π(n) is

≈ Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

, ∀ s ∈ S(n) \Π(n)

Recall

S(n) \Π(n) =

s ∈ S(n)

∣∣∣∣ ‖s− P∗‖2 >1

n1/6

On this set,

C(P∗)− C(s) = Ω

(1

n1/3

).

Thus, by a standard exponential Markov inequality argument

maxs∈S(n)\Π(n)

Pr

1n

n∑k=1

U(s)k ≥ C(P∗)− C(s)

≤ exp

(− O(n1/3)

).

Polynomially many power types.Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 19 / 22

Page 64: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Wrap-Up and Future Work

For parallel Gaussian channels with feedback,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Careful quantization of power allocation vector into power typesBounding main information spectrum term using Curtiss’ theoremFor third-order, need speed of convergence, i.e.,

supt∈R|E[exp(tXn)]− E[exp(tYn)]| ≤ f (n), for some f (n) = o(1),

do we have

supa∈R|PrXn ≤ a − PrYn ≤ a| ≤ g(n), for some desirable g(n)?

Esséen smoothing lemma? Stein’s method?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 20 / 22

Page 65: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Wrap-Up and Future Work

For parallel Gaussian channels with feedback,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Careful quantization of power allocation vector into power types

Bounding main information spectrum term using Curtiss’ theoremFor third-order, need speed of convergence, i.e.,

supt∈R|E[exp(tXn)]− E[exp(tYn)]| ≤ f (n), for some f (n) = o(1),

do we have

supa∈R|PrXn ≤ a − PrYn ≤ a| ≤ g(n), for some desirable g(n)?

Esséen smoothing lemma? Stein’s method?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 20 / 22

Page 66: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Wrap-Up and Future Work

For parallel Gaussian channels with feedback,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Careful quantization of power allocation vector into power typesBounding main information spectrum term using Curtiss’ theorem

For third-order, need speed of convergence, i.e.,

supt∈R|E[exp(tXn)]− E[exp(tYn)]| ≤ f (n), for some f (n) = o(1),

do we have

supa∈R|PrXn ≤ a − PrYn ≤ a| ≤ g(n), for some desirable g(n)?

Esséen smoothing lemma? Stein’s method?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 20 / 22

Page 67: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Wrap-Up and Future Work

For parallel Gaussian channels with feedback,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Careful quantization of power allocation vector into power typesBounding main information spectrum term using Curtiss’ theoremFor third-order, need speed of convergence, i.e.,

supt∈R|E[exp(tXn)]− E[exp(tYn)]| ≤ f (n), for some f (n) = o(1),

do we have

supa∈R|PrXn ≤ a − PrYn ≤ a| ≤ g(n), for some desirable g(n)?

Esséen smoothing lemma? Stein’s method?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 20 / 22

Page 68: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Wrap-Up and Future Work

For parallel Gaussian channels with feedback,

log M∗FB(n, ε,P) = nC(P∗) +√

nV(P∗)Φ−1(ε) + o(√

n).

Careful quantization of power allocation vector into power typesBounding main information spectrum term using Curtiss’ theoremFor third-order, need speed of convergence, i.e.,

supt∈R|E[exp(tXn)]− E[exp(tYn)]| ≤ f (n), for some f (n) = o(1),

do we have

supa∈R|PrXn ≤ a − PrYn ≤ a| ≤ g(n), for some desirable g(n)?

Esséen smoothing lemma? Stein’s method?

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 20 / 22

Page 69: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Full paper

Oct 2017 issue of IEEE Transactions on Information Theory

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 21 / 22

Page 70: A Tight Upper Bound on the Second-Order ... - ece.nus.edu.sg2017 Information Theory Workshop, Kaohsiung, Taiwan Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017

Advertisement: Postdoc Positions in IT and ML at NUS

Vincent Tan (NUS) Second-Order Parallel AWGN with Feedback ITW 2017 22 / 22