/32

Analog-to-Digital CompressionOral PhD Exam

Alon Kipnis

Fundamental performance limits of

Advisor: Andrea Goldsmith

1

/32

Outline

Motivation — Factors affecting analog-to-digital conversion

Main problem — Combined problem sampling and lossy compression

Corollary — Optimal sampling under compression constraints

Summary — Toward a unified spectral theory of analog signal processing and lossy compression

2

samplinganalog

quantization (lossy compression)

0100100110010010000100101010010001…

digital

/32

Motivation

0100100110010010000100001000100111…

information loss

A/D conversion

Challenges: 1) measure 2) minimize

The analog-to-digital (A/D) conversion problem:

3

/32

Motivation: measuring information lossdistortion

A/D parameters

Minimal distortion in A/D:

4

analog

010010011001001000010010101001

digitalreconstruction

analog

quantization (lossy compression)sampling

/32

Background: Lossy Compression

5

sampling quantization (lossy compression) digitalanalog

...

0 . . . 00

0 . . . 01

1 . . . 11

RT

...T0

X(t) Rbitrate:

[bits/sec]

The Source Coding Theorem [Shannon ‘48]:

D(R)Shannon’s

distortion-rate function

=

Theoretic lower bound for distortion in A/DIgnores effect of sampling

= optimization over probability distributions

reconstructionanalog

Enc

/32

fs

The Sampling Theorem [Whittaker, Kotelinkov, Shannon]:

distortion

sampling rate fs

Background: The Sampling Theorem

tX(t)

fs > fNyq , 2fB

tsinc(t)

⇤=

6

fNyq = 2fB

Y [n] = X(t/fs)

Ignores effect of quantization

sampling quantization (lossy compression) digitalanalog

Shannon’s distortion-rate

functionD(R)

/32

Combined sampling and lossy compression

7

sampling optimal lossy compression digitalanalog

D(fs , R) ?=

Minimal distortion under sampling and lossy compression

distortion

sampling rate fs

unlimited

bitrate Shannon’s distortion-rate

functionD(R)

unlimitedsampling rate

fNyq

/32

Sampling under Bitrate Constraints

Can we attain D(R) by sampling below Nyquist ?

The Sampling Theorem [Whittaker, Kotelinkov, Shannon]

tX(t) Y [n] = X(t/fs)

fs > fNyq , 2fB

tsinc(t)

⇤=

8

“we are not interested in exact transmission when we have a continuous [amplitude] source, but only in transmission to within a given tolerance” [Shannon ’48]

D(fs , R) ?=d

istortion

sampling rate fs

D(R)

fNyq = 2fB

/32

Motivation — Summary

2) Can we attain D(R) by sampling below Nyquist ?

9

distortion

sampling rate fs

fNyq = 2fB

D(R)

D(fs , R) ?=

?

1) What is the minimal distortion in sampling and lossy compression?

unlimited

bitrate

unlimitedsampling rate

/32

Combined Sampling and Source Coding

, infenc�dec,T

1

T

Z T

0E⇣X(t)� bX(t)

⌘2dtD(fs, R)

Assumptions:

is zero mean Gaussian stationary with PSD X(t) SX(f)SX(f)

f is unimodal SX(f)

Pointwise uniform sampling Y [n] = X(n/fs)

10

sampling lossy compression reconstruction

Enc Decfs Y [·]

X(t) bX(t)R

/32

Special case I: Gaussian Distortion Rate Function

Enc DecY [·]

fs > fNyq

fs

) = D(R)D(fs, R)

[Pinsker ’54]D✓(R) =

Z 1

�1min {SX(f), ✓} df

R✓ =

1

2

Z 1

�1log

+[SX(f)/✓] df

✓

SX(f)

f

✓

SX(f)

f

R

D(R)

, WF(SX)

(

(water-filling)

X(t) bX(t)

11

R

/32

Special case II: MMSE in sub-Nyquist Sampling

Y [·]Enc

Rfs

R ! 1 ) mmse(X|Y )=D(fs, R) mmse(fs)=

MMSE in sub-Nyquist sampling [Chan & Donaldson ‘71, Matthews ’00]X

k2ZSX(f � fsk) eSX|Y (f) =

Pk S

2X(f � fsk)P

k SX(f � fsk)

fsf

SX(f)

SX(f�f s)S

X (f+fs )

bX(t)DecX(t)

12

/32

Combined Sampling and Source Coding

eSX|Y (f)

f

fs

✓

Distortion dueto sampling

Distortion dueto bitrate constraint

Theorem*[K., Goldsmith, Eldar, Weissman ‘13]

D(fs, R) mmse(fs)= + WF⇣eSX|Y

⌘

(*) A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, ‘Rate-distortion function of sub-Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013 13

Enc Decfs Y [·] R

X(t) bX(t)

/32

Example: Uniform PSD

D(fs , R)

f

SX(f)

fB

distortion

fs

D(R)

D(fs, R) vs fs (R = 1)

mmse(f

s )fNyq = 2fB

14

/32

Achievability SchemeEnc Dec

fs Y [·]

D(fs, R) = +

Y [·]estimatorE [X(t)|Y [·]]

eX(·)

Enc

Enc

Enc

mmse(fs) WF⇣eSX|Y

⌘

orthogonalizing transformation

eX�2 [·]

eX�k [·]

eX�1 [·]

...*

15

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The distortion rate function of cyclostationary Gaussian processes’, (under review) 2016

RX(t) bX(t)

X

i

Ri R

R1

R2

Rk

/32

Pre-Sampling OperationEnc Dec

fsH(f)

eSX|Y (f)

fs✓

without pre-sampling filter

Linear pre-processing can reduce distortion

fs

✓

with pre-sampling filter

eSX|Y (f)

fs

D(R)

distortion

16

bX(t)X(t) R

H(f) ⌘

1

H(f)

/32

Optimal pre-Sampling FilterTheorem* [K., Goldsmith, Eldar, Weissman ’14]The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

(*) A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

SX(f)

fs

✓

H?(f)SX (f)

fs

✓

H?(f)

no aliasing

D?(fs, R) = mmse?(fs) + WF⇣|H?|2 SX

⌘

17

/32

Optimal pre-Sampling Filter

(*) A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

Theorem* [K., Goldsmith, Eldar, Weissman ’14]The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

fs

flow-pass is optimal

fsfsfs

f

maximal aliasing-free set is optimal

18

/32

Why anti-aliasing is optimal ?X1 ⇠ N

�0,�2

1

�X2 ⇠ N

�0,�2

2

�

X1 X2+=Y h1 h2

fsfsfs

f

1(�1 > �2)

1(�1 < �2)

h1

h2

=

=

⇤⇤Answer:

19

h1 h2

argmin ?=Question: {mmse(X1|Y ) +mmse(X2|Y )}

mmse(Xi|Y ) = E (Xi � E[Xi|Y ])2 fs

/32

Critical Sub-Nyquist Sampling RateD?(R, fs) vs fs

D(R)

fNyqfR

mmse(fs)

✓fs

✓

fs

✓

fs

fs

distortion

(R is fixed)

Sub-Nyquist sampling achieves optimal distortion-rate performance

D?(fs, R) = D(R) fs � fR

SX(f)

20

/32

Critical Sub-Nyquist Sampling RateTheorem* [K., Goldsmith, Eldar ’15]

D?(fs, R) = D(R) fs � fR

✓fR

(+) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Gaussian distortion-rate function under sub-Nyquist nonuniform sampling’, Allerton 2014

Extends Kotelnikov-Whittaker-Shannon sampling theorem:

Incorporates lossy compression

Valid when input signal is not band limited

Alignment of degrees of freedom

Holds under non-uniform sampling +

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Sub-Nyquist sampling achieves optimal rate-distortion’, Information Theory Workshop (ITW), 2015

21

/32R

1

fRfNyq

Critical Sub-Nyquist Sampling Rate

critical sub-sampling ratio vs R

*

22

/32

SummaryTransforming analog signals to bits involves sampling and lossy compression

Parts of the signal removed due to lossy compression can be removed at the sampling stage

23

Closed-form expression for the minimal distortion as a function of the sampling rate and bitrate

• Sub-Nyquist sampling is optimal under bitrate constraint

/32

Future Work IDegrees of freedom alignment in other sampling models ?

24

Enc Dec{0, 1}nR

bX

Example: compressed sensing

X Ysampler2 Rn

/32

Noisy Input Signal

(*) A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, ‘Rate-distortion function of sub-Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013 25

Enc Dec

fs

sampler

Y [·]X(t) bX(t)H(f)+

⌘(t)

R

Theorem*[K. Goldsmith, Weissman, Eldar ’13]⇤

fs✓

eSX|Y (f)P

k S2X (f � fsk) |H(f � fsk)|2P

k (SX(f � fsk) + S⌘ (f � fsk)) |H(f � fsk)|2=

sampling quantization (lossy compression) digitalanalog noise

/32

Toward a Unified Spectral Theory of Processing Time Series

eSX|Y (f)

fs✓

H(f)+

⌘(t)

Enc Dec

26

Lossy compression

SamplingLinear filtering

Does not incorporate time-flow

/32

Future Work IIIncorporating time-flow and lossy compression

[Kolmogorov ’56]: “Since a function with a bounded spectrum is always singular in the sense of my work and the observation of such a function is not related … to the stationary flow of new information, then the sense of this kind of argumentation does not remain completely clear”

✓

SX(f)

f

27

Example: minimal distortion in causal estimation under bitrate constraint

X(t)t

past future

/32

References

[3] A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The distortion-rate function of sampled Wiener processes’, (under review) 2016

[2] A. Kipnis, Y. C. Eldar and A. J. Goldsmith, ‘Fundamental distortion limits of analog-to-digital compression’, (under review) 2015

[1] A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

• Conference version: A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Sub-Nyquist sampling achieves optimal rate-distortion’, Information Theory Workshop (ITW), 2015

• Conference version: A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Gaussian distortion-rate function under sub-Nyquist nonuniform sampling’, Allerton 2015

• Conference version: A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, ‘Rate-distortion function of sub-Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013

Analog-to-digital compression:

• Conference version: A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Information rates of sampled Wiener processes’, ISIT 2016

• Conference version: A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Optimal trade-off between sampling rate and quantization precision in Sigma-Delta A/D conversion’, SampTA 2015

• Conference version: A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Optimal Trade-off Between Sampling Rate and Quantization Precision in A/D conversion’, Allerton 2015

28

/32

References (cont.)Lossy source coding:

[4] A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The distortion rate function of cyclostationary Gaussian processes’, (under review) 2016

[5] A. Kipnis, S. Rini and A. J. Goldsmith, ‘Multiterminal compress-and-estimate rate-distortion’, (in progress)

• Conference version: A. Kipnis, and A. J. Goldsmith, ‘Distortion rate function of cyclo-stationary Gaussian processes’, ISIT 2014

• Conference version: A. Kipnis, S. Rini and A. J. Goldsmith, ‘The indirect rate-distortion function of a binary i.i.d source’, ITW 2015

• Conference version: A. Kipnis, S. Rini and A. J. Goldsmith, ‘Multiterminal compress-and-estimate source coding’, ISIT 2016

29

/32

AcknowledgmentsCommittee members:

Emanuel CandesAbbas El-GammalJohn Duchi

Yonina Eldar Tsachy Weissman

WSLers and ISLers StefanoYuxinMilindMainakAlexandrosNimaMahnooshYonathanNarimanJiantaoKartikIdoiaMiguel

30

/32

Acknowledgments

31

/32

The End!

eSX|Y (f)

fs✓

32

D?(R, fs) vs fs

mmse(fs)fs

distortion

D(R)fNyqfR

/32

Appendix I: Sampling in Source Coding

Sampling in practice:

[Berger ’68]: Joint typicality with respect to continuous-time waveform

[Yaglom-Pinsker ‘57, Gallager ’68]: Karhunen–Loève transform

[Shannon ’49]: Degrees of freedom = time X bandwidth

[Berger ’71, Neuhoff & Pradhan 2013]: Analog distortion-rate function by discrete-time time approximations

X(t)

fs

Sampler

Y [n]

continuous-time discrete-time

LTI

Constrained by hardware

Constrained in bandwidth

Modelling constraint33

/32

digital

Relation to Remote Source Coding

Remote source coding [Dubroshin&Tsybakov ’62, Wolf&Ziv ’70]:

Enc DecX(0 : T )

informationsource reconstruction

bX(0 : T )fs

Sampler

Y [·] M 2 {0, 1}bTRc

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The Distortion Rate Function of Cyclostationary Gaussian Processes’, (under review) 2016

cyclo-stationary

⇤

enc-d

ec

D(fs,

R)

enc-dec

D(R)

X(t) sampling Y [n]

bX(t)

enc-dec

D eX(R)

beX(t)

kx� bxk2T

mmse(fs)

estimate eX(t) = E [X(t)|Y [·]]

Decomposition: = +mmse(fs) D eX(R)D(fs, R)

34

/32

Appendix II: Brownian Motion to Bits (analog-to-digital compression for the Wiener process)

is zero-mean GaussianX(t)

EX(t)X(s) = min{t, s}

DecEncfs M 2 {0, 1}bTRc

YT [n] = XT (n/fs)

X(0 : T ) bX(0 : T )

0 T

X(t)

S(t)Z t

0

dS(t)

S(t)= µt+ �X(t)

Model for assets pricing:

35

/32

digital

Analog-to-Digital Compression

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The Distortion-Rate Function of Sampled Wiener Processes’, (under review) 2016

Theorem*“Estimate-and-compress” is optimal for the Wiener process

enc-d

ec

D(fs,

R)

enc-dec

D(R)

X(t) sampling Y [n]

bX(t)

= +mmse(fs) D eX(R)D(fs, R)

enc-dec

D eX(R)

beX(t)

eX(t)mmse(fs)

estimate

D(R)

D(R)[Berger ’70]: =2

⇡2 ln 2R�1

kx� bxk2T enc-dec

DY (R̄)

bY [n]estimate

36

/32

Distortion-Bitrate-Sampling Function

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The Distortion-Rate Function of Sampled Wiener Processes’, (under review) 2016

Theorem*

D(fs, R) = mmse(fs) +1

fs

Z 1

0min

n

eS(�), ✓o

d�

R(✓) =fs2

Z 1

0log

+heS(�)/✓

id�

eS(�) = 1

4 sin2(⇡�/2)� 1

6mmse(fs) =1

6fs

✓

�1

37

/32

Distortion vs Bitrate38

fs=1 fixed

R [bits/sec]

distortion

1

6fs

mmse(fs)

D(fs , R)

D(R)

/32

APP III: Nonuniform Samplingt1 t2 t3 t4 t5 t6 t7· · ·

· · ·⇤Sampler

X(·) Y [n] = X(tn)

tn 2 ⇤

h(t, ⌧)

Theorem*D?

�d�(⇤), R

� Dh,⇤(R)

(*) A. Kipnis, Y. C. Eldar and A. J. Goldsmith, ‘Fundamental Distortion Limits of Analog-to-Digital Compression’, (under review) 2015

d�(⇤) = limr!1

infu2R

|⇤ \ [u, u+ r)|r

is the lower Beurling density of ⇤

Nonuniform sampling cannot improve over uniform

[Landau ’67]: necessary and sufficient condition for zero interpolation error:

d�(⇤) � µ(suppSX)

39

/32

digitalanalog

Appendix IV: PCM Under Bitrate Constraint

X(t)anti-aliasing

filter

XLPF (t)

fsY [n] = XLPF (n/fs)

R̄-bit quantizer

YQ[n]

estimator

bX(t)

DPCM (fs, R) =1

T

Z T

0E⇣X(t)� bX(t)

⌘2dt

(*) A. Kipnis, Y. C. Eldar and A. J. Goldsmith, ‘Fundamental Distortion Limits of Analog-to-Digital Compression’, (under review) 2015

Theorem* (stationary input, linear estimation)

DPCM (fs, R) = mmse(X|Y ) +DQ(R̄, fs)

DQ(R̄, fs) = c02fBfs

�22�2R̄mmse(X|Y ) = �2 �Z fs

2

� fs2

SX(f)df

R = R̄fs

40

/32

PCM Under Bitrate Constraint

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

DPCM (fs, R) = mmse(X|Y ) +DQ(R̄, fs)

DQ(R̄, fs) = c02fBfs

�22�2R̄mmse(X|Y ) = �2 �Z fs

2

� fs2

SX(f)df

Optimal sampling rate in PCM is smaller than Nyquist (!)

(R is fixed)

DPCM (fs, R)

fsfNyq1

�2

distortion

41