     # Oral PhD Exam Alon Kipnis - Stanford University kipnisal/Slides/ آ  /32 Analog-to-Digital

• View
0

0

Embed Size (px)

### Text of Oral PhD Exam Alon Kipnis - Stanford University kipnisal/Slides/ آ  /32 Analog-to-Digital

• /32

Analog-to-Digital Compression Oral PhD Exam

Alon Kipnis

Fundamental performance limits of

1

• /32

Outline

Motivation — Factors affecting analog-to-digital conversion

Main problem — Combined problem sampling and lossy compression

Corollary — Optimal sampling under compression constraints

Summary — Toward a unified spectral theory of analog signal processing and lossy compression

2

sampling analog

quantization (lossy compression)

010010011001001000 0100101010010001…

digital

• /32

Motivation

010010011001 001000010000 1000100111…

information loss

A/D conversion

Challenges: 1) measure 2) minimize

The analog-to-digital (A/D) conversion problem:

3

• /32

Motivation: measuring information loss d i s t o r t i o n

A/D parameters

Minimal distortion in A/D:

4

analog

010010011001001 000010010101001

digital reconstruction

analog

quantization (lossy compression)sampling

• /32

Background: Lossy Compression

5

sampling quantization (lossy compression) digitalanalog

...

0 . . . 00

0 . . . 01

1 . . . 11

RT

...T0

X(t) R bitrate:

[bits/sec]

The Source Coding Theorem [Shannon ‘48]:

D(R) Shannon’s

distortion-rate function

=

Theoretic lower bound for distortion in A/D Ignores effect of sampling

= optimization over probability distributions

reconstruction analog

Enc

• /32

fs

The Sampling Theorem [Whittaker, Kotelinkov, Shannon]:

d i s t o r t i o n

sampling rate fs

Background: The Sampling Theorem

t X(t)

fs > fNyq , 2fB

t sinc(t)

⇤=

6

fNyq = 2fB

Y [n] = X(t/fs)

Ignores effect of quantization

sampling quantization (lossy compression) digitalanalog

Shannon’s distortion-rate

function D(R)

• /32

Combined sampling and lossy compression

7

sampling optimal lossy compression digitalanalog

D (f s , R) ?=

Minimal distortion under sampling and lossy compression

d i s t o r t i o n

sampling rate fs

unlim ited

bitrate Shannon’s distortion-rate

function D(R)

unlimited sampling rate

fNyq

• /32

Sampling under Bitrate Constraints

Can we attain D(R) by sampling below Nyquist ?

The Sampling Theorem [Whittaker, Kotelinkov, Shannon]

t X(t) Y [n] = X(t/fs)

fs > fNyq , 2fB

t sinc(t)

⇤=

8

“we are not interested in exact transmission when we have a continuous [amplitude] source, but only in transmission to within a given tolerance” [Shannon ’48]

D (f s , R) ?=d

i s t o r t i o n

sampling rate fs

D(R)

fNyq = 2fB

• /32

Motivation — Summary

2) Can we attain D(R) by sampling below Nyquist ?

9

d i s t o r t i o n

sampling rate fs

fNyq = 2fB

D(R)

D (f s , R) ?=

?

1) What is the minimal distortion in sampling and lossy compression?

unlim ited

bitrate

unlimited sampling rate

• /32

Combined Sampling and Source Coding

, inf enc�dec,T

1

T

Z T

0 E ⇣ X(t)� bX(t)

⌘2 dtD(fs, R)

Assumptions:

is zero mean Gaussian stationary with PSD X(t) SX(f) SX(f)

f is unimodal SX(f)

Pointwise uniform sampling Y [n] = X(n/fs) 10

sampling lossy compression reconstruction

Enc Dec fs Y [·]

X(t) bX(t) R

• /32

Special case I: Gaussian Distortion Rate Function

Enc Dec Y [·]

fs > fNyq

fs

) = D(R)D(fs, R)

[Pinsker ’54] D✓(R) =

Z 1

�1 min {SX(f), ✓} df

R✓ = 1

2

Z 1

�1 log

+ [SX(f)/✓] df

SX(f)

f

SX(f)

f

R

D(R)

, WF(SX)

(

(water-filling)

X(t) bX(t)

11

R

• /32

Special case II: MMSE in sub-Nyquist Sampling

Y [·] Enc

Rfs

R ! 1 ) mmse(X|Y )=D(fs, R) mmse(fs)= MMSE in sub-Nyquist sampling [Chan & Donaldson ‘71, Matthews ’00]

X

k2Z SX(f � fsk) eSX|Y (f) =

P k S

2 X(f � fsk)P

k SX(f � fsk)

fs f

SX(f)

SX (f � f s )S

X (f + f s )

bX(t)DecX(t)

12

• /32

Combined Sampling and Source Coding

eSX|Y (f)

f

fs

Distortion due to sampling

Distortion due to bitrate constraint

Theorem*[K., Goldsmith, Eldar, Weissman ‘13]

D(fs, R) mmse(fs)= + WF ⇣ eSX|Y

(*) A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, ‘Rate-distortion function of sub- Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013 13

Enc Dec fs Y [·] R

X(t) bX(t)

• /32

Example: Uniform PSD

D (f s , R)

f

SX(f)

fB

d i s t o r t i o n

fs

D(R)

D(fs, R) vs fs (R = 1)

m m se(f

s ) fNyq = 2fB

14

• /32

Achievability Scheme Enc Dec

fs Y [·]

D(fs, R) = +

Y [·] estimator E [X(t)|Y [·]]

eX(·)

Enc

Enc

Enc

mmse(fs) WF ⇣ eSX|Y

orthogonalizing transformation

eX�2 [·]

eX�k [·]

eX�1 [·]

...*

15

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The distortion rate function of cyclostationary Gaussian processes’, (under review) 2016

R X(t) bX(t)

X

i

Ri  R

R1

R2

Rk

• /32

Pre-Sampling Operation Enc Dec

fs H(f)

eSX|Y (f)

fs ✓

without pre-sampling filter

Linear pre-processing can reduce distortion

fs

with pre-sampling filter

eSX|Y (f)

fs

D(R)

d i s t o r t i o n

16

bX(t)X(t) R

H (f) ⌘

1

H (f)

• /32

Optimal pre-Sampling Filter Theorem* [K., Goldsmith, Eldar, Weissman ’14] The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

(*) A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

SX(f)

fs

H?(f) S X (f)

fs

H?(f)

no aliasing

D?(fs, R) = mmse ?(fs) + WF

⇣ |H?|2 SX

17

• /32

Optimal pre-Sampling Filter

(*) A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

Theorem* [K., Goldsmith, Eldar, Weissman ’14] The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

fs

f low-pass is optimal

fsfsfs

f

maximal aliasing-free set is optimal

18

• /32

Why anti-aliasing is optimal ? X1 ⇠ N

� 0,�21

� X2 ⇠ N

� 0,�22

X1 X2+=Y h1 h2

fsfsfs

f

1(�1 > �2)

1(�1 < �2)

h1 h2

=

=

19

h1 h2 argmin ?=Question: {mmse(X1|Y ) +mmse(X2|Y )}

mmse(Xi|Y ) = E (Xi � E[Xi|Y ])2 fs

• /32

Critical Sub-Nyquist Sampling Rate D?(R, fs) vs fs

D(R)

fNyqfR

mmse(fs)

✓ fs

fs

fs

fs

d i s t o r t i o n

(R is fixed)

Sub-Nyquist sampling achieves optimal distortion-rate performance

D?(fs, R) = D(R) fs � fR

SX(f)

20

• /32

Critical Sub-Nyquist Sampling Rate Theorem* [K., Goldsmith, Eldar ’15]

D?(fs, R) = D(R) fs � fR

✓ fR

(+) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Gaussian distortion-rate function under sub- Nyquist nonuniform sampling’, Allerton 2014

Extends Kotelnikov-Whittaker-Shannon sampling theorem:

Incorporates lossy compression

Valid when input signal is not band limited

Alignment of degrees of freedom

Holds under non-uniform sampling +

(*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Sub-Nyquist sampling achieves optimal rate-distortion’, Information Theory Workshop (ITW), 2015

21

• /32 R

1

fR fNyq

Critical Sub-Nyquist Sampling Rate

critical sub-sampling ratio vs R

*

22

• /32

Summary Transforming analog signals to bits involves sampling and lossy compression

Parts of the signal removed due to lossy compression can be removed at the sampling stage

23

Recommended ##### Arguing For/Over the Dignity of Difference Alon Goshen ...· Alon Goshen-Gottstein The Elijah Interfaith
Documents ##### ALON BLUE SQUARE ISRAEL LTD. (NYSE:BSI) ANNOUNCES THE ... Alon Blue Square Israel Ltd. Limor Ganot,
Documents ##### Stochastic Integration for non-Martingales Stationary ... kipnisal/Slides/thesis_ آ
Documents ##### FUNDAMENTAL PERFORMANCE LIMITS OF kipnisal/Papers/fundamental-performance-limits.pdf loving home they
Documents ##### Rough walks in random peledron/Horowitz_seminar/Orenshtein_slides.pdfآ  Kipnis-Varadhan theory in rough
Documents Documents