Oral PhD Exam Alon Kipnis - Stanford University kipnisal/Slides/ آ  /32 Analog-to-Digital

  • View
    0

  • Download
    0

Embed Size (px)

Text of Oral PhD Exam Alon Kipnis - Stanford University kipnisal/Slides/ آ  /32 Analog-to-Digital

  • /32

    Analog-to-Digital Compression Oral PhD Exam

    Alon Kipnis

    Fundamental performance limits of

    Advisor: Andrea Goldsmith

    1

  • /32

    Outline

    Motivation — Factors affecting analog-to-digital conversion

    Main problem — Combined problem sampling and lossy compression

    Corollary — Optimal sampling under compression constraints

    Summary — Toward a unified spectral theory of analog signal processing and lossy compression

    2

    sampling analog

    quantization (lossy compression)

    010010011001001000 0100101010010001…

    digital

  • /32

    Motivation

    010010011001 001000010000 1000100111…

    information loss

    A/D conversion

    Challenges: 1) measure 2) minimize

    The analog-to-digital (A/D) conversion problem:

    3

  • /32

    Motivation: measuring information loss d i s t o r t i o n

    A/D parameters

    Minimal distortion in A/D:

    4

    analog

    010010011001001 000010010101001

    digital reconstruction

    analog

    quantization (lossy compression)sampling

  • /32

    Background: Lossy Compression

    5

    sampling quantization (lossy compression) digitalanalog

    ...

    0 . . . 00

    0 . . . 01

    1 . . . 11

    RT

    ...T0

    X(t) R bitrate:

    [bits/sec]

    The Source Coding Theorem [Shannon ‘48]:

    D(R) Shannon’s

    distortion-rate function

    =

    Theoretic lower bound for distortion in A/D Ignores effect of sampling

    = optimization over probability distributions

    reconstruction analog

    Enc

  • /32

    fs

    The Sampling Theorem [Whittaker, Kotelinkov, Shannon]:

    d i s t o r t i o n

    sampling rate fs

    Background: The Sampling Theorem

    t X(t)

    fs > fNyq , 2fB

    t sinc(t)

    ⇤=

    6

    fNyq = 2fB

    Y [n] = X(t/fs)

    Ignores effect of quantization

    sampling quantization (lossy compression) digitalanalog

    Shannon’s distortion-rate

    function D(R)

  • /32

    Combined sampling and lossy compression

    7

    sampling optimal lossy compression digitalanalog

    D (f s , R) ?=

    Minimal distortion under sampling and lossy compression

    d i s t o r t i o n

    sampling rate fs

    unlim ited

    bitrate Shannon’s distortion-rate

    function D(R)

    unlimited sampling rate

    fNyq

  • /32

    Sampling under Bitrate Constraints

    Can we attain D(R) by sampling below Nyquist ?

    The Sampling Theorem [Whittaker, Kotelinkov, Shannon]

    t X(t) Y [n] = X(t/fs)

    fs > fNyq , 2fB

    t sinc(t)

    ⇤=

    8

    “we are not interested in exact transmission when we have a continuous [amplitude] source, but only in transmission to within a given tolerance” [Shannon ’48]

    D (f s , R) ?=d

    i s t o r t i o n

    sampling rate fs

    D(R)

    fNyq = 2fB

  • /32

    Motivation — Summary

    2) Can we attain D(R) by sampling below Nyquist ?

    9

    d i s t o r t i o n

    sampling rate fs

    fNyq = 2fB

    D(R)

    D (f s , R) ?=

    ?

    1) What is the minimal distortion in sampling and lossy compression?

    unlim ited

    bitrate

    unlimited sampling rate

  • /32

    Combined Sampling and Source Coding

    , inf enc�dec,T

    1

    T

    Z T

    0 E ⇣ X(t)� bX(t)

    ⌘2 dtD(fs, R)

    Assumptions:

    is zero mean Gaussian stationary with PSD X(t) SX(f) SX(f)

    f is unimodal SX(f)

    Pointwise uniform sampling Y [n] = X(n/fs) 10

    sampling lossy compression reconstruction

    Enc Dec fs Y [·]

    X(t) bX(t) R

  • /32

    Special case I: Gaussian Distortion Rate Function

    Enc Dec Y [·]

    fs > fNyq

    fs

    ) = D(R)D(fs, R)

    [Pinsker ’54] D✓(R) =

    Z 1

    �1 min {SX(f), ✓} df

    R✓ = 1

    2

    Z 1

    �1 log

    + [SX(f)/✓] df

    SX(f)

    f

    SX(f)

    f

    R

    D(R)

    , WF(SX)

    (

    (water-filling)

    X(t) bX(t)

    11

    R

  • /32

    Special case II: MMSE in sub-Nyquist Sampling

    Y [·] Enc

    Rfs

    R ! 1 ) mmse(X|Y )=D(fs, R) mmse(fs)= MMSE in sub-Nyquist sampling [Chan & Donaldson ‘71, Matthews ’00]

    X

    k2Z SX(f � fsk) eSX|Y (f) =

    P k S

    2 X(f � fsk)P

    k SX(f � fsk)

    fs f

    SX(f)

    SX (f � f s )S

    X (f + f s )

    bX(t)DecX(t)

    12

  • /32

    Combined Sampling and Source Coding

    eSX|Y (f)

    f

    fs

    Distortion due to sampling

    Distortion due to bitrate constraint

    Theorem*[K., Goldsmith, Eldar, Weissman ‘13]

    D(fs, R) mmse(fs)= + WF ⇣ eSX|Y

    (*) A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, ‘Rate-distortion function of sub- Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013 13

    Enc Dec fs Y [·] R

    X(t) bX(t)

  • /32

    Example: Uniform PSD

    D (f s , R)

    f

    SX(f)

    fB

    d i s t o r t i o n

    fs

    D(R)

    D(fs, R) vs fs (R = 1)

    m m se(f

    s ) fNyq = 2fB

    14

  • /32

    Achievability Scheme Enc Dec

    fs Y [·]

    D(fs, R) = +

    Y [·] estimator E [X(t)|Y [·]]

    eX(·)

    Enc

    Enc

    Enc

    mmse(fs) WF ⇣ eSX|Y

    orthogonalizing transformation

    eX�2 [·]

    eX�k [·]

    eX�1 [·]

    ...*

    15

    (*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘The distortion rate function of cyclostationary Gaussian processes’, (under review) 2016

    R X(t) bX(t)

    X

    i

    Ri  R

    R1

    R2

    Rk

  • /32

    Pre-Sampling Operation Enc Dec

    fs H(f)

    eSX|Y (f)

    fs ✓

    without pre-sampling filter

    Linear pre-processing can reduce distortion

    fs

    with pre-sampling filter

    eSX|Y (f)

    fs

    D(R)

    d i s t o r t i o n

    16

    bX(t)X(t) R

    H (f) ⌘

    1

    H (f)

  • /32

    Optimal pre-Sampling Filter Theorem* [K., Goldsmith, Eldar, Weissman ’14] The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

    (*) A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

    SX(f)

    fs

    H?(f) S X (f)

    fs

    H?(f)

    no aliasing

    D?(fs, R) = mmse ?(fs) + WF

    ⇣ |H?|2 SX

    17

  • /32

    Optimal pre-Sampling Filter

    (*) A. Kipnis, A. J. Goldsmith, Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

    Theorem* [K., Goldsmith, Eldar, Weissman ’14] The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

    fs

    f low-pass is optimal

    fsfsfs

    f

    maximal aliasing-free set is optimal

    18

  • /32

    Why anti-aliasing is optimal ? X1 ⇠ N

    � 0,�21

    � X2 ⇠ N

    � 0,�22

    X1 X2+=Y h1 h2

    fsfsfs

    f

    1(�1 > �2)

    1(�1 < �2)

    h1 h2

    =

    =

    ⇤ ⇤Answer:

    19

    h1 h2 argmin ?=Question: {mmse(X1|Y ) +mmse(X2|Y )}

    mmse(Xi|Y ) = E (Xi � E[Xi|Y ])2 fs

  • /32

    Critical Sub-Nyquist Sampling Rate D?(R, fs) vs fs

    D(R)

    fNyqfR

    mmse(fs)

    ✓ fs

    fs

    fs

    fs

    d i s t o r t i o n

    (R is fixed)

    Sub-Nyquist sampling achieves optimal distortion-rate performance

    D?(fs, R) = D(R) fs � fR

    SX(f)

    20

  • /32

    Critical Sub-Nyquist Sampling Rate Theorem* [K., Goldsmith, Eldar ’15]

    D?(fs, R) = D(R) fs � fR

    ✓ fR

    (+) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Gaussian distortion-rate function under sub- Nyquist nonuniform sampling’, Allerton 2014

    Extends Kotelnikov-Whittaker-Shannon sampling theorem:

    Incorporates lossy compression

    Valid when input signal is not band limited

    Alignment of degrees of freedom

    Holds under non-uniform sampling +

    (*) A. Kipnis, A. J. Goldsmith and Y. C. Eldar, ‘Sub-Nyquist sampling achieves optimal rate-distortion’, Information Theory Workshop (ITW), 2015

    21

  • /32 R

    1

    fR fNyq

    Critical Sub-Nyquist Sampling Rate

    critical sub-sampling ratio vs R

    *

    22

  • /32

    Summary Transforming analog signals to bits involves sampling and lossy compression

    Parts of the signal removed due to lossy compression can be removed at the sampling stage

    23