Foundations of Signal Processingassets.cambridge.org/97811070/38608/frontmatter/... · “Foundations of Signal Processing by Vetterli, Kovaceviˇ c, and Goyal, is a pleasure to read

Foundations of Signal Processing

This comprehensive and engaging textbook introduces the basic principles and tech-niques of signal processing, from the fundamental ideas of signals and systems theoryto real-world applications.

• Introduces students to the powerful foundations of modern signal processing,including the basic geometry of Hilbert space, the mathematics of Fourier transforms,and essentials of sampling, interpolation, approximation, and compression.

• Discusses issues in real-world use of these tools such as effects of truncation andquantization, limitations on localization, and computational costs.

• Includes over 160 homework problems and over 220 worked examples, specificallydesigned to test and expand students’ understanding of the fundamentals of signalprocessing.

• Accompanied by extensive online materials designed to aid learning, includingMathematica� resources and interactive demonstrations.

MARTIN VETTERLI is a Professor of Computer and Communication Sciences at the EcolePolytechnique Federale de Lausanne, and the President of the Swiss National ScienceFoundation. He has formerly held positions at Columbia University and the Universityof California, Berkeley, and has received the IEEE Signal Processing Society TechnicalAchievement Award (2001) and Society Award (2010). He is a Fellow of the ACM,EURASIP, and the IEEE, and is a Thomson Reuters Highly Cited Researcher inEngineering.

JELENA KOVACEVIC is the David Edward Schramm Professor and Head of Electrical andComputer Engineering, and a Professor of Biomedical Engineering, at Carnegie MellonUniversity. She has been awarded the Belgrade October Prize (1986), the E. I. JuryAward (1991) from Columbia University, and the 2010 Philip L. Dowd Fellowship atCarnegie Mellon University. She is a former Editor-in-Chief of IEEE Transactions onImage Processing and a Fellow of the IEEE and EURASIP.

VIVEK K GOYAL is an Assistant Professor of Electrical and Computer Engineering at BostonUniversity, and a former Esther and Harold E. Edgerton Associate Professor of Elec-trical Engineering at the Massachusetts Institute of Technology. He has been awardedthe IEEE Signal Processing Society Magazine Award (2002) and the Eliahu Jury Award(1998) from the University of California, Berkeley, for outstanding achievement in sys-tems, communications, control, and signal processing. He is a Fellow of the IEEE.

www.cambridge.org© in this web service Cambridge University Press

Cambridge University Press978-1-107-03860-8 - Foundations of Signal ProcessingMartin Vetterli, Jelena Kovačević and Vivek K GoyalFrontmatterMore information

http://www.cambridge.org/9781107038608

http://www.cambridge.org


“Foundations of Signal Processing by Martin Vetterli, Jelena Kovacevic, and Vivek K Goyallives up to its title by providing a thorough tour of the subject matter based on selected toolsfrom real analysis which allow sufficient generality to develop the foundations of the classicalFourier methods along with modern wavelet approaches. A key distinction of the book is the useof Hilbert space ideas to provide a geometric interpretation and intuition that enhances both theclassic and modern approaches by providing a unified view of their similarities and relative meritsin the most important special cases. Many of the specific examples of signal processing consid-ered can be viewed as examples of projections onto subspaces, which yield immediate propertiesand descriptions from the underlying fundamentals. The development is both pedagogically andtheoretically sound, proceeding from the underlying mathematics through discrete-time systemsto the more complicated continuous time systems into a wonderfully general and enlighteningtreatment of sampling and interpolation operations connecting discrete and continuous time. Allof the important signal classes are considered and their basic properties and interrelations devel-oped and summarized. The book then develops important topics not ordinarily found in signalprocessing texts – the accuracy of approximations involving the truncation of series expansionsand the quantization of series coefficients, and the localization of signals in the time-frequencyplane.

The completeness of the book results in a lengthy volume of roughly 800 pages, but it is easyto navigate to extract portions of interest while saving the many byways and special topics forfuture reference. The chapter introductions are particularly good at setting the stage in a simplebut informative context and then sketching the details to come for the remainder of the chapter.Each chapter closes with a ‘Chapter at a glance’ section highlighting the primary ideas and results.The book will be a welcome addition to the library of students, practitioners, and researchers insignal processing for learning, reviewing, and referencing the broad array of tools and propertiesnow available to analyze, synthesize, and understand signal processing systems.”

Robert M. Gray, Stanford University and Boston University

“Finally a wonderful and accessible book for teaching modern signal processing to undergraduatestudents.”

Stephane Mallat, Ecole Normale Superieure

“This is a major book about a serious subject – the combination of engineering and mathematicsthat goes into modern signal processing: discrete time, continuous time, sampling, filtering, andcompression. The theory is beautiful and the applications are so important and widespread.”

Gil Strang, Massachusetts Institute of Technology

“This book (FSP) and its companion (FWSP) bring a refreshing new, and comprehensive approachto teaching the fundamentals of signal processing, from analysis and decompositions, to multi-scale representations, approximations, and many other aspects that have a tremendous impactin modern information technology. Whereas classical texts were usually written for students inelectrical or communication engineering programs, FSP and FWSP start from basic concepts inalgebra and geometry, with the benefit of being easily accessible to a much broader set of read-ers, and also help those readers develop strong abstract reasoning and intuition about signals andprocessing operators. A must-read!”

Rico Malvar, Microsoft Research

“This is a wonderful book that connects together all the elements of modern signal processing.From functional analysis and probability theory, to linear algebra and computational methods,it’s all here and seamlessly integrated, along with a summary of history and developments in thefield. A real tour-de-force, and a must-have on every signal processor’s shelf!”

Robert D. Nowak, University of Wisconsin–Madison






“Most introductory signal processing textbooks focus on classical transforms, and study howthese can be used. Instead, Foundations of Signal Processing encourages readers to think of sig-nals first. It develops a ’signal-centric’ view, one that focuses on signals, their representation andapproximation, through the introduction of signal spaces. Unlike most entry-level signal process-ing texts, this general view, which can be applied to many different signal classes, is introducedright at the beginning. From this, starting from basic concepts, and placing an emphasis on intu-ition, this book develops mathematical tools that give the readers a fresh perspective on classicalresults, while providing them with the tools to understand many state-of-the-art signal represen-tation techniques.”

Antonio Ortega, University of Southern California

“Foundations of Signal Processing by Vetterli, Kovacevic, and Goyal, is a pleasure to read. Draw-ing on the authors’ rich experience of research and teaching of signal processing and signal rep-resentations, it provides an intellectually cohesive and modern view of the subject from the geo-metric point of view of vector spaces. Emphasizing Hilbert spaces, where fine technicalities canbe relegated to backstage, this textbook strikes an excellent balance between intuition and math-ematical rigor, that will appeal to both undergraduate and graduate engineering students. The lasttwo chapters, on sampling and interpolation, and on localization and uncertainty, take full advan-tage of the machinery developed in the previous chapters to present these two very importanttopics of modern signal processing, that previously were only found in specialized monographs.The explanations of advanced topics are exceptionally lucid, exposing the reader to the ideas andthought processes behind the results and their derivation. Students will learn not only a substantialbody of knowledge and techniques, but also why things work, at a deep level, which will equipthem for independent further reading and research. I look forward to using this text in my ownteaching.”

Yoram Bresler, University of Illinois at Urbana-Champaign











Foundations of Signal Processing

MARTIN VETTERLIEcole Polytechnique Federale de Lausanne, SwitzerlandUniversity of California, Berkeley, USA

JELENA KOVACEVICCarnegie Mellon University, USA

VIVEK K GOYALBoston University, USAMassachusetts Institute of Technology, USA






University Printing House, Cambridge CB2 8BS, United Kingdom

Cambridge University Press is part of the University of Cambridge.

It furthers the University’s mission by disseminating knowledge in the pursuit ofeducation, learning and research at the highest international levels of excellence.

www.cambridge.orgInformation on this title: www.cambridge.org/9781107038608

c© M. Vetterli, J. Kovacevic & V. K. Goyal 2014

This publication is in copyright. Subject to statutory exceptionand to the provisions of relevant collective licensing agreements,no reproduction of any part may take place without the writtenpermission of Cambridge University Press.

First published 2014

Printed and bound in the United Kingdom by TJ International Ltd. Padstow Cornwall

A catalogue record for this publication is available from the British Library

ISBN 978-1-107-03860-8 Hardback

Additional resources for this publication at www.cambridge.org/vetterliand www.fourierandwavelets.org

Cambridge University Press has no responsibility for the persistence or accuracy ofURLs for external or third-party internet websites referred to in this publication,and does not guarantee that any content on such websites is, or will remain,accurate or appropriate.






To Marie-Laure, for her ∞ patience and many other qualities,Thomas and Noemie, whom I might still convince of the beauty of this material,and my parents, who gave me all the opportunities one can wish for.

— MV

To Danica and Giovanni, who make life beautiful.To my parents, who made me who I am.

— JK

To Allie, Sundeep, and my family, who encourage me unceasingly, andto the educators who made me want to be one of them.

— VKG











The cover illustration captures an experiment first described by Isaac Newton inOpticks in 1730, showing that white light can be split into its color components andthen synthesized back into white light. It is a physical implementation of a decom-position of white light into its Fourier components – the colors of the rainbow –followed by a synthesis to recover the original.











Contents

Quick reference page

Acknowledgments xxiii

Preface xxv

1 On rainbows and spectra 1

2 From Euclid to Hilbert 92.1 Introduction 102.2 Vector spaces 18

2.2.1 Definition and properties 182.2.2 Inner product 232.2.3 Norm 272.2.4 Standard spaces 30

2.3 Hilbert spaces 352.3.1 Convergence 362.3.2 Completeness 372.3.3 Linear operators 40

2.4 Approximations, projections, and decompositions 502.4.1 Projection theorem 512.4.2 Projection operators 542.4.3 Direct sums and subspace decompositions 602.4.4 Minimum mean-squared error estimation 63

2.5 Bases and frames 692.5.1 Bases and Riesz bases 692.5.2 Orthonormal bases 762.5.3 Biorthogonal pairs of bases 862.5.4 Frames 1012.5.5 Matrix representations of vectors and linear operators 109

2.6 Computational aspects 1192.6.1 Cost, complexity, and asymptotic notations 1202.6.2 Precision 1232.6.3 Conditioning 1262.6.4 Solving systems of linear equations 129

2.A Elements of analysis and topology 1352.A.1 Basic definitions 135

xi

xvii






xii Contents

2.A.2 Convergence 1362.A.3 Interchange theorems 1382.A.4 Inequalities 1392.A.5 Integration by parts 140

2.B Elements of linear algebra 1412.B.1 Basic definitions and properties 1412.B.2 Special matrices 147

2.C Elements of probability 1512.C.1 Basic definitions 1512.C.2 Standard distributions 1542.C.3 Estimation 155

2.D Basis concepts 159Chapter at a glance 161Historical remarks 162Further reading 162Exercises with solutions 163Exercises 169

3 Sequences and discrete-time systems 1813.1 Introduction 1823.2 Sequences 185

3.2.1 Infinite-length sequences 1853.2.2 Finite-length sequences 1923.2.3 Two-dimensional sequences 193

3.3 Systems 1953.3.1 Discrete-time systems and their properties 1953.3.2 Difference equations 2023.3.3 Linear shift-invariant systems 205

3.4 Discrete-time Fourier transform 2163.4.1 Definition of the DTFT 2163.4.2 Existence and convergence of the DTFT 2183.4.3 Properties of the DTFT 2213.4.4 Frequency response of filters 227

3.5 z-transform 2333.5.1 Definition of the z-transform 2343.5.2 Existence and convergence of the z-transform 2353.5.3 Properties of the z-transform 2403.5.4 z-transform of filters 249

3.6 Discrete Fourier transform 2523.6.1 Definition of the DFT 2523.6.2 Properties of the DFT 2553.6.3 Frequency response of filters 259

3.7 Multirate sequences and systems 2643.7.1 Downsampling 2653.7.2 Upsampling 2683.7.3 Combinations of downsampling and upsampling 2703.7.4 Combinations of downsampling, upsampling, and filtering 2723.7.5 Polyphase representation 278

3.8 Stochastic processes and systems 2853.8.1 Stochastic processes 285






Contents

3.8.2 Systems 2883.8.3 Discrete-time Fourier transform 2923.8.4 Multirate sequences and systems 2943.8.5 Minimum mean-squared error estimation 300

3.9 Computational aspects 3033.9.1 Fast Fourier transforms 3033.9.2 Convolution 3073.9.3 Multirate operations 311

3.A Elements of analysis 3133.A.1 Complex numbers 3133.A.2 Difference equations 3153.A.3 Convergence of the convolution sum 3163.A.4 Dirac delta function 316

3.B Elements of algebra 3183.B.1 Polynomials 3183.B.2 Vectors and matrices of polynomials 3213.B.3 Kronecker product 324

Chapter at a glance 325Historical remarks 328Further reading 328Exercises with solutions 329Exercises 336

4 Functions and continuous-time systems 3434.1 Introduction 3444.2 Functions 345

4.2.1 Functions on the real line 3454.2.2 Periodic functions 351

4.3 Systems 3514.3.1 Continuous-time systems and their properties 3524.3.2 Differential equations 3554.3.3 Linear shift-invariant systems 355

4.4 Fourier transform 3594.4.1 Definition of the Fourier transform 3594.4.2 Existence and inversion of the Fourier transform 3604.4.3 Properties of the Fourier transform 3654.4.4 Frequency response of filters 3734.4.5 Regularity and spectral decay 3744.4.6 Laplace transform 379

4.5 Fourier series 3804.5.1 Definition of the Fourier series 3814.5.2 Existence and convergence of the Fourier series 3834.5.3 Properties of the Fourier series 3854.5.4 Frequency response of filters 394

4.6 Stochastic processes and systems 3954.6.1 Stochastic processes 3954.6.2 Systems 3974.6.3 Fourier transform 399

Chapter at a glance 401Historical remarks 403

xiii






Contents

Further reading 403Exercises with solutions 404Exercises 406

5 Sampling and interpolation 4115.1 Introduction 4125.2 Finite-dimensional vectors 420

5.2.1 Sampling and interpolation with orthonormal vectors 4215.2.2 Sampling and interpolation with nonorthogonal vectors 425

5.3 Sequences 4295.3.1 Sampling and interpolation with orthonormal sequences 4305.3.2 Sampling and interpolation for bandlimited sequences 4375.3.3 Sampling and interpolation with nonorthogonal sequences 442

5.4 Functions 4475.4.1 Sampling and interpolation with orthonormal functions 4495.4.2 Sampling and interpolation for bandlimited functions 4525.4.3 Sampling and interpolation with nonorthogonal functions 470

5.5 Periodic functions 4775.5.1 Sampling and interpolation with orthonormal periodic functions 4775.5.2 Sampling and interpolation for bandlimited periodic functions 481

5.6 Computational aspects 4895.6.1 Projection onto convex sets 491


6 Approximation and compression 5076.1 Introduction 5086.2 Approximation of functions on finite intervals by polynomials 513

6.2.1 Least-squares approximation 5146.2.2 Lagrange interpolation: Matching points 5176.2.3 Taylor series expansion: Matching derivatives 5206.2.4 Hermite interpolation: Matching points and derivatives 5226.2.5 Minimax polynomial approximation 5236.2.6 Filter design 529

6.3 Approximation of functions by splines 5376.3.1 Splines and spline spaces 5386.3.2 Bases for uniform spline spaces 5416.3.3 Strang–Fix condition for polynomial representation 5486.3.4 Continuous-time operators in spline spaces implemented with discrete-

time processing 5546.4 Approximation of functions and sequences by series truncation 560

6.4.1 Linear and nonlinear approximations 5606.4.2 Linear approximation of random vectors and stochastic processes 5666.4.3 Linear and nonlinear diagonal estimators 571

6.5 Compression 5766.5.1 Lossless compression 5776.5.2 Scalar quantization 579

xiv






Contents xv

6.5.3 Transform coding 5846.6 Computational aspects 591

6.6.1 Huffman algorithm for lossless code design 5916.6.2 Iterative design of quantizers 5936.6.3 Estimating from quantized samples 594


7 Localization and uncertainty 6157.1 Introduction 6167.2 Localization for functions 619

7.2.1 Localization in time 6207.2.2 Localization in frequency 6227.2.3 Uncertainty principle for functions 624

7.3 Localization for sequences 6277.3.1 Localization in time 6297.3.2 Localization in frequency 6307.3.3 Uncertainty principle for sequences 6337.3.4 Uncertainty principle for finite-length sequences 637

7.4 Tiling the time–frequency plane 6387.4.1 Localization for structured sets of functions 6387.4.2 Localization for structured sets of sequences 640

7.5 Examples of local Fourier and wavelet bases 6457.5.1 Local Fourier and wavelet bases for functions 6457.5.2 Local Fourier and wavelet bases for sequences 651

7.6 Recap and a glimpse forward 6567.6.1 Tools 6577.6.2 Adapting tools to real-world problems 6587.6.3 Music analysis, communications, and compression 659


Image and quote attribution 673

References 675

Index 681











Quick reference

Abbreviations

AR AutoregressiveARMA Autoregressive moving averageAWGN Additive white Gaussian noiseBIBO Bounded input, bounded outputCDF Cumulative distribution functionDCT Discrete cosine transformDFT Discrete Fourier transformDTFT Discrete-time Fourier transformDWT Discrete wavelet transformFFT Fast Fourier transformFIR Finite impulse responsei.i.d. Independent and identically distributedIIR Infinite impulse responseKLT Karhunen–Loeve transformLMMSE Linear minimum mean-squared errorLPSV Linear periodically shift-varyingLSI Linear shift-invariantMA Moving averageMAP Maximum a posteriori probabilityML Maximum likelihoodMMSE Minimum mean-squared errorMSE Mean-squared errorPDF Probability density functionPMF Probability mass functionPOCS Projection onto convex setsrad RadiansROC Region of convergenceSNR Signal-to-noise ratioSVD Singular value decompositionWSCS Wide-sense cyclostationaryWSS Wide-sense stationary

Abbreviations used in tables and captions but not in the text

FT Fourier transformFS Fourier seriesLT Laplace transform

xvii






xviii Quick reference

Sets

natural numbers N 0, 1, . . .integers Z . . . , −1, 0, 1, . . .positive integers Z

+ 1, 2, . . .rational numbers Q p/q, p, q ∈ Z, q �= 0real numbers R (−∞, ∞)

positive real numbers R+ (0, ∞)

complex numbers C a+ jb or rejθ with a, b, r, θ ∈ R

a generic index set Ia generic vector space Va generic Hilbert space H

closure of set S S

Real and complex analysis

sequence xn argument n is an integer, n ∈ Z

function x(t) argument t is continuous-valued, t ∈ R

ordered sequence (xn)nset containing xn {xn}nvector x with xn as elements [xn]

Kronecker delta sequence δn δn = 1 for n = 0; δn = 0 otherwise

Dirac delta function δ(t)

∫ ∞

−∞x(t) δ(t) dt = x(0) for x continuous at 0

indicator function of interval I 1I 1I(t) = 1 for t ∈ I ; 1I(t) = 0 otherwise

integration by parts

∫u dv = uv −

∫v du

complex number z a+ jb, rejθ, a, b ∈ R, r ∈ [0, ∞), θ ∈ [0, 2π)

conjugation z∗ a− jb, re−jθ

real part of �( · ) �(a+ jb) = a, a, b ∈ R

imaginary part of �( · ) �(a+ jb) = b, a, b ∈ R

conjugation of coefficients X∗(z) X∗(z∗)principal root of unity WN e−j2π/N

Asymptotic notation

big O x ∈ O(y) 0 ≤ xn ≤ γyn for all n ≥ n0;some n0 and γ > 0

little o x ∈ o(y) 0 ≤ xn ≤ γyn for all n ≥ n0;some n0, any γ > 0

Omega x ∈ Ω(y) xn ≥ γyn for all n ≥ n0;some n0 and γ > 0

Theta x ∈ Θ(y) x ∈ O(y) and x ∈ Ω(y)asymptotic equivalence x y limn→∞ xn/yn = 1






Quick reference xix

Standard vector spaces

Hilbert space of square-summablesequences

�2(Z)

{x : Z→ C

∣∣∣∣∣ ∑n

|xn|2 <∞}

with

inner product 〈x, y〉 =∑n xny

∗n

Hilbert space of square-integrablefunctions

L2(R)

{x : R→ C

∣∣∣∣ ∫ |x(t)|2 dt <∞}

with

inner product 〈x, y〉 = ∫x(t) y(t)∗ dt

normed vector space of sequenceswith finite �p norm, 1 ≤ p <∞ �p(Z)

{x : Z→ C

∣∣∣∣∣ ∑n

|xn|p <∞}

with

norm ‖x‖p =(∑

n |xn|p)1/p

normed vector space of functionswith finite Lp norm, 1 ≤ p <∞ Lp(R)

{x : R→ C

∣∣∣∣ ∫ |x(t)|p dt <∞}

with

norm ‖x‖p =(∫ |x(t)|p dt)1/p

normed vector space of boundedsequences with supremum norm

�∞(Z)

{x : Z→ C

∣∣∣∣ supn|xn| <∞

}with

norm ‖x‖∞ = supn |xn|

normed vector space of boundedfunctions with supremum norm

L∞(R)

{x : R→ C

∣∣∣∣ ess supt

|x(t)| <∞}

with

norm ‖x‖∞ = ess supt |x(t)|

Bases and frames for sequences

standard basis {ek} ek,n = 1 for n = k; ek,n = 0 otherwisevector, element of basis or frame ϕ when applicable, a column vectorbasis or frame Φ set of vectors {ϕk}operator Φ concatenation of {ϕk} in a linear

operator: [ϕ0 ϕ1 . . . ϕN−1]vector, element of dual basis or frame ϕ when applicable, a column vector

Φ set of vectors {ϕk}operator Φ concatenation of {ϕk} in a linear

operator: [ϕ0 ϕ1 . . . ϕN−1]

expansion with a basis or frame x = ΦΦ∗x






xx Quick reference

Discrete-time signal processing

Sequence xn signal, vector

Convolution

linear h ∗ x∑k∈Z

xkhn−k

circular(N-periodic sequences)

h� xN−1∑k=0

xkh(n−k) mod N

(h ∗ x)n convolution result at n

Eigensequence vn eigenvector

infinite time vn = ejωn h ∗ v = H(ejω) v

finite time vn = ej2πkn/N h� v = Hkv

Frequency response eigenvalue corresponding to vn

infinite time H(ejω)∑n∈Z

hne−jωn

finite time Hk

N−1∑n=0

hne−j2πkn/N =

N−1∑n=0

hnWknN

Continuous-time signal processing

Function x(t) signal

Convolution

linear h ∗ x∫ ∞

−∞x(τ )h(t− τ ) dτ

circular(T -periodic functions)

h� x

∫ T/2

−T/2

x(τ )h(t− τ ) dτ

(h ∗ x)(t) convolution result at t

Eigenfunction v(t) eigenvector

infinite time v(t) = ejωt h ∗ v = H(ω)v

finite time v(t) = ej2πkt/T h� v = Hkv

Frequency response eigenvalue corresponding to v(t)

infinite time H(ω)

∫ ∞

−∞h(t) e−jωt dt

finite time Hk

∫ T/2

−T/2

h(τ ) e−j2πkτ/T dτ






Quick reference

Spectral analysis

Fourier transform x(t)FT←→ X(ω) X(ω) =

∫ ∞

−∞x(t)e−jωt dt

inverse x(t) =1

2π

∫ ∞

−∞X(ω) ejωt dω

Fourier series coefficients x(t)FS←→ Xk Xk =

1

T

∫ T/2

−T/2

x(t) e−j(2π/T )kt dt

reconstruction x(t) =∑k∈Z

Xkej(2π/T )kt

discrete-time Fourier transform xnDTFT←→ X(ejω) X(ejω) =

∑n∈Z

xne−jωn

inverse xn =1

2π

∫ π

−π

X(ejω) ejωn dω

discrete Fourier transform xnDFT←→ Xk Xk =

N−1∑n=0

xnWknN

inverse xn =1

N

N−1∑n=0

XkW−knN

z-transform xnZT←→ X(z) X(z) =

∑n∈Z

xnz−n

xxi











Acknowledgments

This book would not exist without the help of many people, whom we attempt tolist below. We apologize for any omissions and welcome corrections and suggestions.

We are grateful to Professor Libero Zuppiroli of EPFL and Christiane Grimmfor the photograph that graces the cover; Professor Zuppiroli proposed an ex-periment from Newton’s treatise, Opticks [70], as emblematic of the book, andMs. Grimm beautifully photographed the apparatus that he designed. FrancoiseBehn, Jocelyne Plantefol, and Jacqueline Aeberhard typed parts of the manuscript,Eric Strattman assisted with some of the figures, Krista Van Guilder designed andimplemented the initial book web site, and Jorge Albaladejo Pomares designedand implemented the book blog. We thank them for their diligence and patience.S. Grace Chang and Yann Barbotin helped organize and edit the problem compan-ion, while Patrick Vandewalle designed and implemented a number of MATLAB R©problems available on the book website.1 We thank them for their expertise andinsight. We are indebted to George Beck for his Mathematica R© editorial commentson Wolfram Demonstrations inspired by this book. We are grateful to GiovanniPacifici for the many useful scripts for automating various book-writing processes.We thank John Cozzens of the US National Science Foundation for his unwaveringsupport over the last two decades. At Cambridge University Press, Steven Holtheld us to high standards in our writing and typesetting through many correctionsand suggestions; Elizabeth Horne and Jonathan Ratcliffe coordinated productionexpertly; and, last but not least, Phil Meyler repeatedly (and most gracefully) re-minded us of our desire to finish the book.

Many instructors have gamely tested pre-alpha, alpha, and beta versions ofthis manuscript. Of these, Amina Chebira, Filipe Condessa, Mihailo Kolundzija,Yue M. Lu, Truong-Thao Nguyen, Reza Parhizkar, and Jayakrishnan Unnikrishnanhave done far more than their share in providing invaluable comments and sugges-tions. We also thank Robert Gray for reviewing the manuscript and providing manyimportant suggestions, in particular a better approach to covering the Dirac deltafunction; Thierry Blu for, among other things, providing a simple proof for a partic-ular case of the Strang–Fix theorem; Matthew Fickus for consulting on some finermathematical points; Michael Unser for his notes and teaching approach; and ZoranCvetkovic, Minh N. Do, and Philip Schniter for teaching with the manuscript andproviding many constructive comments. Useful comments have also been provided

1http://www.fourierandwavelets.org

xxiii






xxiv Acknowledgments

by Pedro Aguilar, Ganesh Ajjanagadde, A. Avudainayagam, Jerry Bauch, Anirud-dha Bhargava, Ozan Caglayan, Andre Tomaz de Carvalho, Gustavo Corach, ToddDoucet, S. Esakkirajan, German Gonzalez, Alexandre Haehlen, Mina Karzand, Ull-rich Monich, Juan Pablo Muszkats, Hossein Rouhani, Noah D. Stein, John Z. Sun,Christophe Tournery, and Lav R. Varshney.

Martin Vetterli thanks current and former EPFL graduate students and post-docs who helped develop the material, solve problems, catch typos, and suggest im-provements, among other things. They include Florence Benezit, Amina Chebira,Minh N. Do, Ivan Dokmanic, Pier Luigi Dragotti, Ali Hormati, Ivana Jovanovic,Mihailo Kolundzija, Jerome Lebrun, Yue M. Lu, Pina Marziliano, Fritz Menzer,Reza Parhizkar, Paolo Prandoni, Juri Ranieri, Olivier Roy, Rahul Shukla, Jayakr-ishnan Unnikrishnan, Patrick Vandewalle and Vladan Velisavljevic. He gratefullyacknowledges support from EPFL, the Swiss NSF through awards 2000-063664,200020-103729, and 200021-121935, and the European Research Council throughaward SPARSAM 247006. In addition, the support from Qualcomm, in particularfrom Dr. Chong Lee, is gratefully acknowledged.

Jelena Kovacevic thanks her present and past graduate students RamamurthyBhagavatula, Amina Chebira, Kuan-Chieh Jackie Chen, Siheng Chen, Filipe Con-dessa, Charles Jackson, Xindian Long, Michael T. McCann, Anupama Kuruvilla,Thomas E. Merryman, Vivek Oak, Aliaksei Sandryhaila, and Gowri Srinivasa, manyof whom served as TAs for her classes at CMU, together with Pablo Hennings Yeo-mans. Thanks are due also to all the students in the following classes at CMU:42-431/18-496, 42-540/42-698, 42-703/18-799, 42-731/18-795, and 42-732/18-790,taught from 2003 to 2013. She gratefully acknowledges support from the US NSFthrough awards 1017278, 1130616, 515152, 633775, and 331657; the NIH throughawards EB008870, EB009875, and 1DC010283; the Carnegie Institute of Tech-nology at CMU through an Infrastructure for Large-Scale Biomedical Computingaward; the Pennsylvania State Tobacco Settlement, the Kamlet–Smith Bioinfor-matics Grant; and the Philip and Marsha Dowd Teaching Fellowship.

Vivek Goyal thanks TAs and students at MIT and BU for suggestions, in par-ticular Giuseppe Bombara, Baris Erkmen, Ying-zong Huang, Zahi Karam, AhmedKirmani, Srilalitha Kumaresan, Ranko Sredojevic, Ramesh Sridharan, Watchara-pan Suwansantisuk, Vincent Tan, Archana Venkataraman, Adam Zelinski, and Ser-hii Zhak. He gratefully acknowledges support from the US NSF through awards0643836, 0729069, 1101147, 1115159, and 1161413; Texas Instruments throughits Leadership University Program; Hewlett–Packard; Qualcomm; Siemens; MITthrough the Esther and Harold E. Edgerton Career Development Chair; and theCentre Bernoulli of EPFL.






Preface

Our main goals in this book and its companion volume, Fourier and Wavelet SignalProcessing (FWSP) [57], are to enable an understanding of state-of-the-art signalprocessing methods and techniques, as well as to provide a solid foundation for thosehoping to advance the theory and practice of signal processing. We believe that thebest way to grasp and internalize the fundamental concepts in signal processing isthrough the geometry of Hilbert spaces, as this leverages the great innate humancapacity for spatial reasoning. While using geometry should ultimately simplify thesubject, the connection between signals and geometry is not innate. The reader willhave to invest effort to see signals as vectors in Hilbert spaces before reaping thebenefits of this view; we believe that effort to be well placed.

Many of the results and techniques presented in the two volumes, while rootedin classic Fourier techniques for signal representation, first appeared during a flurryof activity in the 1980s and 1990s. New constructions of local Fourier transforms andorthonormal wavelet bases during that period were motivated both by theoreticalinterest and by applications, multimedia communications in particular. New baseswith specified time–frequency behavior were found, with impact well beyond theoriginal fields of application. Areas as diverse as computer graphics and numericalanalysis embraced some of the new constructions – no surprise given the pervasiverole of Fourier analysis in science and engineering.

Many of these new tools for signal processing were developed in the appliedharmonic analysis community. The resulting high level of mathematical sophistica-tion was a barrier to entry for many signal processing practitioners. Now that thedust has settled, some of what was new and esoteric has become fundamental; wewant to bring these new fundamentals to a broader audience. The Hilbert spaceformalism gives us a way to begin with the classical Fourier analysis of signals andsystems and reach structured representations with time–frequency locality and theirvaried applications. Whenever possible, we use explanations rooted in elementaryanalysis over those that would require more advanced background (such as measuretheory). We hope to have balanced the competing virtues of accessibility to thestudent, rigor, and adequate analytical power to reach important conclusions.

The book can be used as a self-contained text on the foundations of signalprocessing, where discrete and continuous time are treated on equal footing. Allthe necessary mathematical background is included, with examples illustrating theapplicability of the results. In addition, the book serves as a precursor to FWSP,which relies on the framework built here; the two books are thus integrally related.

xxv






xxvi Preface

Foundations of Signal Processing This book covers the foundations for an in-depth understanding of modern signal processing. It contains material that manyreaders may have seen before scattered across multiple sources, but without theHilbert space interpretations, which are essential in signal processing. Our aim isto teach signal processing with geometry, that is, to extend Euclidean geometric in-sights to abstract signals; we use Hilbert space geometry to accomplish that. Withthis approach, fundamental concepts – such as properties of bases, Fourier rep-resentations, sampling, interpolation, approximation, and compression – are oftenunified across finite dimensions, discrete time, and continuous time, thus makingit easier to point out the few essential differences. Unifying results geometricallyhelps generalize beyond Fourier-domain insights, pushing the understanding farther,faster.

Chapter 2, From Euclid to Hilbert, is our main vehicle for drawing out unifyingcommonalities; it develops the basic geometric intuition central to Hilbert spaces,together with the necessary tools underlying the constructions of bases and frames.

The next two chapters cover signal processing on discrete-time and continuous-time signals, specializing general concepts from Chapter 2. Chapter 3, Sequencesand discrete-time systems, is a crash course on processing signals in discrete time ordiscrete space together with spectral analysis with the discrete-time Fourier trans-form and discrete Fourier transform. Chapter 4, Functions and continuous-timesystems, is its continuous-time counterpart, including spectral analysis with theFourier transform and Fourier series.

Chapter 5, Sampling and interpolation, presents the critical link between dis-crete and continuous domains given by sampling and interpolation theorems. Chap-ter 6, Approximation and compression, veers from exact representations to approx-imate ones. The final chapter in the book, Chapter 7, Localization and uncertainty,considers time–frequency behavior of the abstract representation objects studiedthus far. It also discusses issues arising in applications as well as ways of adaptingthe previously introduced tools for use in the real world.

Fourier and Wavelet Signal Processing The companion volume focuses on signalrepresentations using local Fourier and wavelet bases and frames. It covers thetwo-channel filter bank in detail, and then uses it as the implementation vehiclefor all sequence representations that follow. The local Fourier and wavelet methodsare presented side-by-side, without favoring any one in particular; the truth is thateach representation is a tool in the toolbox of the practitioner, and the problem orapplication at hand ultimately determines the appropriate one to use. We end withexamples of state-of-the-art signal processing and communication problems, withsparsity as a guiding principle.

Teaching points Our aim is to present a synergistic view of signal representationsand processing, starting from basic mathematical principles and going all the wayto actual constructions of bases and frames, always with an eye on concrete applica-tions. While the benefit is a self-contained presentation, the cost is a rather sizablemanuscript. Referencing in the main text is sparse; pointers to the bibliography aregiven in Further reading at the end of each chapter.






Preface xxvii

The material grew out of teaching signal processing, wavelets, and applicationsin various settings. Two of us (MV and JK) authored a graduate textbook, Waveletsand Subband Coding (originally with Prentice Hall in 1995, now open access2),which we and others used to teach graduate courses at various US and Europeaninstitutions. With the maturing of the field and the interest arising from and forthese topics, the time was right for the three of us to write entirely new texts gearedtoward a broader audience. We and others have taught with these books, in theirentirety or in parts, a number of times and to a number of different audiences: fromsenior undergraduate to graduate level, and from engineering to mixes that includelife-science students.

The books and their website3 provide a number of features for teaching andlearning:

• Exercises are an integral part of the material and come in two forms: solvedexercises with explicit solutions within the text, and regular exercises thatallow students to test their knowledge. Regular exercises are marked with (1),(2), or (3) in increasing order of difficulty.

• Numerous examples illustrate concepts throughout the book.

• An electronic version of the text is provided with the printed copy. It includesPDF hyperlinks and an additional color to enhance interpretation of figures.

• A free electronic version of the text without PDF hyperlinks, exercises orsolved exercises, and with figures in grayscale, is available at the book website.

• AMathematica R© companion, which contains the code to produce all numericalfigures in the book, is provided with the printed version.

• Several interactive Mathematica R© demonstrations using the free CDF playerare available at the book website.

• Additional material, such as lecture slides, is available at the book website.

• To instructors, we provide a Solutions Manual, with solutions to all regularexercises in the book.

Notational points To traverse the book efficiently, it will help to know the variousnumbering conventions that we have employed. In each chapter, a single counter isused for definitions, theorems, and corollaries (which are all shaded) and anotherfor examples (which are slightly indented). Equations, figures, and tables are alsonumbered within each chapter. A prefix En.m– in the number of an equation,figure, or table indicates that it is part of Solved Exercise n.m in Chapter n. Theletter P is used similarly for statements of regular exercises.

Martin Vetterli, Jelena Kovacevic, and Vivek K Goyal

2http://waveletsandsubbandcoding.org/3http://www.fourierandwavelets.org/











Documents

Foundations of Signal Processingassets.cambridge.org/97811070/38608/frontmatter/... · “Foundations of Signal Processing by Vetterli, Kovaceviˇ c, and Goyal, is a pleasure to read