COMP4121 Lecture Notes

Foundations of Signal Processing

LiC: Aleks Ignjatovic

ignjat@cse.unsw.edu.au

THE UNIVERSITY OFNEW SOUTH WALES

School of Computer Science and EngineeringThe University of New South Wales

Sydney 2052, Australia

1 The Fourier Transform

What happens if f(t) is not periodic, but decays with time? In this case arepresentation of a signal in a way which is somewhat analogous to the Fourierseries for the periodic functions, is accomplished using the Fourier transform.Since the rigorous mathematical development is somewhat tricky, rather thandoing things rigorously, we will only motivate the Fourier transform heuristically.

So assume that f(t) is vanishingly small outside [−L,L], where L is chosensufficiently large, and then extend f(t) periodically outside (−L,L], simply bymaking shifted copies of f(t) over (−L,L], thus obtaining fL(t) which is 2Lperiodic; see Figure 1.

-1000 -500 500 1000

-5

5

10

⇓

-1000 -500 500 1000

-5

5

10

Figure 1.1: Turning a function which is small outside [−L,L] (between the redlines) into a periodic function, by making its shifted copies.

We can now expand fL(t) into Fourier series

fL(t) =

∞∑k=−∞

cLk eikπL t (1.1)

with

cLk =1

2L

∫ L

−LfL(t)e− i kπL tdt ≈ 1

2L

∫ ∞−∞

f(t)e− i kπL tdt, (1.2)

because f(t) is small outside [−L,L]. Let us now define the Fourier transform

f(ω) of f(t) as

f(ω) =

∫ ∞−∞

f(t)e− iωtdt. (1.3)

Then, comparing (1.2) and (1.3), we see that cLk ≈ 12L f(kπ/L), and substi-

tuting in (1.1) we get

fL(t) ≈∞∑

k=−∞

1

2Lf(kπ/L)ei

kπL t =

1

2π

∞∑k=−∞

π

Lf(kπ/L)ei

kπL t.

1

The last sum is the Riemann sum of an integral:

fL(t) ≈ 1

2π

∫ ∞−∞

f(ω)eiωtdω,

with approximations becoming better as L becomes larger; thus, we can expectthat in the limit

f(t) =1

2π

∫ ∞−∞

f(ω)eiωtdω. (1.4)

Unfortunately, things are a bit more complicated than that. We mentionedthat the Fourier series of functions with a finite L2-norm, i.e., such that

‖f‖2 =

√1

2a

∫ a

−a|f(t)|2 dt <∞

converge almost everywhere to f(t), but that for functions with a finite L1-norm,

‖f‖1 =1

2a

∫ a

−a|f(t)| dt <∞

the Fourier series might diverge everywhere.1

When it comes to the convergence of the Fourier transform of aperiodicfunctions, the situation is somewhat reversed: the Fourier integral converges forall L1 functions, i.e., for all functions such that

‖f‖1 =

∫ ∞−∞|f(t)| dt <∞

but it does not necessarily converge for all L2 functions, i.e., functions whichsatisfy

‖f‖2 =

√∫ ∞−∞|f(t)|2 dt <∞

Fortunately, one can introduce Fourier transform of L2 functions via a comple-tion, i.e., by representing each function f(t) ∈ L2 as a limit of functions fn(t)which are both in L1 and L2, and defining the Fourier transform of f(t) as thelimit of the Fourier transforms of fn(t). Thus, for all practical purposes, we canwork as if the limit in

f(t) =1

2π

∫ ∞−∞

f(ω)eiωtdω = lima,b→∞

1

2π

∫ b

−af(ω)eiωtdω

really existed, i.e., as if this, so called improper integral (because the limits ofintegration are infinite) converged.

1The almost everywhere convergence of the Fourier series of functions in L2[−a, a] is a verydeep theorem; here is what the Wikipedia has to say about it: “The problem whether theFourier series of any continuous function converges almost everywhere was posed by NikolaiLusin in the 1920s and remained open until finally resolved positively in 1966 by LennartCarleson. Indeed, Carleson showed that the Fourier expansion of any function in L2 convergesalmost everywhere. This result is now known as Carleson’s theorem. . . Despite a number ofattempts at simplifying the proof, it is still one of the most difficult results in analysis.

Contrariwise, Andrey Kolmogorov, in his very first paper published [in 1923] when he was21, constructed an example of a function in L1 whose Fourier series diverges almost everywhere(later improved to divergence everywhere).”

2

One can see f(ω) as providing the amplitude and phase of the “ω-frequency

component” of the signal. To see this, recall that for each ω, the value of f(ω)is a complex number that can be represented via its absolute value ρ(ω) =

|f(ω)|, ρ(ω) ≥ 0, and its argument θ(ω) = arg f(ω), −π < θ(ω) ≤ π, i.e.,

f(ω) = ρ(ω)ei θ(ω)

and so

f(t) =1

2π

∫ ∞−∞

f(ω)eiωtdω =1

2π

∫ ∞−∞

ρ(ω)ei θ(ω)eiωtdω =1

2π

∫ ∞−∞

ρ(ω)ei(ωt+θ(ω))dω

which shows that ρ(ω) is the amplitude and θ(ω) is the phase shift for eiωt

harmonic oscillation of frequency ω, and the signal can be seen representedas an “infinitely refined” sum (i.e., an integral) of such harmonic oscillationsρ(ω)ei(ωt+θ(ω)).

Comparing (1.3) and (3.1) we see a remarkable property of the Fourier trans-

form: the transformation from f(t) into f(ω) and back from f(ω) to f(t) areessentially one and the same operation, save the normalization factor 1

2π andthe minus sign in the exponent. The normalization factor disappears if we re-place everywhere in our definitions eiωt with e2π iωt; thus, if we take the Fouriertransform of a signal f(t), thus obtaining its “frequency profile” f , and then

once more take the Fourier transform of f obtaining the frequency profilef of

the frequency profile f of the original signal, we getf(t) = f(−t) i.e., we just

get back the original signal only with the direction of time reversed!

2 Band-limited signals

Most signals of engineering interest, due to physical limitations, cannot containarbitrarily high frequencies, or we are simply not interested in frequencies abovecertain limit, treating them as unwanted noise. For example, we cannot hearsounds of frequencies above 20kHz.

Assume that f(t) has a finite bandwidth; we will chose a unit interval of timeso that all frequencies present in the signal are smaller than ±π, i.e., that1

f(ω) = 0 if |ω| ≥ π.

It will be clear later how to choose such time measuring unit.

We will also assume that signals have finite energy, i.e., that their usual L2

norm is finite: ∫ ∞−∞

f(t)2 dt <∞

The space of such signals, with the usual scalar product and norm of L2

are called the space of band limited signals of finite energy or the Paley-Wienerspace and is usually denoted by BL(π) or PWπ.

1It might look paradoxical that frequencies can be negative. While this is mainly a technicalconvenience which makes signal representation considerably simpler, one can also rememberthat a wheel can spin with certain angular velocity in two opposite directions. . .

3

Since f(ω) = 0 outside (−π, π), we have

f(t) =1

2π

∫ ∞−∞

f(ω)eiωtdω =1

2π

∫ π

−πf(ω)eiωtd.ω (2.1)

We can now represent f(ω) over the interval (−π, π) by its Fourier series:

f(ω) =

∞∑k=−∞

ckei kω,

where

ck =1

2π

∫ π

−πf(ω)e− i kωdω.

Comparing this equation with (2.1), we see that ck = f(−k), i.e., the Fouriercoefficients are just the samples of the signal! Thus, the Fourier transform of aπ-band limited signal of finite energy is given by

f(ω) =

∞∑k=−∞

f(−k)ei kω. (2.2)

We can now substitute f(ω) in with its Fourier series we obtain

f(t) =1

2π

∫ π

−π

( ∞∑k=−∞

f(−k)ei kω

)eiωtdω (2.3)

We can now use the substitution k = −n and, changing the order of summationand integration, get‡

f(t) =1

2π

∞∑n=−∞

f(n)

∫ π

−πe− iωneiωtdω

=1

2π

∞∑n=−∞

f(n)

∫ π

−πeiω(t−n)dω

=1

2π

∞∑n=−∞

f(n)

[eiω(t−n)

i(t− n)

]ω=πω=−π

=1

2π

∞∑n=−∞

f(n)

[cosω(t− n) + i sinω(t− n)

i(t− n)

]ω=πω=−π

Note that cosπ(t−n)− cos(−π(t−n)) = 0 while sinπ(t−n)− sin(−π(t−n)) =2 sinπ(t− n) and we get

f(t) =

∞∑n=−∞

f(n)sinπ(t− n)

π(t− n). (2.4)

A representation of a band limited signal by formula (2.4) is usually called theShannon Sampling Theorem, even though he did not discovered it but has only

‡The mathematicians will complain that we should have first investigated if the seriesconverges uniformly, in order to be able to exchange the order of integration and summation. . .

4

popularized it. A more proper name should be “Whittaker - Nyquist - Kotel-nikov - Shannon Sampling Theorem”. It asserts that it is possible to obtain aperfect reconstruction of a band limited signal from its discrete samples, sam-pled at intervals which are half the period of the sinusoid of the maximal allowedfrequency, because the sine wave sinπt of frequency π has period equal to 2, andwe sample the signal at all integers. This is usually expressed as “the samplingfrequency should be twice the maximal frequency present in the signal”. This, intheory, makes digital signal processing (DSP) possible, because in principle, acontinuous time signal can be perfectly captured by its (sufficiently dense uni-formly spaced) discrete samples. However, there is a “minor glitch” there: ifyou look at the sampling expansion (2.4) we see that the interpolation func-tions decay only as 1/n. Thus, to obtain a good reconstruction of the “analogwaveform” of the signal at a non integer point t, we need a very large numberof samples around t. Luckily, DSP (Digital Signal Processing) engineers havedifferent, more clever ways of interpolating band limited signals which do notproduce unwanted artifacts which would occur if we simply truncated the series(2.4) at any but extremely large number of terms.

The functionsin t

tis of cardinal importance to signal processing; in fact it

is usually called the cardinal sine function or the sinc function and is denotedby sinc(t). Thus we have

f(t) =

∞∑n=−∞

f(n) sincπ(t− n) (2.5)

Since

1

2π

∫ π

−πeiωtdω =

1

2π

eiωt

i t

∣∣∣πω=−π

=sinπt

πt= sincπt; (2.6)

we see that the Fourier transform sinc(ω) of sincπt is given by

sinc(ω) =

{1 if |ω| < π

0 if |ω| > π(2.7)

If we substitute t = 0 in (2.6) then, since 12π

∫ π−π dω = 1, we get sinc 0 = 1; at

all other integers n 6= 0 we have

1

2π

∫ π

−πeiωndω = 0

because we have n many complete periods of eiωn within [−π, π]. Thus, if wesample sincπt at integers we get 1 at t = 0 and 0 at all other integer samplingpoints, as one would expect from (2.5).

3 Fourier series in the base {sinc π(t− n)}n∈NEvery function f(x) which is square integrable, i.e., such that∫ ∞

−∞f2(t)dt <∞

5

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

-10 -5 0 5 10

f(t)

-0.5

0

0.5

1

1.5

-π 0 π

f(ω)

Figure 2.1: sincπt (left) and its Fourier transform (right)

(this is usually denoted by f ∈ L2) has a Fourier transform f(ω) such that1

f(t) =1

2π

∫ ∞−∞

f(ω)eiωtdω; (3.1)

f(ω) =

∫ ∞−∞

f(t)e− iωtdt; (3.2)

One can show that the mapping F : f(t) 7→ f(ω) has the following importantproperty of isometry : for every two functions f(t), g(t) ∈ L2

〈f(t), g(t)〉 =

∫ ∞−∞

f(t)g(t)dt =1

2π

∫ π

−πf(ω)g(ω)dω = 〈f(ω), g(ω)〉 (3.3)

and in particular,

‖f(t)‖2 = 〈f(t), f(t)〉 =

∫ ∞−∞

f(t)2dt =1

2π

∫ π

−πf(ω)f(ω)dω = ‖f(ω)‖2 (3.4)

Thus, F preserves both the “lengths” (i.e., norms) of vectors as well as theangles between vectors, i.e., it preserves the “geometry” of the two spaces ofsignals, one in the time domain, the other in the “frequency” domain. Notethat (2.6) implies

sincπ(t− n) =1

2π

∫ π

−πeiω(t−n)dω =

1

2π

∫ π

−πe− inωeiωtdω; (3.5)

Thus,

F [sincπ(t− n)](ω) =

{e− inω if |ω| < π

0 if |ω| > π(3.6)

1As we have already mentioned, this is not quite true, because there are functions f ∈ L2

for which integral (3.1) diverges. This is circumvented by representing f as a limit of functionsfn which are both in L1 and in L2, and the Fourier Transform of such f is defined as thelimit of the Fourier Transforms of functions fn.

6

∫ ∞−∞

sincπ(t− n)sincπ(t−m)dt =1

2π

∫ π

−πe− inωeimωdω =

{1 if n = m

0 if n 6= m

(3.7)This means that functions {sincπ(t− k)}k∈Z are orthonormal; in fact, they forman orthonormal base of the vector space of band limited functions of finite en-ergy, and the Shannon expansion (2.5) simply represents a signal f(t) in thatbase, i.e., is the Fourier series of f(t) with respect to the base {sincπ(t− k)}k∈Z.To see that, using the isometry property, we have that the values of the projec-tions of f(t) onto the base vectors sincπ(t− n) are equal to:∫ ∞

−∞f(t)sincπ(t− n) = 〈f, sincπ(t− n)〉 = 〈f , sincπ(t− n)〉

=1

2π

∫ π

−πf(ω)e− i(−n)ωdω =

1

2π

∫ π

−πf(ω)einωdω

= f(n)

Since for any orthonormal base {bn} the corresponding Fourier series expansionis of the form

f =∑〈f, bn〉bn (3.8)

we get that

f(t) =

∞∑n=−∞

〈f, sincπ(t− n)〉sincπ(t− n) =

∞∑n=−∞

f(n)sincπ(t− n)

Thus, we see that the values of a π-band limited signal are the coefficientsof two Fourier series.

1. In the frequency domain (i.e., in the space of the Fourier transforms of

the band limited signals) they are the Fourier coefficients of f(ω) in theusual base of complex exponentials {einω : n ∈ Z}:

f(ω) =

∞∑n=−∞

f(−n)einω =

∞∑n=−∞

f(n)ei(−n)ω (3.9)

2. In the time domain, taking the Inverse Fourier Transform of both sides,we get the Shannon formula which is just the (generalised) Fourier seriesin the base {sincπ(t− k)}k∈Z:

f(t) =

∞∑n=−∞

f(n) sincπ(t− n) (3.10)

As mentioned, the Fourier transform defines an isometry between the twodomains: the “length” of a vector i.e., a signal f(t) is the same if measuredeither in time or in the frequency domain, in the sense that

‖f(t)‖2 =

∫ ∞−∞

f(t)2dt =

∫ ∞−∞|f(ω)|2dt = ‖f(ω)‖2

and also 〈f(t), g(t)〉 = 〈f(ω), g(ω)〉.

7

We now establish yet another important isometry. Note that

〈f(t), g(t)〉 =

∫ ∞−∞

f(t)g(t)dt

=

∫ ∞−∞

( ∞∑n=−∞

f(n) sincπ(t− n)

)( ∞∑m=−∞

g(m) sincπ(t−m)

)dt

=

∞∑n=−∞

∞∑m=−∞

f(n)g(m)

∫ ∞−∞

sincπ(t− n) sincπ(t−m)dt

=

∞∑n=−∞

f(n)g(n)

where in the last step we used the fact that, by orhonormality of the base{sincπ(t− n)} we have∫ ∞

−∞sincπ(t− n) sincπ(t−m)dt =

{1 if m = n

0 otherwise(3.11)

Thus,

〈f(t), g(t)〉 =

∞∑n=−∞

f(n)g(n)

In particular, we also get

‖f(t)‖2 = 〈f(t), f(t)〉 =

∫ ∞−∞

f(t)2dt =

∞∑n=−∞

f(n)2

Thus, the space of π-band limited signals of finite energy is isometric to thespace l2 of square summable sequences, l2 = {an :

∑∞n=−∞ a2n < ∞}, where

the numbers an can be interpreted as samples of a π-band limited signal offinite energy. Note that in such isometry function sincπ(t − n) is mapped tothe sequence

en(m) =

{1 if m = n

0 otherwise(3.12)

Again, we have that in the base {en : n ∈ Z} the Fourier series of the

sequence of samples ~f = 〈f(n)〉n∈Z which corresponds to the band limited signalf(t) has the coefficients f(n), i.e.,

~f =

∞∑n=−∞

f(n)en. (3.13)

8

Thus, we get the following isometric spaces:

frequency domain:

f(ω) base: {einω : n ∈ Z} Fourier series: f(ω) =

∞∑n=−∞

f(n)e− inω

time domain:

f(t) base: { sincπ(t− n) : n ∈ Z} Fourier series: f(t) =

∞∑n=−∞

f(n)sincπ(t− n)

sequences from l2

{f(n)} base: { en = 〈. . . 0, 1, 0 . . .〉 : n ∈ Z} Fourier series: ~f =

∞∑n=−∞

f(n)en

4 Filters

So we now know how to represent band-limited signals, and would like to “pro-cess” them. For example, we might want to remove certain frequencies, suchas the 50 Hz hum which could have come from the mains power supply, orthe “hissing” from an old vinyl LPs. Such operations are examples of filtering.In general, a filter L acts on signals f(t) producing an output signal L[f ](t).Note that both the input signal f(t) and the output signal L[f ](t) are taken as“wholes’, in their entire duration; filters do NOT produce outputs L[f ](t0) atany given instant of time t0 from the value of f(t0) alone. Thus, filters operateon signals as points of an appropriate inner product space of signals; for thatreason they are also called operators acting on such spaces; to emphasis this,we place inputs of filters into square brackets: L[f ]; since the output is a signalwhich is a function of time we also write L[f ](t) to denote the output signal.Filters have to satisfy the following three properties:1

1. L is continuous, i.e., if a sequence of signals fn(t) converges to a signalf(t) in the sense of the norm, then the image of the limit signal is thelimit of the images. More precisely,

if limn→∞

‖fn(t)− f(t)‖2 = 0, then also limn→∞

‖L[fn](t)− L[f ](t)‖2 = 0.

2. L is a linear operator, i.e., for arbitrary scalars c1, c2 and arbitrary twoband limited signals f(t), g(t),

L[c1f + c2g](t) = c1L[f ](t) + c2L[g](t)

3. L is time invariant (also called shift invariant) if an image of a timeshifted signal is the just an equally shifted image of the original signal.More precisely,

1We will restrict ourselves to the space of π-band limited signals with the usual scalarproduct and the L2 norm.

9

L[f(t+ τ)] = L[f(t)](t+ τ)

What a filter does to an arbitrary signal in BL(π) is completely determinedby what it does to the sinc(πt)

L(f(t)) = L

[ ∞∑k=−∞

f(k)sinπ(t− k)

π(t− k)

]

= L

[limN→∞

N∑k=−N

f(k)sinπ(t− k)

π(t− k)

]

= limN→∞

L

[N∑

k=−N

f(k)sinπ(t− k)

π(t− k)

](by continuity of L)

= limN→∞

N∑k=−N

f(k)L

[sinπ(t− k)

π(t− k)

](by linearity of L)

= limN→∞

N∑k=−N

f(k)L

[sinπu

πu

]u=(t−k)

(by time invariance of L)

=

∞∑k=−∞

f(k)L[sinc πu](t− k)

Here L[sinc πu](t− k) stands for first obtaining the image l(t) = L[sinc πt] andthen replacing t in l(t) with the shifted variable t − k, i.e., L[sinc πu](t − k) =

l(t− k). Recall that by (2.7) the Fourier transform sinc(ω) of sincπt = sinπtπt is

given by

sinc(ω) =

{1 −π ≤ ω ≤ π0 otherwise

Consider the response of L for input sincπt,

l(t) = L

[sinπt

πt

]and let its Fourier transform be L(ω). Then

l(t) = L

[sinπt

πt

]=

1

2π

∫ π

−πL(ω)eiωtdω

and so

l(t− k) =1

2π

∫ π

−πL(ω)eiω(t−k)dω.

Using the Shannon formula and the linearity of L, for an arbitrary input

10

f(t) using (2.2) we get

L[f ](t) =

∞∑k=−∞

f(k)L[sinc](t− k)

=

∞∑k=−∞

f(k)1

2π

∫ π

−πL(ω)eiω(t−k)dω

=1

2π

∫ π

−π

∞∑k=−∞

f(k)e− iωkeiωtL(ω)dω

=1

2π

∫ ∞−∞

f(ω)L(ω)eiωtdω.

Thus, a linear operator acts on a signal by multiplying the Fourier transform ofthe signal by the transfer function of the operator, i.e. by the Fourier transformL(ω) of L

(sinπtπt

). Note that

L[f ](t) =1

2π

∫ π

−πf(ω)L(ω)eiωtdω

=1

2π

∫ π

−π

∫ ∞−∞

f(u)e− iωuL(ω)eiωtdudω

=

∫ ∞−∞

f(u)1

2π

∫ π

−πL(ω)eiω(t−u)dωdu

=

∫ ∞−∞

f(u)l(t− u)du (4.1)

where

l(t) = L

(sinπt

πt

).

The integral ∫ ∞−∞

f(u)l(t− u)du

is called a convolution of f and l, f∗l. Note that the convolution of two functionscorresponds to a product of their Fourier transforms. We also have for everyinteger k

L[f ](k) =1

2π

∫ π

−πf(ω)L(ω)eiωkdω

=1

2π

∫ π

−π

∞∑n=−∞

f(n)e− inω∞∑

m=−∞l(m)e− imωeiωkdω

=

∞∑n,m=−∞

f(n)l(m)1

2π

∫ π

−πei(k−(n+m))ωdω

However,

1

2π

∫ π

−πei(k−(n+m))ωdω =

{1 if k = m+ n

0 otherwise

11

Thus, a summand is non zero only for k = m+ n, i.e., only for m = k − n, andwe get:

L[f ](k) =

∞∑n=−∞

f(n− k)l(k).

In practice both f(n) and l(k) have only finitely many samples. Thus,

L[~f ](n) =

∞∑k=−∞

f(n− k)l(k)

becomes

L[~f ](n) =

M/2∑k=−M/2

f(n− k)l(k) for all n = 0 . . .K

M is usually less than 1000 but K can be tens of millions of samples for asingle song, so we split ~f into frames of size equal to the size of ~l and computethe convolution using the FFT and then put the resulting pieces of the outputsignal together by aligning them appropriately.

12