'

&

$

%

TheEvolutionofComputationalHarmonicAnalysis

JoeLakey

DepartmentofMathematicalSciences

NewMexicoStateUniversity

1

'

&

$

%

AnalysisinAntiquity

Pythagoras(c.569-475BC):MusicoftheSpheres:

Thereisgeometryinthehummingofthestrings...thereismusic

inthespacingofthespheres

2

'

&

$

%Figure1:Pythagoras:Lovedrighttriangles,hatedirrationals

3

Figure2:Circularplanetaryorbits:SimpleHar-monicMotion

3-1

'

&

$

%

Archimedes(287-212BC)

•ProvedareaofcircleisπR2

•Estimatedπ

•Rannakedthroughthestreets.

4

'

&

$

%Figure3:Archimedes:givememathematicsorgivemedeath

5

'

&

$

%

Mathematization:Theneedforcertainty

Galileo(1564–1642):

“Philosophyiswritteninthisgrandbook,theuniverse,which

standscontinuallyopentoourgaze.Butthebookcannotbe

understoodunlessonefirstlearnstocomprehendthelanguageand

readthecharactersinwhichitiswritten.Itiswritteninthe

languageofmathematics,anditscharactersaretriangles,circles,

andothergeometricfigureswithoutwhichitishumanlyimpossible

tounderstandasinglewordofit;withouttheseoneiswanderingin

adarklabyrinth.”OpereIlSaggiatorep.171.

6

'

&

$

%Figure4:BestfriendswithPopeUrbanVIII

7

'

&

$

%

Inthebeginningtherewere...

Differentialequations

8

'

&

$

%

Newton(1643-1727)Heavenlyandterrestrialbodiesobeysame

physicallaws.

Newton’sSecondLaw:F=ma=md2x

dt2

Hooke’sLawforsprings:F=−kx

Harmonicoscillatorequation:x(t)=−km

d2x

dt2

9

'

&

$

%

NewtonalsousedbinomialtheoremandFTCtoestimateπ:

π/4:areaofquadrant:

∫

1

0

√

1−x2dx=

∫

1

0

(

1−1

2x

2−

1

4x

4−

1

16x

6−

5

128x

8−...

)

dx

10

'

&

$

%Figure5:NewtonandLiebniz:whowasfirst?

11

'

&

$

%

d’Alembert(1717-1783),Euler(1707-1783),D.Bernoulli

(1700-1782),Lagrange(1736-1813)(c.late1700s):Vibratingstring.

∂2w

∂x2=1

c2∂

2w

∂t2

D’alembert:w(x,t)=F(x+ct)+E(x−ct)

Vibratingstring:methodofreflections.

12

'

&

$

%Figure6:D’Alembert,D.BernoulliandEuler:

13

'

&

$

%

Fourier(c.1807):TheorieAnalytiquedeChaleur.

∂u

∂t=a

∂2u

∂x2

Separationofvariables...

u(x,t)=∞∑

n=1

bke−(πk/L)

2atsin(2πkx/L)

PROVIDED

u(x,0)=

∞∑

k=1

bksin(πkx/L)

Sinesandcosines:Musicofthespheres...

14

'

&

$

%

Fourier(c.1807):TheorieAnalytiquedeChaleur.

•ParisInstitute(1811)–prizecompetition∼propagationof

heat.Fouriersubmits1807memoir(oneoftwoentries)

•Fourierwins,but...objectionsbyLagrangeandLaplace:“...

themannerinwhichtheauthorarrivesattheseequations...

leavessomethingtobedesiredonthescoreofgeneralityand

evenrigor.”

•Particularcontroversy:expansionsintrigonometricseries.

•ElectedtoAcademie(1817),publication(1822).

15

'

&

$

%Figure7:Fourierbeforeandafter1812

16

'

&

$

%Figure8:Lagrange(left)andLaplace:dubiousofFourier

17

'

&

$

%

Morehistory:

Basicproblem:Whatdoesitmeanforaseriesofsinesandcosines

toconvergetoafunctionf?

Dirichlet(1837):definitionofafunction

“Ifavariableyissorelatedtoavariablexthatwhenevera

numericalvalueisassignedtox,thereisaruleaccordingtowhich

auniquevalueofyisdetermined,thenyissaidtobeafunctionof

theindependentvariablex.”

18

'

&

$

%Figure9:Dirichlet:Firstseriousattempt(1928)

19

'

&

$

%

DuBoisReymond(1873):ContinuousfwithFourierseries

divergingatapoint.

Figure10:DuBois-Reymond

20

'

&

$

%

Mathematization:Tominimizeuncertainty

21

'

&

$

%

Gibbs(1839-1903):

Gibbs,J.W.”FourierSeries.”Nature59,200and606,1899.

Figure11:J.W.Gibbs:fatherofthermodynamics

22

'

&

$

%Figure12:WilliamThomson(1824-1907):usedheatequationto

estimatetheageoftheearth

23

'

&

$

%

Figure13:Michelson’sinterferometer

Fouriertransformspectrometer(FTS):Michelsoninterferometer

withmovablemirror.

Scanningmovablemirroroverdistance:producesinterference

patternencodingFT(spectrum)ofsource.

A.Michelson’s1880interferometer:thearmskeptsamelength,

interferenceifphasevelocitiesdiffer.

24

'

&

$

%

Michelson-Morley:disprovedetherhypothesis,setstagefor

relativity

DifficultyincomputingFouriertransforms:earlyinvestigators

guessedaspectrum,inverted,thencomparedtomeasured

interferogram.

Highresolutionspectrometersnotavailableuntilearly1950s.

25

'

&

$

%Figure14:MichelsonandMorleycomputedc

26

'

&

$

%Figure15:Michelson-Strattonanalyzer(c.1897):80Fouriercoeffi-

cients

27

'

&

$

%

Figure16:Nowyoucanbuildyourownanalyzer

28

'

&

$

%

Wiener(1894-1964)

Whilestudyinganti-aircraftfirecontrol,conceivedofconsidering

‘operator’partofthesteeringmechanism

Problemofprediction:‘project’futureontopast

Solutionofintegralequation...inspiredbyV.Bush’sdifferential

analyzer...

“devisedanapparatustorealizeinmetalwhatwehadfiguredout

onpaper...”

“...Wemadeanapparatuswhichtranslatedtheheightofapoint

aboveagivenbaselineintoanelectricalvoltage.”

29

'

&

$

%Figure17:Wiener:firstdiscoveredHeisenberg’suncertaintyprinci-

ple,amongothers

30

'

&

$

%

TheInformationAge

VonNeumann(1903-1957)

VonNeumann’sinterestincomputers(c.1944)differedfromthat

ofhispeersbyhisquicklyperceivingtheapplicationofcomputers

toappliedmathematicsforspecificproblems,ratherthantheir

mereapplicationtothedevelopmentoftables.

“Dissatisfiedwiththecomputingmachinesavailableimmediately

afterthewar,hewasledtoexaminefromitsfoundationsthe

optimalmethodthatsuchmachinesshouldfollow,andhe

introducednewproceduresinthelogicalorganization,the“codes”

bywhichafixedsystemofwiringcouldsolveagreatvarietyof

problems.”–Dieudonne

31

'

&

$

%Figure18:JohnnyvonNeumann:fasterthanthisherecomputer

32

'

&

$

%

ClaudeShannon(1916–2001)

AMathematicalTheoryofCommunication(1948)

Pre-Shannon:electromagneticwavestobesentdownawire.

Post-Shannon:astreamof1sand0ssentdownawire

“Probablynosingleworkinthiscenturyhasmoreprofoundly

alteredman’sunderstandingofcommunicationthanCEShannon’s

article,“Amathematicaltheoryofcommunication...”–D.Slepian

33

'

&

$

%Figure19:ClaudeShannon:Fatherofinformationtheory

34

'

&

$

%

Shannon’ssamplingtheorem:

Iff(x)isΩ-bandlimited(ithasnofrequencyamplitudesaboveΩ/2

then

f(t)=∞∑

k=−∞

f(

k

Ω

)

sinc(t−k)

wheresinc(x)=sinπxπx.

Whatitsays:fcanberecoveredfrom(digital)samples.

35

'

&

$

%

CooleyandTukey(1965):FFTalgorithm

Allowed(discrete)Fouriertransformstobecomputedefficiently

usingarecursivealgorithmwhichcouldbeimplementedeasilyon

primitiveelectroniccomputers,

FFT:Orthogonaltransformmappingsamplesto‘frequencies’.

FAST:O(NlogN)

36

'

&

$

%Figure20:Tukey(left)andCarleson:breakthroughandresolution

37

'

&

$

% Figure21:Gauss:firstinventedtheFFT,amongothers

38

'

&

$

%

BeyondFourier

Orthogonaltransformations:

DatasequenceX(samples):vectorin‘highdimensionalspace’

OrthogonalmatrixO:rows(columns)areorthonormalor

uncorrelated:O∗=O

−1:rotation

OisgoodforXifX:Xcorrelatedwithasmallsubsetofrows

Imagecompression:imageAasmatrixO:eigenvectorsofA.

AiscompressibleifeigenvaluesofA∗Adecayrapidly.

SVD:bestpossibletransformationforA,notgoodforanythingelse

39

'

&

$

%

Whatisawavelet?

Scalingfunction:

1

2ϕ(

x

2

)

=∑

k

hkϕ(x−k)

H(ξ):Fourierseriesofhk

Quadraturemirrorfilter:

|H(ξ)|2+|H(ξ+1/2)|

2=1

Orthogonalfilter:G(ξ):gk=(−1)kh1−k

Wavelet:

1

2ψ(

x

2

)

=∑

k

gkϕ(x−k)

40

'

&

$

%

Thefunctions2j/2ψ(2

jx−k)formanorthonormalbasis

FastWaveletTransform:Computing(discrete)expansioninthis

basisisO(N).

41

'

&

$

%

0.20.40.60.81−0.3

−0.2

−0.1

0

0.1

0.2 Mother D4 Wavelet

0.20.40.60.81−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2 Mother S8 Symmlet

0.20.40.60.81−0.05

0

0.05

0.1

0.15

0.2 Father D4 Wavelet

0.20.40.60.81−0.05

0

0.05

0.1

0.15 Father S8 Symmlet

Figure22:Fatherandmotherwavelets

42

'

&

$

%

0.20.40.60.81−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15 Haar Wavelet

0.20.40.60.81−0.3

−0.2

−0.1

0

0.1

0.2 D4 Wavelet

0.20.40.60.81−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2 C3 Coiflet

0.20.40.60.81−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2 S8 Symmlet

Figure23:Moremotherwavelets

43

'

&

$

%

00.10.20.30.40.50.60.70.80.910

1

2

3

4

5

6

7

8

9Some S8 Symmlets at Various Scales and Locations

(3, 2)

(3, 5)

(4, 8)

(5,13)

(6,21)

(6,32)

(6,43)

(7,95)

Figure24:Theseareallorthogonal

44

'

&

$

%Figure25:MallatandMeyerinvented‘MultiresolutionAnalysis’

45

'

&

$

%Figure26:IngridDaubechies–themotherofwavelets

46

'

&

$

%

original image

50100150200250

50

100

150

200

250

reconstruction from largest eigenvalues

50100150200250

50

100

150

200

250

Figure27:Zebra:originalandSVDcompressed–top10percent

47

'

&

$

%

original image

50100150200250

50

100

150

200

250

reconstruction from large Fourier coefficients

50100150200250

50

100

150

200

250

Figure28:Zebra:originalandFouriercompressed–top10percent

48

'

&

$

%

original image

50100150200250

50

100

150

200

250

reconstruction of Zebra from largest wavelet coefficients

50100150200250

50

100

150

200

250

Figure29:Zebra:originalandwaveletcompressed–top10percent

49

'

&

$

%

Waveletpackets:thenextstepoftheevolution.

00.10.20.30.40.50.60.70.80.910

2

4

6

8

10

12Some Wavelet Packets

(6, 0, 1)

(6, 1, 8)

(6, 2,12)

(6, 3, 4)

(6, 4, 6)

(6, 5,14)

(6, 6,10)

(6, 7, 2)

(6, 8, 3)

(6, 9,11)

00.10.20.30.40.50.60.70.80.910

1

2

3

4

5

6

7

8Some Cosine Packets

(1, 0,64)

(2, 1,32)

(3, 2,16)

(4, 4, 8)

(5, 8, 4)

(6,16, 2)

(7,32, 1)

Figure30:mixtureofFourierandwaveletmodes

50

'

&

$

%Figure31:Coifman-suggestsweneedanewnumbersystem

51

'

&

$

%Figure32:George:wouldhavelikedPythagoras

52

'

&

$

%Figure33:Donoho:Edgelets,wedgelets,ridgelets,curvelets,what’s-

next-lets

53

'

&

$

%

Curvelets

54

'

&

$

%

Figure34:TheartistformerlyknownasBigMac

55

'

&

$

%Figure35:CanBigMaccomeback?

56