2/7/17  

1  

A6523 Modeling, Inference, and Mining �Jim Cordes, Cornell University�

•  Lecture  4  –  Fourier  transform  shi6,  convolu:on  theorem  examples  – DFT  of  complex  exponen:al  –  Stochas:c  processes  

•  Reading  –  See  web  page  tomorrow  

•  Webpage:        www.astro.cornell.edu/~cordes/A6523    

 

1  

Fourier Transforms�•  Examples  on  board  – Shi6  theorem  •  Finding  maximum  of  func:on  •  Shi6ing  discrete  data  

2  

2/7/17  

2  

Discrete Fourier Transform (DFT)The DFT of a uniformly spaced array of data {xn, n = 0, . . . , N � 1} is defined as

Xk = N�1N�1�

n=0

xn e�2⇥i nk/N

The inverse transform is

xn =N�1�

k=0

Xk e+2⇥i nk/N

which may be shown to have the correct normalization, etc. by substituting for Xk:

xn = CN�1�

k=0

Xk e+2⇥i nk/N

= CN�1�

k=0

N�1N�1�

n⇤=0

xn⇤ e�2⇥i n⇤k/Ne+2⇥i nk/N

= C N�1N�1�

n⇤=0

xn⇤N�1�

k=0

e2⇥i (n�n⇤)k/N

= C N�1N�1�

n⇤=0

xn⇤ N �nn⇤

= Cxn = xn for C ⇥ 1

1

The  FFT  is  simply  a  fast  algorithm  for  calcula4ng  the  DFT.      It  exploits  redundancies  in  the  exponen4al  when  N  is  factorable;  esp.  N  =  2M    but  any  prime  will  do.  

A  direct  calcula:on  of  the  DFT  requires  ~  N2  opera:ons.    The  FFT  requires  ~  NlogN  opera:ons.  

3  

• The normalization calculation expresses the orthogonality property of the basis functions(exponentials).

• In calculations of this kind, identifying the implied �-function is typical. Here it relied onsumming over products of basis functions.

• In other contexts, the �-function will arise from statistical independence of random vari-ables.

• What happens if the sampling of xn is not uniform? Orthogonality is broken. So what?

• Notation: we often designate that xn and Xk are Fourier transform pairs by writing

xn ⇤⌅ Xk

and we say that n and k are conjugate variables.

• n can be time, a spatial coordinate, a wavelength, anything.

• Extension to ND dimensions is trivial:– E.g. a 2D DFT of an N �M size object can be calculated as a series of M 1D-DFTs of

length N followed by N 1D-DFTs of length M

• From a systems point of view, the DFT is a linear operation and does not lose information.

• An alternative approach is to fit a sinusoidal model to the data using an assumed frequencyk/N . It can be shown that the DFT is the least-squares solution for the amplitudes ofthe sinusoids for all k. We will show this later on by using matrix algebra to solve forleast-squares solutions.

2 4  

2/7/17  

3  

Symmetry Properties of the DFT

Xk = N�1N�1�

n=0

xn e�2⇥i nk/N and xn =

N�1�

k=0

Xk e+2⇥i nk/N

• Periodic with period N in both domains

• Time series:

– Discrete functions with sample intervals �t and �f

– T = N�t = time total time span– �f = 1/T

– Nyquist frequency = maximum frequency that is represented without distortion:

fN =N�f

2=

1

2�t

3 5  

Symmetry Properties of the DFT

Xk = N�1N�1�

n=0

xn e�2�i nk/N and xn =

N�1�

k=0

Xk e+2�i nk/N

• Hermitian (show by substituting into the DFT expression for xn)

x⇥n ⌅⇧ X⇥N�k

If xn is real then

xn ⌅⇧ X⇥N�k

X⇥N�k = Xk

• The symmetry properties tell us how to fill an array with data to achieve specific results

• What are the symmetry properties of a 2D DFT?

Xkl =1

NM

�

n

�

m

xnme�2�i (nk+ml)/NM

4 6  

2/7/17  

4  

How do we fill an array to get a real signal in the other domain?

•  k = 0, 1, …, N-1 • Need to know N/2-1+2 = N/2+1

unique values

0 1 2 3 N/2-1 N/2 N/2+1 N-2 N-1 N … … X

N⌘

X0 X1 ⌘ XN�1

X2 ⌘ XN�2

unique values

7  

Gaussian  example  

DFT of a Complex Exponential + NoiseConsider a time series

xn = A ei⇥o n�t + nn, n = 0, . . . , N � 1

where nn is complex white noise.

What are the properties of white noise? By definition, white noise has a flat spectrum. But this meansflat in the mean. Mean over what? Over a statistical ensemble. What is an ensemble?

We will define these later. But it needs to be clear that we have one realization of data that is conceptu-ally part of an ensemble of all possible realizations.

3

8  

2/7/17  

5  

0 200 400 600 800 1000Frequency Index t

�0.02

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Spe

ctru

m

NegativeFrequencies

PositiveFrequencies

N = 1024 P = 4.0 samples (S/N)t = 0.500

�0.6 �0.4 �0.2 0.0 0.2 0.4 0.6Frequency (Hz)

�0.02

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Spe

ctru

m

10�510�410�310�210�1100

log

Sy

Zoom in

9  

Mapping Frequency to Frequency BinConsider

x(t) = ei⇤0t = e2⇥if0t

In an N-point DFT, in which bin does the signal fall (mostly)?

As before we have T = N�t and �f = 1/T

The frequency mapping is

fj = j�f for j = 0, . . . , N/2

= (N � j)�f for j = N/2 + 1, . . . , N � 1

The Nyquist frequency is the maximum frequency in the spectrum

fN =1

2�t

So for f0 ⇥ fN we the corresponding frequency bin k in the DFT is

k0 = f0N�t

Negative frequencies correspond to k above the Nyquist frequency.

Frequencies |f0| > fN are still represented in the DFT (remember ... it is lossless) but they appear ataliased frequencies.

The frequency bin k0: varies with N for fixed f0 and �t and varies with �t for fixed f0 and N .

6 10  

2/7/17  

6  

1st and 2nd-order MomentsWe characterize the noise using the statistical moments. These are the first moment (the mean) and thesecond moment, the autocorrelation function (ACF).

Ensemble average moments are designated with angular brackets ⇧· · · ⌃:

The mean of the white noise is

⇧nn⌃ = 0 zero mean

and the ACF is written in terms of a Kronecker delta,

⇧nn n⇤m⌃ = ⇥2

n �nm white noise.

The �-form of the ACF is consistent with any pair of values nn and nm being statistically independent.

Note that we take a conjugate (*) because the noise is complex. One gets a different answer without theconjugate!

By definition (as we will see formally later on), the power spectrum is the Fourier transform of theensemble-average ACF where the ACF is assumed to depend only on the difference between the twotimes n and m. This is a property of stochastic processes that have stationary statistics.

For white noise we could write the ACF as

R(n,m) �⌅ R(n�m) = ⇥2n�(n�m)0.

The DFT of a delta function is a constant so we have shown that our definitions are consistent.

4

Ensemble  averages  technically  require  knowledge  of  an  N-­‐dimensional  PDF.    We  consider  cases  that  are  much  simpler.  

11  

Characterizing  the  noise  

We can calculate the DFT of the complex exponential because it is simply a geometric series. Usuallyit is not so simple!

The DFT of Xn is

Xk = N�1N�1�

n=0

Xn e�2⇥ink/N

= A N�1N�1�

n=0

ei (⇧0�t�2⇥k/N)n +N�1N�1�

n=0

nn e�2⇥ink/N

= A N�1 ei⌅nsin N

2 (⇧0�t� 2⇥k/N)

sin 12(⇧0�t� 2⇥k/N)

+ Nk

where ⌅n is an uninteresting phase factor and Nk is the DFT of the white noise.

The amplitude of the spectral line term is A (the limit where the arguments of the sin functions ⇤ 0).

The noise term Nk is a zero mean random process with second moment

⇧Nk N⇥k⌅⌃ = N�2

�

n

�

n⌅

⇧nn n⇥n⌅⌃ e�2⇥i(nk�n⌅k⌅)/N

= N�2�

n

�

n⌅

⇤2n �nn⌅ e

�2⇥i(nk�n⌅k⌅)/N

= (⇤2n/N

2)�

n

e�2⇥in(k�k⌅)/N

= (⇤2n/N) �kk⌅.

The second moment of the noise has the same form in both the time and frequency domains.

5 12  

2/7/17  

7  

We can calculate the DFT of the complex exponential because it is simply a geometric series. Usuallyit is not so simple!

The DFT of Xn is

Xk = N�1N�1�

n=0

Xn e�2⇥ink/N

= A N�1N�1�

n=0

ei (⇧0�t�2⇥k/N)n +N�1N�1�

n=0

nn e�2⇥ink/N

= A N�1 ei⌅nsin N

2 (⇧0�t� 2⇥k/N)

sin 12(⇧0�t� 2⇥k/N)

+ Nk

where ⌅n is an uninteresting phase factor and Nk is the DFT of the white noise.

The amplitude of the spectral line term is A (the limit where the arguments of the sin functions ⇤ 0).

The noise term Nk is a zero mean random process with second moment

⇧Nk N⇥k⌅⌃ = N�2

�

n

�

n⌅

⇧nn n⇥n⌅⌃ e�2⇥i(nk�n⌅k⌅)/N

= N�2�

n

�

n⌅

⇤2n �nn⌅ e

�2⇥i(nk�n⌅k⌅)/N

= (⇤2n/N

2)�

n

e�2⇥in(k�k⌅)/N

= (⇤2n/N) �kk⌅.

The second moment of the noise has the same form in both the time and frequency domains.

5

We  can’t  calculate  the  noise  DFT.    But  we  can  calculate  its  second  moment.    

O6en  but  not  always,  the  Fourier  transform  of  a  stochas:c  process  will  be  delta  correlated  

What  is  the  amplitude  of  the  signal  part  as  the  argument  of  the  sin()’s  à  0?  

What  does  the  signal  part  look  like?  

13  

Detection . . . or not?Suppose you have a data set that you think may have the form of the model given above. To answer thequestion “is there a signal in the data” we have to assess what are the fluctuations in the DFT (or, moreusefully, the squared magnitude of the DFT = an estimate for the power spectrum) due to the additivenoise. We would like to have confidence that a feature in the DFT or the spectrum is “real” as opposedto being a noise fluctuation that is spurious. To quantify our confidence, we need to know the propertiesof our test statistic. The following develops an approach that is applicable to the particular problem andillustrates generally how we go about assessing test statistics.

6 14  

2/7/17  

8  

Signal to noise ratio:

The rms amplitude of the noise term (in the frequency domain) is therefore ⇤N = ⇤n/⌃N and the

signal-to-noise ratio is

(S/N)DFT =line peak

rms noise=⌃N

A

⇤n.

Thus, the S/N of the line is⌃N larger than the S/N of the time series

(S/N)time series =amplitude of exponential

rms noise=

A

⇤n.

In practice, we must investigate the S/N of the squared magnitude of the DFT. Let ⌅0�t = 2⇥f0 �t =2⇥ ko/N so that the frequency is commensurate with the sampling in frequency space. Then Xk =A �kk0 + Nk and the spectral estimate becomes

Sk ⇥ |Xk|2 = |A �kk0 + Nk|2 (1)

= A2 �kk0 + A �kk0 (Nk + N �k ) + |Nk|2.

The ensemble average of the estimator is

⇤Sk⌅ = ⇤|Xk|2⌅ = A2 �kk0 + ⇤|Nk|2⌅ (2)

= A2 �kk0 + ⇤2n/N

The ratio of the peak to the off line mean is N A2/⇤2, consistent with (S/N)DFT calculated before.

7 15  

The Probability of False Alarm:

Suppose we want to test whether a feature in a spectrum is signal or noise. Let’s suppose that there isno signal (a ‘null’ hypothesis) in which case we can calculate the probability that a given amplitude isjust a noise fluctuation.

If there is only noise, the probability density function of Sk for any given k is a one-sided exponentialbecause Sk is �2

2:

fSk(S) =1

⇥Sk⇤e�S/⇥Sk⇤ U(S)

10

Why  is  the  spectrum  distributed  as  chi2    with  two  degrees  of  freedom?    

16  

2/7/17  

9  

Suppose there is a spike in the spectrum of amplitude �⌃Sk⌥

The noise-like aspect of Sk implies that there can be spikes above a specified detection threshold thatare spurious (“false alarms”). The probability that a spike has an amplitude ⌅ �⌃Sk⌥ is

P (S ⌅ �⌃Sk⌥) =� ⇧

�⌃Sk⌥ds fSk(s) ⇥ e��

If the DFT length is NDFT, there are NDFT unique values of the spectrum.

Note this is true for a complex process but not for a real one. Why?

The expected number of spurious (i.e. false-alarm) spikes that equal or exceed �⌃Sk⌥ is

Nspurious = NDFT e��

To have Nspurious ⇤ 1 we must have

NDFT e�� ⇤ 1

we need

� ⌅ lnNDFT

11 17  

NDFT � to have Nspurious � 1

128 4.9

1k 6.9

16k 9.7

1M 13.9

1G 20.8

1T 27.7

12

The  larger  the  number  of  trials,  the  higher  the  threshold  that  is  needed  to  have  a  specified  number  of  false  posi:ves.    There  are  never  zero  false  posi:ves!  

18