II. Nonparametric Spectrum Estimation for Stationary...

04/01/09 EC4440.SpFY09/MPF -Section II 1

II. Nonparametric Spectrum Estimationfor Stationary Random Signals

- Non-parametric Methods --

[p. 3] Periodogram

[p. 12] Periodogram properties-

[p. 23] Modified periodogram

[p. 25] Bartlett’s method-

[p. 30] Blackman-Tukey

procedure

- [p. 33] Welch’s procedure

- [p. 36] Minimum Variance (MV) Method- [p. 42] MultiTaper

Method (MTM)

-Goal: To estimate the Power Spectral Density (PSD) of a wss

process

-Two main types of approaches exist:-Non parametric methods: do not rely on data model

-based on FT and FFT, or

-Parametric methods: -based on different parametric models of the data:

-Moving Average: MA, -AutoRegressive: AR, -AutoRegressive

& Moving Average: ARMA

-based on subspace methods

I. Classical methodsuse FT of the data sequence or its correlation sequence

1) Periodogram• Easy to compute• Limited ability to produce accurate PS estimate

Recall PSD defined as:

( ) ( )j j kx x

P e R k eω ω∞

=−∞

In practice: Rx (k) is approximated with estimated correlation sequence ( )xR k

Recall: Biased correlation estimate of x(n), n=0,…N-1, defined as

( ) ( ) ( )1

1ˆ 0 1N k

R k x k x k NN

− −

= + ≤ < −∑

10% L N≤

( ) ( )ˆ ˆL

j j kx

k LP e R k e L Nω ω−

= ≤∑

( ) ( )1

Assume L=N-1

ˆ ˆN

j j kx

k NP e R k eω ω

−−

=− +

= ∑ periodogram

PSD estimate sometimes known as correlogram

( ) ( ) ( ) ( )*1ˆ * for 0 0 1xR k x k x k x k k NN

= − ≠ ≤ < −

( ) ( ) ( )

( ) ( )

jx N N

j jN N

P e FT x k x kN

X e X eN

⎡ ⎤⇒ = −⎢ ⎥⎣ ⎦

( ) ( ) 21ˆ j jx NP e X e

Nω ω=

Periodogram defined in terms of data directly

Recall

( ) ( ) ( ) ( )DFTN Nx n x n w n X k= ⋅ ⎯⎯⎯→

Define:

How to Compute the Periodogram

( ) ( ) ( ) ( )( )( ) ( )

22 1ˆ = , 0,... 1

j k Nx N

x n x n w n X k

P e X k k NN

= ⋅ ⎯⎯⎯→

Rectangular window of length N

Zero padding xN (n) in the time-domain increases the number of frequency PSD points available

Example: x(n) white noise

[Hayes]

( ){ } ( )

( ){ }

E P e E R k e

E R k e

−−

= − +

⎧ ⎫• = ⎨ ⎬⎩ ⎭

( ){ } ( ) ( )

( ) ( ) ( )

E R k E x k xNN k R k

NN k k N

R k w k w k Now

− −

⎡ ⎤• = +⎣ ⎦

−⎧ ⎫≤⎪ ⎪= ⋅ = ⎨ ⎬⎪ ⎪⎩ ⎭

Average behavior of the periodogram:

( ){ } ( ){ }1

ˆ ˆN

j j kx x

k NE P e E R k eω ω

−−

= − +

• = ∑

( ){ } ( ) ( ) ( )| |

x x B B

N k k NE R k R k w k w k N

−⎧ ⎫≤⎪ ⎪• = ⋅ = ⎨ ⎬⎪ ⎪⎩ ⎭

( ){ } ( ) ( )( )1

j j kx x B

k NE P e R k w k eω ω

−−

=− +

⇒ = ∑

( ){ } ( ) ( )1ˆ *2

j j jx x BE P e P e W eω ω ω

21 sin( /2)

sin( / 2)N

ω⎡ ⎤⎢ ⎥⎣ ⎦

[Hayes]

( ) ( ) ( )0sinx n A n v nω φ= + +

{ }0 0

2( ) ( )2

1( ) ( ) ( )2

( ) ( )4

j j jx x B

j jv B B

E P e P e W e

A W e W e

ω ω ω

ω ω ω ω

σ + −

⎡ ⎤= + +⎣ ⎦

Example: x(n) 1 tone in white noise

x(n) = sin(2π*150*t) + 2*sin(2*π*140*t) + 0.1*randn(size(t)); fs

Example: set of tones in white noise x(n)

periodogram(xn,[],'twosided',1024,fs); plots the two sided periodogram(note: better to stick to one-sided when dealing with real-data)

Obtained using the MATLAB function

periodogram.m

[Mathworks]

Properties of the Periodogram1)

Periodogram is biased when dealing with finite windows2)

Periodogram is asymptotically unbiasedi.e.,

lim ( ) ( )j jxN xE P e P eω ω

→+∞⎡ ⎤ =⎣ ⎦

3) Periodogram has limited ability to resolve closely spaced narrowband components (limiting factor due to the rectangular window, the shorter the data length, the worse off) (not good)4) Values of the periodogram spaced in frequency by integer multiples of 2π/N are approximately uncorrelated . As data length increases, rate of fluctuation in the periodogram increases (not good).5) Variance of the periodogram does not decrease as data length increases (not good).

In statistical terms, it means the periodogram is not a consistent estimator of the PSD. Nevertheless, the periodogram can be a useful tool for spectral estimation in situations where SNR is high, and especially if the data is long.

= sin(2*pi*150*t) + 2*sin(2*pi*140*t) + 0.1*randn(size(t));; fs=1000; nfft=1024;

[MathWorks]

CommentsPeriodogram has limited ability to resolve closely spaced narrowband components (limiting factor due to the rectangular window, the shorter the data length, the worse off) (not good)

67 samples

100 samples

[Hayes]

Comments:Variance of the periodogram does not decrease as the data length increases (not good).

For white noise x(n) one can show [Therrien, p.590]

lim var 0

var ( )

→+∞⎡ ⎤ ≠⎣ ⎦

⎡ ⎤ =⎣ ⎦

04/01/09 EC4440.SpFY09/MPF -Section II 15[Kay]

Comments: As data length increases, rate of fluctuation in the periodogram increases (not good).

Periodogram can be viewed as obtained by using a filterbank structure.

Recall periodogram defined for x(n), n=0,…N-1, as:

( ) ( ) ( ) ( )

( )( ) ( )

1DFT 2 /

1ˆ = , 0,... 1

Nj kp N

N N Np

j k Nx N

x n x n w n X k x p e

P e X k k NN

−−

= ⋅ ⎯⎯⎯→ =

Connections between Periodogram and filterbank

For x(n) defined over a finite time window {x(n-N+1),…x(n)}

( ) ( ) ( ) ( )1

DFT 2 /

Nj kp N

N N Np

x n x n w n X k x n p e π−

= ⋅ ⎯⎯⎯→ = −∑( )h p

( ) ( ) ( ) ( )1

0( ) ( )

N N Np

x n x n w n X k x n p h p−

= ⋅ ⎯⎯⎯→ = −∑k=0, …N-1

For x(n) defined over a finite time window {x(n-N+1),…x(n)}

( ) ( ) ( ) ( )1

0( ) ( )

N N Np

x n x n w n X k x n p h p−

= ⋅ ⎯⎯⎯→ = −∑

2xN (n) ( )( )2ˆ j k NxP e π

Filter ( )1( ) j kp N

h p eN

Periodogram DFT filterbank

k=0, …N-1( )( ) ( ) 22 1ˆ = j k Nx NP e X k

xN (n)

2( )( )2 0ˆ j N

xP e π

Filter ( )1( ) , 0

h p kN

2 ( )( )2ˆ j NxP e π

- 2 /1

Filter ( )1( ) , 1j p N

h p e kN

( )( )2 ( 1)ˆ j N NxP e π −

- 2 ( 1) /1

Filter ( )1( ) , 1j p N N

h p e k NN

π −− = = −

( )( ) ( ) 22 1ˆ = j k Nx NP e X k

xN (n)

2( )( )2 0ˆ j N

xP e π

Filter ( )1( ) , 0

h p kN

2 ( )( )2ˆ j NxP e π

- 2 /1

Filter ( )1( ) , 1j p N

h p e kN

( )( )2 ( 1)ˆ j N NxP e π −

- 2 ( 1) /1

Filter ( )1( ) , 1j p N N

h p e k NN

π −− = = −

Lowpass filter

Modulation term

( )( ) ( ) 22 1ˆ = j k Nx NP e X k

xN (n)

2( )( )2 0ˆ j N

xP e π

Filter ( )1( ) , 0

h p kN

2 ( )( )2ˆ j NxP e π

- 2 /1 0

Filter ( )( ) ( ) , 1j p N

H zh p h p e kπ= =

( )( )2 ( 1)ˆ j N NxP e π −

- 2 ( 1) /1 0

Filter ( )( ) ( ) , 1j p N N

H zh p h p e k Nπ −

− = = −

Lowpass filter

Modulation term

( )( ) ( ) 22 1ˆ = j k Nx NP e X k

2 / 2 ( 1) /

2 ( 1) / 2 ( 1) /

1 1 1(0)1(1)

( 1) 1

1/ N ( )( 1)1/ N

( 1)1/ N

Nj N j N N

j N N j N NN

X N e e

x nx n

− − −

− − − −

⎡ ⎤⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥ = ⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥− ⎢ ⎥⎣ ⎦ ⎣ ⎦

⎛ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥−⎢ ⎥ ⎢ ⎥× ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥− +⎢ ⎥ ⎣ ⎦⎣ ⎦

⎞⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

Periodogram DFT matrix formulation

( )( ) ( ) 22 1ˆ = j k Nx NP e X k

May be viewed as a rectangular window applied to data before FT operation

2. Modified Periodogram─

can apply a general window to x(n) prior to computing the periodogram (periodogram extension)

( ) ( ) ( ) 21ˆ j jnx R

P e x n w n eN

ω ω+ ∞

= − ∞

( ) ( ) ( ) 21ˆ *2

j j jx x RE P e P e W e

Nω ω ω

π⎡ ⎤ =⎣ ⎦

Note: amount of smoothing is determined by the window type

Matlab implementation:Hs = spectrum.periodogram('Hamming'); create modified periodogram objectpsd(Hs,xn,'Fs',fs,'NFFT',1024,'SpectrumType',‘onesided') plot

• Example:

[MathWorks]

Note: lower Hamming window sidelobes, but wider mainlobes

with the Hamming window than the rect. window

= sin(2*pi*150*t) + 2*sin(2*pi*140*t) + 0.1*randn(size(t));; fs=1000; nfft=1024; data length=100.

Hamming window Rectangular window

3. Bartlett’s Method

( ) ( ) ( )1

1ˆ ˆD efine: K

kj jB x

kP e P e

Kω ω

average several periodograms (rectangular window applied to each)

leads to consistent estimate of the PSD

where K: number of segments in the data.

pointsL

pointsL points

…n n n

(n) xK (n)

1( ) | ( ( 1) ) |K L

k nP e x n k L e

KLω ω

−−

= + −∑ ∑

( ) ( ) ( )

1ˆ ˆvar var

kj jB x

P e P eK

⎡ ⎤⎡ ⎤⎣ ⎦ ⎣ ⎦

⎡ ⎤⎣ ⎦

Examples: ─

white noise

[Hayes]

two sinusoids in noise

04/01/09 EC4440.SpFY09/MPF -Section II 27[Hayes, Fig. 8.9]

Compare: basic periodograms

Periodogram of white noise with variance 1.

Overlay 50 plots, N=64 & b) average

Overlay 50 plots, N=128 & d) average

Overlay 50 plots, N=256 & f) average

04/01/09 EC4440.SpFY09/MPF -Section II 28[Hayes, Fig. 8.14]

K: number of segmentsL: segment length

Spectrum estimation of white noise with unit variance

50-plot overlay, N=512, b) ensemble average.

Overlay of 50 Bartlett estimates with K=4 & L=128, N=512, d) ensemble average.

Overlay of 50 Bartlett estimates with K=8 & L=64, N=512, f) ensemble average.

Bartlett estimates

[Hayes, Fig. 8.15]( ) ( )1 2( ) sin 0.2 sin 0.25 ( )x n A n n v nπ φ π φ= + + + +

Note: Spectral peaks broadening, why?

Spectrum estimation of 2 tones in white noise

50-plot overlay, N=512, b) ensemble average.

Overlay of 50 Bartlett estimates with K=4 & L=128, N=512, d) ensemble average.

Overlay of 50 Bartlett estimates with K=8 & L=64, N=512, f) ensemble average.

4. Blackman-Tukey (BT) Procedure

( ) ( ) ( )ˆ ˆM

j j kBT x

k MP e w k R k eω ω−

apply windowing to correlation samples prior to transformation to reduce variance of the estimator

where the window w(k) length is much smaller than data length, w(k) = 0 for |k| > M where M << N

Maximum recommended window length M<N/5

04/01/09 EC4440.SpFY09/MPF -Section II 31[Kay, Fig.4.9]

( ) 10 exp(2 (0.15) ) 20 exp(2 (0.2) ) ( )x n j n j n w nπ π= + +

Periodogram BT PSD estimateTriangular window, length=4

Data length=20

( ) ~ (0,0.1)w n N

( ) 10 exp(2 (0.15) ) 20 exp(2 (0.2) ) ( )x n j n j n w nπ π= + +

Periodogram

Data length=100

( ) ~ (0,0.1)w n N BT PSD estimateTriangular window, length=20

[Kay, Fig.4.10]

5. Welch’s Procedure─

Modification to the Bartlett procedure: Averaging modified periodograms

Dataset split into K possibly overlapping segments of length L

window applied to each segment

all K periodograms are averaged

( ) ( ) ( )1

1ˆ ˆK

kj jw x

kP e P e

Kω ω

( ) ( ) ( ) ( )1 2

k k j nx

nP w n x n e

−−

= ∑window data x(n) located in Kth

segment

MATLABHs = spectrum.welch('rectangular', segmentLength, overlap %); define Welch spectrum objectpsd(Hs,xn,'Fs',fs,'NFFT',512); plot

Note: what do we gain with the Welch’s method?Reduction in spectral leakage that takes place thru

the sidelobes of the data window

[Hayes]

Performance comparison

variability resolution Figure of merit

Periodogram 1 0.89 (2π/N) 0.89 (2π/N)

Bartlett 1/K 0.89 K(2π/N) 0.89 (2π/N)

Welch (50% overlap, triang. window)

9/8K 1.28 (2π/L) 0.72 (2π/N)

BT 2M/2N 0.64 (2π/M) 0.43 (2π/N)

[Table 8.7, Hayes]

{ }{ }2

Definitions:

var ( )variability ( )

figure of merit variability resoluti

normalized variance

should be as small on

.resolution mainlobe width a

as possibt its 1/2

power (6dB) down level

K: number of data segmentsN: overall data lengthL: data section lengthM: BT window length

All schemes limited by data length

6. Minimum Variance (MV) Spectrum Estimate – Capon PSD

PSD estimated by a filterbank where each filter of length P<=N (data length) adapts itself to data characteristics, i.e.,

has center frequency fi (spread between 0 and fs

has bandwidth Δ ( P filters spread over [0,fs

] Δ∼1/fs )-

depends on Rx-

designed so components at fi are passed without distortionand rejects maximum amount of out-of-band power

[1, ,..., ]

j j P T

PP ee R e

[Manolakis, p.474]

( ) ( 1)1

ˆ , [1, ,..., ]j j j P TMV H

PP e e e ee R e

ω ω ω−−

MATLAB implementation

= ( ) ( )

H H HH Hx

e R e e U U e e U U e

e U e U

− − −

= Λ = Λ

[1, ,..., ]

= FT( ), , FT( )

H j j PP

e U e e u u

ω ω− − −

⎡ ⎤⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

[ ][ ]

2 211 P

( ) ( )

FT( ), , FT( )

= (1/ )|FT( ) | +(1/ )|FT( ) |

= (1/ )|FT( ) |

H H H Hx

e R e e U e U

e R e u

− −

⎡ ⎤+⎣ ⎦

⇒ ∑

( ) P2

ˆ(1/ )|FT( ) |

Application:2 (0.1) 2 (0.12( ) 10( ) ( ), ( ) ~ (0,1)j n j nx n e e w n w n Nπ π= + +

[Manolakis, ex. 9.5.1]

Filter order M=PData length N=500100 averaged trials

MV: solid lineBartlett method: small dashed lineAll-pole: large dashed line

0.02Wind. Length ~ 50

fΔ =⇒

MATLAB Implementation

R=covar(x,P);[v,d]=eig®;UI=diag(inv(d)+eps);VI=abs(fft(v,nfft)).^2;Px=10log10(P)-10log10(VI*UI);

In practice, -

pick P <N

zeropad

FFT to insure accurate

measurements

7. MultiTaper Method (MTM) Spectrum Estimate (Thompson)

Periodogram can be viewed as obtained by using a filterbank structure, (using rectangular windows).

•Multitaper

method (MTM) can be viewed the same way, with different filters, i.e., different windows. These optimal FIR filters are derived from a set of sequences known as discrete prolate spheroidal sequences.

•A set of independent estimates of the power spectrum is computed, by pre-

multiplying the data by orthogonal tapered windows designed to minimize spectral leakage.

[Mathworks, statistical signal processing, spectral analysis, non parametric methods]

[Victorin]

7. MultiTaper Method (MTM) cont’•

PSD variance is reduced by averaging over different tapered windows using the full data. Since the data length is not shortened, the overall PSD bias is smaller than that obtained when splitting data in segments.

MTM method provides a time-bandwidth parameter, NW, with which to balance the variance and resolution. NW is directly related to the number of tapers used to compute the spectrum. There are always 2*NW-1 tapers used to form the estimate.

• As NW increases,-

there are more estimates of the power spectrum, and the variance of the estimate decreases, -

each estimate exhibits more spectral leakage (i.e., wider peaks) and the overall spectral estimate is more biased.

For each data set, there is usually a value for NW that allows an optimal trade-off between bias and variance.

[Mathworks, statistical signal processing, spectral analysis, non parametric methods]

= 1000Hz; N=1000; 2 sines, Amplitude = [1 2]; Sinusoid frequencies f = [150;140]; Hs1 = spectrum.mtm(4,'adapt'); psd(Hs1,xn,'Fs',fs,'NFFT',1024)

Periodogram

MTM, NW=4 MTM, NW=3/2

References:[Manolakis][Hayes] Statistical digital signal processing and modeling, M. M. Hayes, 1996, Wiley.[Kay] Modern spectral estimation, S. M. Kay, 1988, Prentice Hall.[Victorin] A. Victorin, Multi-Taper Method for Spectral Analysis and Signal Reconstruction of Solar Wind Data, MS Plasma Physics, Sept. 2007, Royal Institute of Technology, Sweeden, http://www.ee.kth.se/php/modules/publications/reports/2007/TRITA-

EE_2007_054.pdf

II. Nonparametric Spectrum Estimation for Stationary...

Documents

STATIONARY AUTOREGRESSIVE MODELS VIA A BAYESIAN ... · STATIONARY AUTOREGRESSIVE MODELS VIA A BAYESIAN NONPARAMETRIC APPROACH By Ramse´s H. Mena and Stephen G. Walker IIMAS–UNAM,

Nonparametric Test

Bootstrapping stationary invertible VARMA models in echelon … · 2011-04-20 · is rather unknown, nonparametric bootstrap techniques such as the “blockwise bootstrap” of K¨unsch

“Nonparametric” “Nonparametric” methodsmethods

Dependent nonparametric trees for dynamic hierarchical ... · evolving trees in the literature. There exist a variety of distributions over stationary trees [1, 14, 5, 13, 10], and

Nonparametric regression estimation for random ˝elds in a ... · errors are given by a stationary ˝eld of dependent random variables which show spatial interaction. We are most

Bayesian Nonparametric Covariance RegressionBayesian Nonparametric Covariance Regression model’s expressivity. A nonparametric Nadaraya-Watson kernel estimator was proposed by Yin

Nonparametric Notes

Chapter 5 Nonparametric Sequential Prediction of ...oti/portfolio/icpproject/20110104/ch5.pdfChapter 5 Nonparametric Sequential Prediction of Stationary Time Series LászlóGyo¨rfiandGyo¨rgyOttucsák

Nonparametric Counterfactual Predictions in Neoclassical ... · Nonparametric Counterfactual Predictions in Neoclassical Models . ... Nonparametric Counterfactual Predictions in Neoclassical

Nonparametric Statistics

FUNCTIONAL COEFFICIENT NONSTATIONARY … COEFFICIENT NONSTATIONARY ... stationary data and is well suited to a variety of models that ... pioneered a nonparametric kernel regression

Nonparametric Methods

Introduction to Nonparametric Analysis - SAS Support for Normality ... SAS/STAT software provides several nonparametric tests for ... Introduction to Nonparametric Analysis

Nonparametric Confidence Intervals: Nonparametric Bootstrap

Introduction to Nonparametric Estimation A.B.Tsybakovpluto.huji.ac.il/~pchiga/teaching/Nonparametric... · 2016-12-06 · Solutions to the exercises from \Introduction to Nonparametric

Nonparametric specification for non-stationary time series ... · tests; local linear regression; local stationarity; weak dependence; wild bootstrap 1. Introduction Speciﬁcation

Introduction to Nonparametric Statisticsshare.its.ac.id/pluginfile.php/2647/mod_resource/content/1/8._Nonpar_Stat.ppt...Nonparametric Statistics Nonparametric Statistics Fewer restrictive

Using Simulink in Signal Processing Applicationsfaculty.nps.edu/fargues/teaching/ec4440/SimulinkIntroFallFY12.pdf · Basic Simulink blocks discussed • How to: – 1) Specify configuration

Nonparametric Model-Based Reinforcement Learningpapers.nips.cc/paper/1476-nonparametric-model... · Nonparametric Model-Based Reinforcement Learning 1011 //If\