Analysis of distorted waveforms using parametric spectrum

estimation methods and robust averaging

Zbigniew LEONOWICZ

13th Workshop on High Voltage Engineering Söllerhaus Austria 11-15.09.2006

Robust averaging

• Averaging is probably the most widely used basic Averaging is probably the most widely used basic statistical procedure in experimental science.statistical procedure in experimental science.

• Estimation of the location of data („central Estimation of the location of data („central tendency”) in the presence of random variations tendency”) in the presence of random variations among the observationsamong the observations

• Data variations can be a result of variations in the Data variations can be a result of variations in the phenomenon of interest or of some unavoidable phenomenon of interest or of some unavoidable measuring errors. measuring errors.

• In signal processing terms, this can be considered In signal processing terms, this can be considered as contamination of useful „signal” by useless as contamination of useful „signal” by useless „noise” linearly added to it. „noise” linearly added to it.

• Since the noise usually has Since the noise usually has zero meanzero mean, averaging , averaging minimizes its contribution, while the signal is minimizes its contribution, while the signal is preserved, and the preserved, and the signal to noise ratiosignal to noise ratio is is improvedimproved

Synchronization

• Averaging consists of applying of any Averaging consists of applying of any statistical procedure tostatistical procedure to extract the useful extract the useful information from the background noise. information from the background noise.

• When useful data When useful data are timeare time--locked to locked to some some event andevent and the the noise isnoise is not time not time--locked, locked, it it allows the cancellation of allows the cancellation of the noisethe noise by by simple pointsimple point--byby--point data summationpoint data summation..

• This procedure is equivalent to the use of This procedure is equivalent to the use of the arithmetic meanthe arithmetic mean

Review of robust avearging methods• SSensitivityensitivity of an estimator to the presence of of an estimator to the presence of

outliersoutliers (i.e. (i.e. data points that deviate from the data points that deviate from the pattern set by the majority of the datapattern set by the majority of the data setset))

• Robustness of an estimator is measured by the Robustness of an estimator is measured by the breakdown valuebreakdown value

• HHow many data points need to be replaced by ow many data points need to be replaced by arbitrary valuesarbitrary values in order to make the estimator in order to make the estimator explodeexplode (tend to in (tend to infifinity) or nity) or implodeimplode (tend to (tend to zero)zero) ? ?

• AArithmetic mean has 0% breakdownrithmetic mean has 0% breakdown• MMedian is veryedian is very robust with breakdown value robust with breakdown value

50%50%

Robust location estimators

• Many location estimators can be presented Many location estimators can be presented in uniin unifified way by ordering theed way by ordering the values of the values of the sample as sample as

and then applying the and then applying the weightweight functionfunction

• where where is a function designed to reduce is a function designed to reduce the inthe inflfluence of certainuence of certain observations (data observations (data points)points) in form of weighting and in form of weighting and represents orderedrepresents ordered data. data.

Examples

• MedianMedianWhenWhen the data have the size of (2 the data have the size of (2MM+1), the median +1), the median

is the value of the (is the value of the (M M +1)+1)thth ordered observation. ordered observation.

• Trimmed meanTrimmed meanFor the For the --trimmed mean (where trimmed mean (where p p = = NN) the weights ) the weights

can becan be d deefifined as:ned as:

p p highest and highest and p p lowest samples are removedlowest samples are removed..

Winsorized mean

• Winsorized mean replaces each Winsorized mean replaces each observation in each observation in each fraction (fraction (p p = = NN) of) of the tail of the distribution by the value of the tail of the distribution by the value of the nearest the nearest unaffectedunaffected observation. observation.

• 0 0 p p 00,,2525N N usuallyusually, depending on, depending on the the heaviness of the tails of the distribution.heaviness of the tails of the distribution.

Weight functions

Weight functions - other

• TL-mean applies TL-mean applies higher weights for the higher weights for the middle observationsmiddle observations

• tanh estimator tanh estimator appliesapplies smoothly smoothly changing weights to changing weights to the values close to the values close to extreme, it extreme, it can becan be set set to ignoreto ignore extreme extreme valuevaluess

Comparison

Investigations

• IEC harmonic and interharmonic IEC harmonic and interharmonic subgroups calculation IEC Std 61000-subgroups calculation IEC Std 61000-4-7, 61000-4-304-7, 61000-4-30

• DFT with 5 Hz resolution in frequency DFT with 5 Hz resolution in frequency characterize the waveform distortionscharacterize the waveform distortions

Parametric methods

• MUSICMUSICEEigenvalues of the correlation matrix igenvalues of the correlation matrix which which

correspond to the noise subspacecorrespond to the noise subspace used for used for parameter estimationparameter estimation

• ESPRITESPRITbased on naturally existing based on naturally existing shift invarianceshift invariance

between the discrete time series, which between the discrete time series, which leads to rotational invariance between the leads to rotational invariance between the corresponding signal subspaces. corresponding signal subspaces. Uses Uses signal subspace.signal subspace.

Progr. average of harmonic groups

• dc arc furnace supplydc arc furnace supply• 11th harmonic group11th harmonic group• 2nd interharmonic 2nd interharmonic

groupgroup

Results MSE

MethodMethod MSE groupsMSE groups MSE MSE subgroupssubgroups

DFTDFT 0.0590.059 0.7910.791

ESPRITESPRIT 0.0210.021 0.1690.169

MUSICMUSIC 0.0270.027 0.2010.201

Advantage of Winsorized mean

• When comparing values of power quality indices When comparing values of power quality indices obtained from different parts of the same recorded obtained from different parts of the same recorded waveform, a high variability of results appears. To waveform, a high variability of results appears. To alleviate this problem, winsorized mean alleviate this problem, winsorized mean was was appplied appplied to compute averages from to compute averages from spectral dataspectral data. . When using the value of a=0.2 which means that When using the value of a=0.2 which means that 20% of ordered data points were discarded and 20% of ordered data points were discarded and replaced by nearest unaffected data. replaced by nearest unaffected data.

• In such way the outliers were removed and In such way the outliers were removed and replaced by data, which are assumed to belong to replaced by data, which are assumed to belong to “true” spectral content of investigated waveform. “true” spectral content of investigated waveform.

• The use of winsorized mean instead of usual The use of winsorized mean instead of usual arithmetic mean allowed reducing the variance of arithmetic mean allowed reducing the variance of results by nearly results by nearly 35%.35%.

Conclusions

• Results show that the highest improvement Results show that the highest improvement of accuracy can be obtained by using the of accuracy can be obtained by using the ESPRIT method (especially for ESPRIT method (especially for interharmonics estimation), closely followed interharmonics estimation), closely followed by MUSIC method, which outperform by MUSIC method, which outperform classical DFT approach by over 50%. classical DFT approach by over 50%.

• Partially stochastic nature of investigated Partially stochastic nature of investigated arc furnace waveforms caused high arc furnace waveforms caused high variability of calculated power quality variability of calculated power quality indices. The use of robust averaging indices. The use of robust averaging (winsorized mean) helped to reduce this (winsorized mean) helped to reduce this unwanted variability.unwanted variability.

Conclusions

Trimmed estimators are a class of robust estimators Trimmed estimators are a class of robust estimators of data locations whichof data locations which can help to improve can help to improve averaging of averaging of experimental data experimental data whenwhen::

number of number of experimentsexperiments is small is smalldata are highly nonstationarydata are highly nonstationary

data data include outliers. include outliers. Their advantages can beTheir advantages can be understood as a reasonable understood as a reasonable

compromise between median which is very robustcompromise between median which is very robust but discard too much information and arithmetic but discard too much information and arithmetic mean conventionally used formean conventionally used for averaging which use averaging which use all data but, due of this, is sensitive to outliers. all data but, due of this, is sensitive to outliers.

AdditionalAdditional improvement of averaging can be gained improvement of averaging can be gained by introducingby introducing advanced advanced weighting of orderedweighting of ordered datadata