Statistika Kekar(Robust Statistics)

Dr. Kusman Sadik, S.Si, M.Si

Sekolah Pascasarjana

Departemen Statistika IPB, 2018

1

Pendahuluan

Deals with deviations from ideal models and

their dangers for corresponding inference

procedures.

Primary goal is the development of procedures

which are still reliable and reasonably efficient

under small deviations from the model.

In statistics, classical estimation methods rely

heavily on assumptions which are often not

met in practice.

2

There are various definitions of a "robust statistics“.

Strictly speaking, a robust statistics is resistant to

errors in the results, produced by deviations from

assumptions (e.g., of normality).

This means that if the assumptions are only

approximately met, the robust estimator will still have

a reasonable efficiency, and reasonably small bias,

as well as being asymptotically unbiased, meaning

having a bias tending towards 0 as the sample size

tends towards infinity.

Definisi (1)

3

One of the most important cases is

distributional robustness. Classical statistical

procedures are typically sensitive to

"longtailedness" (e.g., when the distribution of

the data has longer tails than the assumed

normal distribution).

Thus, in the context of robust statistics,

distributionally robust and outlier-resistant are

effectively synonymous.

Definisi (2)

4

Find the structure best fitting the majority

of the data;

Identify deviating points (outliers) and

substructures for further treatment;

In unbalanced situations : identify and

give a warning about highly influential

data points (leverage points).

Tujuan Metode Kekar

5

Robust statistics replaces classical statistics.

The normality assumption is "guaranteed“ by the

central limit theorem.

If the errors are non-normal, we change the

specification of the errors.

We use classical procedures after removing outliers.

Therefore we do not need any robust procedures.

Robust statistics cannot be used when the errors

are asymmetric.

Beberapa Kesalahfahaman

6

Robustness vs Diagnostics

Robustness Diagnostics

Its purpose is to

safeguard against

deviations from the

assumptions.

Its purpose is to find

and identify

deviations from the

assumptions.

7

Traditionally, statisticians would manually screen

data for outliers, and remove them, usually

checking the source of the data to see if the

outliers were erroneously recorded.

However, in modern times, data sets often

consist of large numbers of variables being

measured on large numbers of experimental

units. Therefore, manual screening for outliers is

often impractical.

Mendeteksi Pencilan

8

This problem worse as the complexity of the data

increases. For example, in regression problems,

diagnostic plots are used to identify outliers.

However, it is common that once a few outliers

have been removed, others become visible. The

problem is even worse in higher dimensions.

Robust methods provide automatic ways of

detecting, down weighting (or removing), and

flagging outliers, largely removing the need for

manual screening.

Mendeteksi Pencilan (2)

9

Breakdown point: Breakdown point of an estimator

is a fraction of a sample if changed arbitrarily that

does not affect the estimation significantly. For

example for mean value if we change one point

arbitrarily we can change the mean value as much

as we want.

Empirical influence function: It is a measure of the

dependence of the estimator on the value of one of

the points in the sample. It is a model-free measure

in the sense that it simply relies on calculating the

estimator again with a different sample.

Mengukur Kekekaran

10

Ilustrasi : Breakdown Point

Breakdown point of an estimator is a fraction of a sample if

changed arbitrarily that does not affect the estimation

significantly. For example for mean value if we change one point

arbitrarily we can change the mean value as much as we want.

Let us take an example:

-1.1 -0.8 -0.6 -0.5 -0.4 -0.3 -0.3 -0.3 0.1 0.2 4.0

The sample size is 11, the mean value is -0.09. If we change the

last value to 100 then the mean value becomes 8.72. So

breakdown point of mean is 1/n.

Another limiting case is median. Median of the above sample is

-0.3. If we change one value and make it extremely large then

median will not change much.

11

Ilustrasi : Breakdown Point (2)

-1.1 -0.8 -0.6 -0.5 -0.4 -0.3 -0.3 -0.3 0.1 0.2 4.0

For example if we change the last value to -100 then

median will become -0.4.

Breakdown point for median is 0.5, i.e. more than 50% of

the sample should be changed arbitrarily to change the

median arbitrarily. Breakdown point 0.5 is the theoretical

limit.

Efficiency of estimators with high breakdown point is

usually worse than those with lower breakdown point. In

other words variances of estimators with high breakdown

point are larger. 12

The median is a robust measure of central tendency, while the

mean is not.

The median absolute deviation and interquartile range are

robust measures of statistical dispersion, while the standard

deviation and range are not.

Trimmed estimators and Winsorised estimators are general

methods to make statistics more robust.

L-estimators are a general class of simple statistics, often

robust.

M-estimators are a general class of robust statistics, and are

now the preferred solution, though they can be quite involved

to calculate.

Contoh Penduga Kekar

13

Winsorised Estimator

14

Historically, several approaches to robust

estimation were proposed, including

R-estimators and L-estimators.

However, M-estimators now appear to

dominate the field as a result of their

generality, high breakdown point, and their

efficiency. See Huber (2009).

M-estimators

15

M-estimators are a generalization of maximum

likelihood estimators (MLEs). What we try to do

with MLE's is to maximize or, equivalently,

minimize.

Huber proposed to generalize this to the

minimization of, where is some function. MLE

are therefore a special case of M-estimators

(hence the name: "Maximum likelihood type"

estimators).

M-estimators (2)

16

M-estimators (3)

17

Bentuk penduga kekar yang pupuler:

1) Huber

2) Tukey’s bisquare

3) Geman and Mcclure

4) Welsch

5) t-distribution (actually it is a little bit modified form of –log t distribution)

(x)

x 2 /2, | x | k

k(| x | k

2) otherwise

(x) x 2

c 2 x 2

(x) c 2

2(1 ex

2 / c 2

)

(x) c 2

2log(1 x 2 /c 2)

(x)

c 2

6(1 (1 (

x

c)2)2) | x | c

c 2 /6 otherwise

18

Notice that M-estimators do not necessarily

relate to a probability density function. Therefore,

off-the-shelf approaches to inference that arise

from likelihood theory can not, in general, be

used.

It can be shown that M-estimators are

asymptotically normally distributed, so that as

long as their standard errors can be computed,

an approximate approach to inference is

available.

M-estimators (4)

19

Since M-estimators are normal only asymptotically,

for small sample sizes it might be appropriate to use

an alternative approach to inference, such as the

bootstrap.

However, M-estimates are not necessarily unique

(i.e., there might be more than one solution that

satisfies the equations).

Also, it is possible that any particular bootstrap

sample can contain more outliers than the

estimator's breakdown point. Therefore, some care

is needed when designing bootstrap schemes.

M-estimators (5)

20

Ilustrasi :

Regresi Kekar

21

Outliers and Regression

Let us remind us the form of the least-squares

equations for regressions. Again x is a vector of input

(predictor) parameters, β is a vector of parameters, y

is output, the number of sample points is n.

As we know in special case when g(x,β) =β, and β is

a single value then least-squares estimation gives

mean value of y. We can consider above estimation

as an extension of mean value estimation.

Breakdown point of this estimation is 0, so least-

squares is very sensitive to outliers.

There are several approaches to deal with outliers in

regression analysis. We will consider only two of

them: (1) Least-trimmed squares; (2) M-estimators

(y i g(x i,))2

i1

n

==> min

Regression: no outliers

Regression: with an outlier

Outlier

22

Least Trimmed Squares

Least trimmed squares works iteratively.

1) Set up initial values for the model parameters

(for example using simple least squares

method)

2) Calculate squared residuals ri2=(yi-g(xi,β))2

3) Sort squared residuals

4) Remove fraction of observations for which

squared residuals are large

5) Minimise least squares using these

observations only

6) Repeat (2)-(5) until convergence achieved.

The result of default LTS

23

Robust M-estimators

An extension of least-squares to deal with outliers is written as:

Form the function ρ defines various forms of robust M-estimators. When ρ(z)=z2 it becomes

simple least-squares.

Let us first this function. To minimise this function let us use Gauss-Newton method. To use

this method we need the first and second derivatives (more precisely an approximation for

the second derivative)

Where ρ’, ρ’’ are the first and the second derivative of ρ. In Gauss-Newton methods the

second term of the second derivative equation is usually ignored. Usually ρ’=ψ and ρ’’=w

notations are used. If we look at the equations we can see that it looks like an extension of

least-squares equations. The minimisation of the function is done iteratively using iteratively

reweighted least squares (IRLS or IWLS).

ψ function is an influence function. Analysis of values of this function at the observations

may help to understand outliers in the data and how are dealt with.

f () (yi g(xi,)i1

n

) min

n

iTii

n

iTiiT

n

i

ii

gxgy

ggxgy

f

gxgy

f

1

2

1

2

1

)),((')),((''

)),(('

24

Forms of Robust Regression

Robust M-estimators are usually chosen so that to make contribution of gradients for large residuals small, in other words to weight down large deviations. They can be chosen either using ρ or ψ.

Basic idea behind robust estimators is: For small differences behaviour of the function should be similar to that of least squares and for large deviations contributions should be weighted down. Different functions differ by degree of weighting.

Example of ρ and ψ (Geman

and Mcclure function)

25

Buku

Huber, P.J. and Ronchetti, E.M. 2009. Robust Statistics 2nd. John Wiley & Sons.

Marona, R., Martin, R.D., and Yohai, V. 2006. Robust Statistics: Theory and

Methods. John Wiley & Sons.

Rousseeuw, P.J. and Leroy, A.M. 2003. Robust Regression and Outlier Detection.

John Wiley & Sons.

Jurnal

He, X. and Portnoy, S. Reweighted LS Estimators Converge at the same Rate as

the Initial Estimator. Annals of Statistics Vol. 20, No. 4 (1992), p. 2161–2167

He, X., Simpson, D.G. and Portnoy, S., Breakdown Robustness of Tests. Journal

of the American Statistical Association Vol. 85, No. 40, (1990), p. 446-452

Portnoy S. and He, X. A Robust Journey in the New Millennium. Journal of the

American Statistical Association Vol. 95, No. 452 (Dec., 2000), p.1331–1335

Rousseeuw, P.J. and Croux, C. Alternatives to the Median Absolute Deviation.

Journal of the American Statistical Association Vol. 88 (1993).

Pustaka

26

27

Buku dan Catatan Kuliah

Bisa di-download di

kusmansadik.wordpress.com

28