1

ROBUST COVARIANCESCommon Risk versus Specific Risk Outliers

5/30/2012

R. Douglas Martin

Professor of Applied MathematicsDirector of Computational Finance Program

University of Washingtondoug@amath.washington.edu

R-Finance Conference 2013Chicago, May 17-18

R Packages and Code Used

R robust package

PerformanceAnalytics package

Global minimum variance portfolios with constraints

– GmvPortfolios.r: gmv, gmv.mcd, gmv.qc, etc.

Backtesting

– btShell.portopt.r: btTimes, backtet.weight – gmvlo & gmvlo.robust.r

2

Robust Covariance Uses in Finance

Asset returns EDA, multi-D outlier detection and portfolio unusual movement alerts

– SM (2005), MGC (2010), Martin (2012)

Data cleaning pre-processing– BPC (2008)

Reverse stress testing– Example to follow

Robust mean-variance portfolio optimization– Is it usefull ???? If so, which method ????

3

4

Robust vs. Classical Correlations(Two assets in a larger fund-of-funds portfolio)

-0.0

40.0

00.0

4

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

1995 1996 1997 1998 1999 2000 2001

USHYHinDM

-0.0

40.0

00.0

4

Pfand

5

ROBUST CORR. = .65

A more realistic view of a lower diversification benefit!

CLASSIC CORR. = .30

What you get from every stats package. Gives anoverly optimistic view ofdiversification benefit!

Tolerance Ellipses (95%)

USHYHinDM

Pfa

nd

-0.04 -0.02 0.0 0.02 0.04

-0.0

4-0

.02

0.0

0.02

0.04

0.06

ROBUSTCLASSICAL

6

Hedge Fund Returns Example

-0

.05

0.00

0.05

F1

0.00

0.05

2000 2001 2002 2003 2004

F2

0.00

0.05

F3

0.0

0.1

F4

0.00

0.05

F5

-0.1

0.0

0.1

2000 2001 2002 2003 2004

F6

0.0

0.1

2000 2001 2002 2003 2004

F7

-0.0

50.

000.

05

F8

7

0.26

0.3

0.34

0.05

0.36

0.37

0.14

0.3

0.28

-0.13

0.32

0.35

-0.03

0.45

0.06

0.5

0.53

0.44

0.01

0.3

0.24

0.09

-0.3

0.16

-0.01

0.4

0.51 0.38

0.32

0.29

0.44

0.26

0.34

0.16

0.2

0.23

0.45

0.2

-0.15

-0.04

0.06

0.31

0.36

0.36

0.24

0.16

0.21

0.1

-0.01

0.13

0.02

-0.31

-0.19

0.26

0.28 0.33

F1

F2

F3

F4

F5

F6

F7

F8

F1

F2

F3

F4

F5

F6

F7

F8

ROBUSTCLASSICAL

Hedge Fund Returns Example

Portfolio Unusual Movement Alerts

Retrospective analysis

Dynamic alerts

8

2 1ˆˆ ˆt t td r μ Σ r μ

Mahalanobis Squared Distance (MSD)

Crucial to use a robust covariance matrix estimate ! Σ

9

Commodities Example

(see Appendix A of Martin, Clark and Green, 2009)

-0.2

0.0

0.2

CATTLE

2004 2005 2006 2007 2008 2009

HOGS COPPER

COFFEE SILVER

-0.2

0.0

0.2

SUGAR

-0.2

0.0

0.2

2004 2005 2006 2007 2008 2009

OJ PLANTINUM

2004 2005 2006 2007 2008 2009

OILC

10

Robust Alerts

Unreliable alerts!

Classical Alerts

1

2

3

4

5

6

7

0 10 20 30 40 50 60

CLASSICAL

57

0 10 20 30 40 50 60

ROBUST

9

11

28

50 52

57

58

59

60

Index

CO

MM

OD

ITY

RE

TU

RN

S D

IST

AN

CE

S

11

library(xts)library(robust)library(lattice)

ret = read.zoo("commodities9.csv",sep=",",header = T,format = "%m/%d/%Y")ret = as.xts(ret)ret = ret['2004-01-31/2008-12-31',]xyplot(ret,layout = c(3,3)) # Not the same as slidedata = coredata(ret)

cov.fm <- fit.models(CLASSICAL = covMLE(data), ROBUST = covRob(data,estim = "mcd",quan = .7))plot(cov.fm,which.plots = 3)

Robust Covariance Choices in R “robust”

Min. covariance determinant (MCD)

M-estimate (M)

Donoho-Stahel (DS)

Pairwise estimates (PW)– Quadrant correlation and GK versions– Positive definite (Maronna & Zamar, 2002)

For details see the R robust package reference manual. See also Chapter 10.2.2 of Pfaff (2013) Financial Risk Modeling and Portfolio Optimization with R, Wiley.

12

affine equivariant

not affine equivariant

13

(1 ) ( , ) t F N H r μ Σ

A natural model for common factor outliers– Market crashes

The Usual Robustness Outliers Model

tT nR rtable of returns with rows

1. Probability of a row containing an outlier is independent of the

dimension n, so the majority of the rows of are outlier-free.

2. Fraction of rows that have outliers is unchanged under affine transformations, so use affine equivariant estimators, e.g., MCD

Rr

14

Independent Outliers Across Assets (IOA)

Suppose for example that . Then

1,2, ,( ) , i i nP B the probability of a row not containing an outlier is ,

which decreases rapidly with increasing p . E.g., for : tr (1 )n

# of assets n 5 10 15 20

prob. clean row .77 .60 .46 .36

N.B. Affine transformations increase the percent of rows with outliers, so no need to restrict attention to affine equivariant estimators.

A natural model for specific risk outliers

(AKMZ, 2002 and AVYZ, 2009) 1 (0) i iB Let if asset is (is not) an outlier.

1 2 , , , ( )n i iB B B P B Assume are independent with

.05

Choice of Outliers Model and Estimator

Both are useful, but:

The usual outliers model handles market events outliers and for these an affine equivariant robust covariance matrix estimator will suffice, e.g., MCD.

The independent outliers across assets model is needed for specific risk outliers, and for these one may need to use a pairwise estimator to avoid breakdown!

Goal: Determine when pairwise robust covariance matrix estimator performs better than MCD, etc.

15

Asset Class & Frequency Considerations

Specific risk outliers are more frequent in the case of:

Higher returns frequency, e.g., weekly and daily Smaller market-cap stocks Hedge funds Commodities … ???

16

17

Non-Normality Increases with Frequency

-0.4

-0.2

0.0

0.2

0.4

-0.2 0.0 0.2 0.4

MONTHLY11403

-0.4

-0.2

0.0

0.2

-0.1 0.0 0.1 0.2

WEEKLY11403

-0.4

-0.3

-0.2

-0.1

0.0

0.1

-0.05 0.0 0.05

DAILY11403

-0.2

0.0

0.1

0.2

0.3

-0.2 -0.1 0.0 0.1 0.2

MONTHLY20220

-0.2

-0.1

0.0

0.1

0.2

-0.10 0.0 0.05 0.10

WEEKLY20220

-0.1

0-0

.05

0.0

0.0

50

.10

-0.06 -0.02 0.02 0.06

DAILY20220

Quantiles of fitted normal distribution

Qu

an

tile

s o

f s

am

ple

da

ta

Comparison of Non-normality over Different Time Scales

Outlier Detection Rule for Counting

optimal 90% efficient bias robust location estimate*

associated robust scale estimate*

Outliers: returns outside of

Probability of normal return being an outlier: 0.5%

* Use lmRob with intercept only in R package robust

18

ˆ ˆ ˆ ˆ( 2.83, 2.83)s s

s

19

Empirical Study of IOA Model Validity

Four market-cap groups of 20 stocks, weekly returns 1997 – 2010 in three regimes:

– 1997-01-07 to 2002-12-31

– 2003-01-07 to 2008-01-01

– 2008-01-08 to 2010-12-28

1. Estimate outlier probability for each asset, and hence the probability that a row is free of outliers under the IOA model.

2. Directly estimate the probability that a row has at least one outlier.

3. Compare results from 1 and 2 across market-caps and regimes.

i 11n

i i

row

20

-0.3

-0.1

0.1

0.3

PLX

S

-0.2

0.0

0.1

0.2

BW

INB

-0.2

0.0

0.1

0.2

HG

IC

2000 2005 2010

Index

-0.1

0.0

0.1

0.2

WT

S

SMALL-CAPS

4 of the 20 Small-Caps for Entire History

21

1 3 5 7 9 11 14 17 20

% OUTLIERS IN EACH ASSET

ASSETS

PE

RC

EN

T

02

46

8

2008 2009 2010 20110

24

68

1012

Index

CO

UN

T

# OF ASSETS WITH AN OUTLIER

Small-Caps Outliers in Third Regime

22

Large-Caps Outliers in Third Regime

2008 2009 2010 20110

510

15Index

CO

UN

T

# OF ASSETS WITH AN OUTLIER

1 3 5 7 9 11 14 17 20

% OUTLIERS IN EACH ASSET

ASSETS

PE

RC

EN

T

01

23

45

6

23

Evaluation of IOA Model for Weekly Returns

1997-01-07 to 2002-12-31 MICRO SMALL MID LARGE% Clean Rows IOA Model 32 39 46 59

% Clean Rows Direct Count 37 48 58 69

2003-01-07 to 2008-01-01 MICRO SMALL MID LARGE% Clean Rows IOA Model 33 46 52 57

% Clean Rows Direct Count 37 49 62 66

2008-01-08 to 2010-12-28 MICRO SMALL MID LARGE% Clean Rows IOA Model 23 31 39 46

% Clean Rows Direct Count 43 48 57 62

24

1.0

1.5

2.0

2.5

gmv.logmv.lo.mcdgmv.lo.qcmkt

Cum

ulat

ive

Ret

urn

Weekly Returns, Window = 60, Rebalance = Weekly

LONG-ONLY GMV,GMV.MCD, GMV.PW, MKT

-0.1

50.

00

Wee

kly

Ret

urn

1998-03-03 2001-07-03 2005-01-04 2008-07-01

Date

-0.5

-0.1

Dra

wdo

wn

25

1.0

1.5

2.0 gmv.lo

gmv.lo.mcdgmv.lo.qcmkt

Cum

ulat

ive

Ret

urn

Weekly Returns, Window = 60, Rebalance = Monthly

LONG-ONLY GMV,GMV.MCD, GMV.PW, MKT

-0.1

50.

00

Wee

kly

Ret

urn

1998-03-03 2001-07-03 2005-01-04 2008-07-01

Date

-0.5

-0.1

Dra

wdo

wn

26Time

0.40.50.60.70.80.9

dvi.gmv

0.40.50.60.70.80.9

dvi.gmv.mcd

0.40.50.60.70.80.9

1998 2000 2002 2004 2006 2008 2010

dvi.gmv.qc

HHI Diversification Index (sum-of-squared wts.)

27

library(PerformanceAnalytics)library(robust)source("GmvPortfolios.r")source("btShell.portopt.r")source("btTimes.r")

# Diversification Index Functiondvi =function(x){1-sum(x^2)}

# Input returnsret.all = read.zoo("smallcap_weekly.csv",sep=",",header = T,format = "%m/%d/%Y")mkt = ret.all[,"VWMKT"]ret = ret.all[,1:20]n.assets <- ncol(ret)

# get returns datesall.date = index(ret)

Back-Test Code

28

# compute the backtest timest.mw <- btTimes.mw(all.date, 4, 60)

# backtestingweight.gmv.lo <- backtest.weight(ret, t.mw,gmv.lo)$weightweight.gmv.lo.mcd <- backtest.weight(ret, t.mw, gmv.lo.mcd)$weightweight.gmv.lo.qc <- backtest.weight(ret, t.mw, gmv.lo.qc)$weight# The Diversification Index Plotsgmvdat = coredata(weight.gmv.lo)gmvdat.mcd = coredata(weight.gmv.lo.mcd)gmvdat.qc = coredata(weight.gmv.lo.qc)dvi.gmv = apply(gmvdat,1,dvi)dvi.gmv.mcd = apply(gmvdat.mcd,1,dvi)dvi.gmv.qc = apply(gmvdat.qc,1,dvi)dvi.all = cbind(dvi.gmv,dvi.gmv.mcd,dvi.gmv.qc)dvi.all.ts = as.zoo(dvi.all)index(dvi.all.ts) = index(weight.gmv.lo)xyplot(dvi.all.ts, scales = list(y="same"))

29

# compute cumulative returns of portfoliogmv.lo <- Return.rebalancing(ret, weight.gmv.lo)gmv.lo.mcd <- Return.rebalancing(ret, weight.gmv.lo.mcd)gmv.lo.qc <- Return.rebalancing(ret, weight.gmv.lo.qc)

# combined returnsret.comb <- na.omit(merge(gmv.lo, gmv.lo.mcd, gmv.lo.qc, mkt, all=F))

# return analysischarts.PerformanceSummary(ret.comb,wealth.index = T, lty = c(1,1,1,4),colorset = c("black","red","blue","black"), cex.legend = 1.3,cex.axis = 1.3, cex.lab = 1.5, main = "Weekly Returns, Window = 60, Rebalance = Monthly \n LONG-ONLY GMV,GMV.MCD, GMV.PW, MKT")

30

“Statistics is a science in my opinion, and it is no more a branch of mathematics than are physics, chemistry and economics; for if its methods fail the test of experience – not the test of logic – they will be discarded”

- J. W. Tukey

Thank You!

Proprietary, for use only by permission.

Thank You!

31

“Statistics is a science in my opinion, and it is no more a branch of mathematics than are physics, chemistry and economics; for if its methods fail the test of experience – not the test of logic – they will be discarded”

- J. W. Tukey

References Alqallaf, Konis, Martin and Zamar (2002). “Scalable robust covariance and

correlation estimates for data mining”, Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 14-23. ACM.

Scherer and Martin (2005). Modern Portfolio Optimization, Chapter 6.6 – 6.9, Springer

Maronna, Martin, and Yohai (2006). Robust Statistics : Theory and Methods, Wiley.

Boudt, Peterson & Croux (2008). “Estimation and Decomposition of Downside Risk for Portfolios with Non-Normal Returns”, Journal of Risk, 11, No. 2, pp. 79-103.

Alqallaf, Van Aelst, Yohai and Zamar (2009). “Propagation of Outliers in Multivariate Data”, Annals of Statistics, 37(1). p.311-331.

Martin, R. D., Clark, A and Green, C. G. (2010). “Robust Portfolio Construction”, in Handbook of Portfolio Construction: Contemporary Applications of Markowitz Techniques, J. B. Guerard, Jr., ed., Springer.

Martin, R. D. (2012). “Robust Statistics in Portfolio Construction”, Tutorial Presentation, R-Finance 2012, Chicago,

32