Expected Shortfall Backtest

8/10/2019 Expected Shortfall Backtest

1/89

Testing Expected Shortfall

C. Acerbi and B. Szekely

MSCI Inc.

Workshop on systemic risk and regulatory market risk measures

Pullach, Germany, June 2014

Carlo Acerbi and Balazs Szekely Testing Expected Shortfall June 2014 1 / 59
http://find/


2/89

Outline

1 Motivation and goals

2 Testing settingBaselVaRbacktest

Three tests forES. Plus one

3 Results

4 ConclusionsPost Scriptum

http://find/


3/89




3 Results


http://goforward/http://find/http://goback/


4/89

Motivation

in theVaR/ESdebate, backtesting has always been the main problemwithES. See for instance Yamai and Yoshiba (01)

last obstacle for the adoption ofESin BaselN, finally occurred in 2013

but model testing still based on VaR

rich literature onVaRbacktesting: Basel I (96), Kupiec (95),

Christoffersen (98), Berkowitz (00), Engle and Manganelli (04), amongothers

few works onESbacktesting: noticeably Kerkhof and Melenberg (04)Angelidis and Degiannakis (06)

Why is it difficult to test ES?

Fundamental reasons? Practical aspects? Power of the test? Model risk?

http://find/


5/89

Motivation

in theVaR/ESdebate, backtesting has always been the main problemwithES. See for instance Yamai and Yoshiba (01)

last obstacle for the adoption ofESin BaselN, finally occurred in 2013

but model testing still based on VaR

rich literature onVaRbacktesting: Basel I (96), Kupiec (95),

Christoffersen (98), Berkowitz (00), Engle and Manganelli (04), amongothers

few works onESbacktesting: noticeably Kerkhof and Melenberg (04)Angelidis and Degiannakis (06)

Why is it difficult to test ES?

Fundamental reasons? Practical aspects? Power of the test? Model risk?



6/89

Confusion

The nice thing about VaR is its more or less transparentlyback-testable. You know what youre getting. With ES its all cloudedup with assumptions about distribution and arbitrary choices. Whenhave you breached it? What exactly are you testing? When you gointo the tail you are never quite sure...

RISK Magazine, last week

http://find/


7/89

The drama of nonelicitability of ES

Gneiting (11): VaRis elicitable,ESis not

This negative result may challenge the use of the ES functional as a

predictive measure of risk, and may provide a partial explanation for thelack of literature on the evaluation of ES forecasts, as opposed to

quantile or VaR forecasts

elicitability is a subtle concept: x=arg minxE[S(x, Y)]

What most people understood

ESis not backtestable, at all

a magnum champagne bottle gift for the VaRnostalgic

panic followedES cannot be back-tested because it fails to satisfy elicitability ... If you

held a gun to my head and said: We have to decide by the end of the

day if Basel 3.5 should move to ES, or do we stick with VaR, I would

say: Stick with VaR

Paul Embrechts, Imperial College, 2013

http://find/http://goback/


8/89













say: Stick with VaR


http://find/


9/89













say: Stick with VaR




10/89













say: Stick with VaR


http://find/


11/89













say: Stick with VaR


http://find/


12/89













say: Stick with VaR

certainly not a VaR fanatic! Paul Embrechts, Imperial College, 2013

http://find/


13/89

Examples of elicitable statistics

the mean is elicitable

x=arg minm

EX[S(m, X)] S(m, x) = (X m)2

aquantile is elicitable

q

=arg minq

EX

[S(q, X)] S(q, x) = (x q)( (x q


14/89

Something is not quite right

if elicitable means backtestable isnt it a bit strange that

banks have always backtested VaRbut never by exploiting its elicitability?

even standard deviation is not elicitable?

Kerkhof and Melenberg, back in (04), had found that

...contrary to common belief, ES is not harder to backtest than VaR ifwe adjust the level of ES. Furthermore, the power of the test for ES is

considerably higher than that of VaR.

as a matter of fact, others reacted quite differently

ES is not elicitable. So, what? Dirk Tasche


S hi i i i h
http://find/


15/89











S thi i t it i ht
http://find/


16/89











S thi i t it i ht
http://find/


17/89











S thi i t it i ht
http://find/


18/89











http://find/


19/89










http://find/


20/89




3 Results



Setting
http://find/


21/89

Setting

we look atESbacktesting from a regulatory point of view

profitloss: independent (but not i.i.d.) XtFt, therealdistributions,t=1, . . . , T (=250)

Ptpredicted(model) distributions

VaRandES(with Basel confidence levels)

VaR =P1

() = 1%

ES =1

0

P1(q) dq = 2.5%

we assumePtcontinuous and strictly monotonic (just for simplicity,

inessential here). Then

ES =E[X|X+VaR


22/89

ESestimators

standard estimator ofES forN i.i.d. drawsXiP

ES,N(X) = 1N

[N]i

Xi:N+ (N [N]) X[N+1:N]

coherentN, , consistent, asymptotically normal, known variance

generalizes the idea of average of the Nworst cases toN / N

but biased. It always underestimates risk for finite N. No unbiasedestimator known for unknownP

conditional estimator; assumingVaR is known exactly

ES,N(X) =Ni=1 Xi1Xi+VaR


23/89

ESestimators

standard estimator ofES forN i.i.d. drawsXiP

ES,N(X) = 1N

[N]i

Xi:N+ (N [N]) X[N+1:N]

coherentN, , consistent, asymptotically normal, known variance

generalizes the idea of average of the Nworst cases toN / N

but biased. It always underestimates risk for finite N. No unbiasedestimator known for unknownP

conditional estimator; assumingVaR is known exactly

ES,N(X) =Ni=1 Xi1Xi+VaR


24/89

Hypothesis testing

Goal

testingVaRt andEStpredictions against observed profitloss realizationsxt

H0: Pt=FtH1: Ftis riskier thanPt

ES

F

t >ES

P

t

we test only in the direction of risk underestimation

more specificH1s in the following, for computing test power

Modelfree testWe avoid any assumption on the nature of the predicted distributions Pt (nolocation-scale family, no parametric models, ...)We do not assume asymptotic convergence of any statistics either


Hypothesis testing
http://find/


25/89

Hypothesis testing

Goal

testingVaRt andEStpredictions against observed profitloss realizationsxt

H0: Pt=FtH1: Ftis riskier thanPt

ESFt >

ESPt

we test only in the direction of risk underestimation

more specificH1s in the following, for computing test power

Modelfree testWe avoid any assumption on the nature of the predicted distributions Pt (nolocation-scale family, no parametric models, ...)We do not assume asymptotic convergence of any statistics either

http://find/


26/89




3 Results



Basel test for VaR exceptions (96)
http://find/


27/89

Basel test forVaRexceptions (96)

H0: bt= 1xt+VaRt BB(T, )

one says that coverage is not 1 =99%but only 1 (say 98%)

trafficlight system: yellow zone from 95% significance level and red zonefrom 99.99%


Basel VaR test: power vs coverage
http://find/


28/89

BaselVaRtest: power vs coverage

Figure:Fundamental review of the trading book: a revised market risk framework,

Basel Committee 2013


Basel VaR test: traffic light system
http://find/


29/89

BaselVaRtest: traffic light system


Criticism


30/89

Basel test addresses onlyunconditionalcoverage

independence of time arrival should be tested separately

Christoffersen (98): likelihood ratio test forconditionalcoverage

LRcc=LRuc+LRind

in most practical cases, however, independence testing is left to visualinspection, which helps interpreting exception clusters. Basel did notintroduce any independence formal test

in the following we assume that independence is tested separately. Wefocus on unconditionalEScoverage


Criticism
http://find/


31/89




LRcc=LRuc+LRind




Criticism


32/89




LRcc=LRuc+LRind




Visual inspection
http://find/


33/89

p



34/89




3 Results

4

ConclusionsPost Scriptum


Test 1
http://find/


35/89

testESafter having testedVaR

from

E

Xt+ESt

ESt

Xt+VaRt


36/89

testESafter having testedVaR

from

E

Xt+ESt

ESt

Xt+VaRt


37/89

under H0, the distributionPZ1 ofZ1(X)is simulated by drawing

independentXtPt,t

the realizationZ1(x)provides apvaluep=FZ1 (Z1(x))

acceptance/rejection based on a chosen significance level, say 5%

type2 probabilities and test power are computed based on specific

alternatives H1

Main difficulty

Storage of thetail of each distributionPt, to simulateZ1 underH0.Technologically elementary, but a challenge for auditing

the observations in this slide apply to all the tests proposed in thefollowing


Computing apvalue
http://find/


38/89

under H0, the distributionPZ1 ofZ1(X)is simulated by drawing

independentXtPt,t

the realizationZ1(x)provides apvaluep=FZ1 (Z1(x))

acceptance/rejection based on a chosen significance level, say 5%

type2 probabilities and test power are computed based on specific

alternatives H1

Main difficulty

Storage of thetail of each distributionPt, to simulateZ1 underH0.Technologically elementary, but a challenge for auditing

the observations in this slide apply to all the tests proposed in thefollowing


Test 2
http://find/


39/89

direct test forES

from the unconditional expectation

E

XtIt

= ES,t

introduce

Test statistic 2

Z2(X) =T

t=1

XtIt

T ESt+1

EH0[Z2] =0. ESunderestimated ifZ2


40/89

direct test forES

from the unconditional expectation

E

XtIt

= ES,t

introduce

Test statistic 2

Z2(X) =T

t=1

XtIt

T ESt+1

EH0[Z2] =0. ESunderestimated ifZ2


41/89

direct test forES

consider the r.v.sUt=Pt(Xt). Under H0,Uti.i.d U(0, 1)

Berkowitz (01) proposes to test for uniformity the tail of the empiricaldistribution of thext

We use this pseudouniform sample Uto estimateES

Test statistic 3

Z3(X) =1

T

Tt=1

EST,(P1t (U))EV

EST,(P1t (V)) +1

where Vi.i.d U(0, 1)EH0[Z3] =0. ESunderestimated ifZ3


42/89

direct test forES

consider the r.v.sUt=Pt(Xt). Under H0,Uti.i.d U(0, 1)

Berkowitz (01) proposes to test for uniformity the tail of the empiricaldistribution of thext

We use this pseudouniform sample Uto estimateES

Test statistic 3

Z3(X) =1

T

Tt=1

EST,(P1t (U))EV

EST,(P1t (V)) +1

where Vi.i.d U(0, 1)EH0[Z3] =0. ESunderestimated ifZ3


43/89

similar to Berkowitz (01), we can directly test the tail density, via the ESofthe uniform distribution

Test statistic 4

Z4(X) = EST,

(U)

EVEST,(V) 1

where Vi.i.d U(0, 1)

EH0[Z4] =0. Risk underestimated ifZ4


44/89

similar to Berkowitz (01), we can directly test the tail density, via the ESofthe uniform distribution

Test statistic 4

Z4(X) = EST,

(U)

EVEST,(V) 1

where Vi.i.d U(0, 1)

EH0[Z4] =0. Risk underestimated ifZ4


45/89

tests 2 and 3 can naturally be extended to all spectral measures

test 1 can be extended to simple spectral measures, with piecewiseconstant spectrum

http://find/


46/89


2 Testing settingBaselVaRbacktestThree tests forES. Plus one

3 Results



H0: Student-t; H1: scaled distributions
http://find/


47/89

H0: Ft=Pt, Student-t distribution

H1: Ft(x) =Pt(x/), scaled distribution ( >1)


H0: Student-t, = 100; H1: scaled distributions
http://find/


48/89


http://find/


49/89




50/89


H0: Student-t; H1: EScoverage 95%, 90%
http://find/


51/89


H1: Ft(x) =Pt(x/), again scaled distribution, but labeled in terms of EScoverage

ESP =ESF , with

=5%, 10%

analogous to the BaselVaRcoverage tables


H0: Student-t, = 100; H1: EScoverage 95%, 90%


52/89


http://find/


53/89




54/89




55/89

; Carlo Acerbi and Balazs Szekely Testing Expected Shortfall June 2014 35 / 59

H0: Student-t, = 100; H1: = 10, 5, 3;


56/89


H1: Student-t distribution with lower

notice that the standard deviation is larger = /( 2)


H0: Student-t, = 100; H1: = 10, 5, 3;
http://find/


57/89


H0: Student-t, = 100; H1: = 10, 5, 3;
http://find/


58/89


H0: Student-t, = 10; H1: = 5, 3;
http://find/


59/89


H0: Normalized Student-t, = 100; H1: = 10, 5, 3;


60/89

H0: Ft=Pt, Student-t distribution with =1

H1: Normalized Student-t distribution with lowerand =1


http://find/


61/89




62/89


H0: Normalized Student-t, = 10; H1: = 5, 3;
http://find/


63/89


H0: Normalized Student-t; H1: fixedVaR97.5%


64/89

H0: Ft=Pt, Student-t distribution with =1

H1: Normalized Student-t distribution with lowerand =1

the distribution are offset in such a way to have all the same VaR97.5%alternative hypotheses built to analyze test 1


H0: Normalized Student-t; H1: fixedVaR97.5%
http://find/


65/89


H0: Student-t, = 100; H1: fixedVaR97.5%


66/89


H0: Student-t, = 10; H1: fixedVaR97.5%


67/89


H0: Norm. Student-t, = 100; H1: fixedVaR97.5%


68/89


H0: Norm. Student-t, = 10; H1: fixedVaR97.5%


69/89


Summary of results
http://find/


70/89

all tests forES97.5% generally display more power than the Basel testforVaR99% in identical conditions

test 1 is subordinated to testing VaR, but has strong power for model

misspecifications in the tailtest 2 and test 3 excel in different cases. Test 2 is more powerful onscaled distributions. Test 3 is more powerful on distributions with differenttail index


Test 2: a very practical test
http://find/


71/89

Test 2 has critical levels that are almost invariant with respect to the tailproperties, in a range

= [5

, +)that spans all realistic cases of a

firmwide bank portfolio

it allows to define a traffic light system that does not require the collectionof the entiretail ofPt, but just the three numbersxt,ESt andIt

Critical levelsTest 1 Test 2 Test 3significance 5% 10% 5% 10% 5% 10%=3 -0.43 -0.27 -0.82 -0.59 -0.49 -0.32=5 -0.26 -0.17 -0.74 -0.55 -0.30 -0.22=10 -0.17 -0.12 -0.71 -0.53 -0.21 -0.16=100 -0.12 -0.08 -0.70 -0.53 -0.15 -0.12Gaussian -0.11 -0.08 -0.70 -0.53 -0.15 -0.11

http://find/


72/89



3 Results

4 Conclusions

Post Scriptum


Our results

ES i b k bl hi i i l l b i i l
http://find/


73/89

ESis backtestable; this is certainly not a new result, but surprisingly its worth reaffirming it

we propose three tests forES: the novelty of these tests is that they arenonparametric and contain no model assumptions. For this reason theyrepresent valid proposals for regulatory purposes

all of these tests display superior power to the standard Basel VaRbacktesting methodology

the main difficulty with backtestingESis that you need to store the tail ofall predictive distributionsPt. If this is not a conceptual problem andcertainly no more a technological one either, this is still a challenge for anauditable process. This is the only difference between backtestingESandVaR

one of the proposed tests displays a remarkable stability of the criticallevels, which provides an opportunity to set up practical tests for whichthe storage of the predictive distributions is not needed


Our results

ES i b kt t bl thi i t i l t lt b t i i l
http://find/


74/89







Our results

ES i b kt t bl thi i t i l t lt b t i i l
http://find/


75/89







Our results

ES is backtestable; this is certainly not a new result but surprisingly
http://find/


76/89







Our results

ES is backtestable; this is certainly not a new result but surprisingly
http://find/


77/89







Elicitability
http://find/


78/89

Elicitability ofVaRhas no relevance in the regulatory debate

Elicitability allows you to compare models which forecast the exact sameprocess, based on point forecasts only. But to score the performance of amodel against an absolute significance level, one still needs (or at least

we dont see how one would not) either model assumptions or recordingall predictive distributions

Its no coincidence that VaRin banks is backtested without exploiting itselicitability

http://find/


79/89

1

Motivation and goals


3 Results

4 Conclusions

Post Scriptum


By the way,ESis elicitable

ll t tl b t id th i f ti
http://find/


80/89

well, not exactly but consider the scoring function

S(v, e, x) =e2/2ev((x+v


81/89


S(v, e, x) =e2/2ev((x+v


82/89


S(v, e, x) =e2/2ev((x+v


83/89


S(v, e, x) =e2/2ev((x+v


84/89


S(v, e, x) =e2/2ev((x+v


85/89


86/89

General score function


87/89

most general scoring function, for allW

SW(v, e, x) = e2/2 +Wv2/2 ev+ e(v+x) +W(x

2 v2)/2 (x+v


88/89



89/89

Thanks!

http://find/

Documents

Expected Shortfall Backtest