Backtesting Value-at-Risk Models - univ-orleans.fr€¦ · Introduction Backtesting Principles Testing strategies Recommandations Backtesting Value-at-Risk Models Christophe Hurlin

IntroductionBacktesting Principles

Testing strategiesRecommandations

Backtesting Value-at-Risk Models

Christophe Hurlin

University of Orléans

Séminaire Validation des Modèles Financiers. 29 Avril 2013

Christophe Hurlin Backtesting



Introduction

The Value-at-Risk (VaR) and more generally the Distortion RiskMeasures (Expected Shortfall, etc.) are standard risk measuresused in the current regulations introduced in Finance (Basel 2), orInsurance (Solvency 2) to x the required capital (Pillar 1), or tomonitor the risk by means of internal risk models (Pillar 2).




Introduction

Denition

Let frtgTt=1 be a given P&L series. The daily (conditional) VaR fora nominal coverage rate α is dened as

Pr[ rt < VaR t jt1(α) Ft1] = α

where Ft1 denotes the set of information available at time t 1.




Introduction

Who does use VaR? What for?

Bank risk manager Measure rm-level market, credit, op. risk

Bank executives Set limits (management)

Banking regulators Determine capital requirements

Exchanges Compute margins

Regulators Forecast systemic risk (CoVaR)

Industry Ex: EDF, spot prices of electricity




"Disclosure of quantitative measures of market risk, suchas value-at-risk, is enligthening only when accompaniedby a thorough discussion of how the risk measures werecalculated and how they related to actual performance",Alan Greenspan (1996)




Introduction

DenitionBacktesting is a set of statistical procedures designed to check ifthe real losses are in line with VaR forecasts (Jorion, 2007).




Introduction

Whatever the type of use of VaR, the VaR forecasts aregenerated by an internal risk model.

This model is used to produced a sequence of pseudo out-ofsample VaR forecasts for a past period (typically one year)

The backtesting is based on the comparison of the observedP&L to these VaR forecasts.




Outlines

1 How to test the validity of a VaR model?2 What are the backtesting strategies?3 What are the good practices?




Backtesting Principles





Remark 1: Ex-post VaR is not observable, so it is impossible tocompute traditional statistics or criteria such as MSFE.

Remark 2: There is no proxy for the VaR contrary to the volatility(realized volatility, Andersen and Bollerslev 1998)

Patton, A.J. (2011) Volatility forecast comparison using imperfect volatility

proxies, Journal of Econometrics, 260, 246-256.





Backtesting procedures are based on VaR exceptions

Denition

We denote It (α) the hit variable associated to the ex-postobservation of an α% VaR exception at time t :

It (α) =

(1 if rt < VaR t jt1(α)0 else





Christo¤ersen (1998) : VaR forecasts are valid if and only if theviolation process It (α) satises the following two assumptions:

1 The unconditional coverage (UC) hypothesis.

2 The independence (IND) hypothesis.

Christo¤ersen P.F. (1998), Evaluating interval forecasts, International Economic

Review, 39, pp. 841-862.





Denition (unconditional coverage hypothesis)The unconditionnal probability of a violation must be equal to theα coverage rate

Pr [It (α) = 1] = E [It (α)] = α.

If Pr [It (α) = 1] > α, the risk is under-estimated

If Pr [It (α) = 1] < α, the risk is over-estimated





Denition (independence hypothesis)VaR violations observed at two di¤erent dates must beindependently distributed.

It (α) and Is (α) are independently distributed for t 6= s





Figure: Illustration: violationscluster

0 50 100 150 200 2506

4

2

0

2

4

6

8 VaR(95%) P&L





Figure: Illustration: violationscluster

0 50 100 150 200 2506

4

2

0

2

4

6

8 VaR(95%) P&L





Denition (conditional coverage hypothesis)The violation process satises a di¤erence martingale assumption.

E [ It (α) j Ft1] = α





Remark: These assumptions can be expressed as distributionalassumptions.

Under the UC assumption, each variable It (α) has a Bernouillidistribution with a probability α.

Itt (α) Bernouilli (α)

Under the IND assumption, these variables are independent, andthe number of violations has a Binomial distribution.

T

∑t=1It (α) B (T , α)




Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests

Testing strategiesWhat are the backtesting strategies?





Testing strategies

Let us consider a sequence of daily VaR out-of-sample forecastsVaR t jt1(α)

Tt=1

and the corresponding observed P&L.

How to test the validity of the internal risk model?

Hurlin C. and Pérignon C. (2012), Margin Backtesting,Review of Futures Market, 20, pp. 179-194





Testing strategies

Testing strategies:

1 Frequency-based tests

2 Magnitude-based tests

3 Multivariate tests

4 Independence tests

5 Duration-based tests





Testing strategies: frequency-based tests (1/5)

Figure: BIS "Tra¢ c Light" System

Note: VaR(1%, 1 day), 250 daily observations






Denition

Christo¤ersen (1998) proposes a Likelihood Ratio statistic for UCdened as:

LRUC = 2 lnh(1 α)TH αH

i+2 ln

h(1H/T )TH (H/T )H

id!

T!∞χ2 (1)

where H = ∑Tt=1 It (α) denotes the total number of exceedances.

For a nominal risk of 5%, the null of UC can not be rejected ifand only if H < 7 for T = 250 and α = 1%.






Example

Berkowitz and O-Brien (2002) consider the VaR forecasts of six UScommercial banks

Berkowitz, J., and O-Brien J. (2002), How Accurate are theValue-at-Risk Models at Commercial Banks, Journal ofFinance.






Figure: Bank Daily VaR Models






Figure: Violations of Banks99% VaR















Testing strategies

Testing strategies:










Testing strategies: magnitude-based tests (2/5)

All these tests do not take into account the magnitude of thelosses beyond the VaR

Example

Consider two banks that both have a one-day 1%-VaR of $100million. Assume each bank reports three VaR exceptions, but theaverage VaR exceedance is $1 million for bank A and $500 millionfor bank B.In this case, standard backtesting methodologies would indicatethat the performance of both models is equal and acceptable.






Figure: Daily VaR and P/L for SocGen 2008






Figure: Daily VaR and P/L for SocGen 2008






The Risk Map

Colletaz G., Hurlin C. and Perignon C. (2013), The RiskMap: a new tool for Risk Management, forthcoming inJournal of Banking and Finance






We propose a VaR backtesting methodology based on the numberand the severity of VaR exceptions: this approach exploits theconcept of "super exception".

DenitionWe dene a super exception using a VaR with a much smallercoverage probability α0, with α0 < α. In this case, a superexception is dened as a loss greater than VaRt (α0).






Figure: VaR Exception vs. VaR Super Exception






Solution

Given VaR exceptions It (α) and VaR super exception It (α0), wedene a Risk Map that jointly accounts for the number and themagnitude of the VaR exceptions

Let us consider a given UC test with a statistic Z (α) based on theviolations sequence fIt (α)gTt=1 .

H0 : E [It (α)] = α

H1 : E [It (α)] 6= α.






Num

ber o

f VaR

Exc

eptio

ns (N

)

Nonrejection area for teston VaR exceptions






Based on the same UC test, it is possible to test for themagnitude of VaR exceptions, via the VaR super exceptionsfIt (α0)gTt=1

H0 : EItα0= α0

H1 : EItα06= α0






Nonrejectionarea for teston VaR superexceptions

Number of VaR Super Exceptions (N’)






We can also test jointly for both magnitude and frequency of VaRexceptions:

H0 : E [It (α)] = α and EItα0= α0

Multivariate approach

Perignon C. and Smith, D. (2008), A New Approach toComparing VaR Estimation Methods, Journal ofDerivatives






0 1 2 3 4 5 6 7 80123456789

101112131415

Number of VaR Super Exceptions (N')

Num

ber o

f VaR

Exc

eptio

ns (N

)

Nominal risk 5%

Nominal risk 1%






Figure: Backtesting Bank VaR: La Caixa (2007-2008)






0 1 2 3 4 5 6 7 80123456789

101112131415

Number of VaR Super Exceptions (N')

Num

ber o

f VaR

Exc

eptio

ns (N

)

Nominal risk 5%

Nominal risk 1%





Testing strategies

Testing strategies:










Testing strategies: multivariate tests (3/5)

Intuition: Testing the validity of the VaR model for M coveragerates, with M > 1.

Perignon C. and Smith, D. (2008), A New Approach toComparing VaR Estimation Methods, Journal ofDerivatives

Hurlin C. and Tokpavi, S. (2006), BacktestingValue-at-Risk Accuracy: A Simple New Test, Journal ofRisk






Perignon and Smith (2008) consider the null:

H0,MUC : E [It (α)] = α and EItα0= α0.

Let us denote:

J0,t = 1 J1,t J2,t

J1,t =

1 if VaR t jt1(α0) < rt < VaR t jt1(α)0 otherwise

J2,t =

1 if rt < VaR t jt1(α0)0 otherwise

.






Denition (Perignon and Smith, 2008)

The multivariate unconditional coverage test is a LR test given by:

LRMUC = 2 lnh(1 α)H0

α α0

H1 α0H2i+2 ln

"1 H0

T

H0 H0T H1T

H1 H2T

H2#.

where Hi = ∑Tt=1 Ji ,t , for i = 0, 1, 2, denote the count variable

associated with each of the Bernoulli variables.






Hurlin and Tokpavi (2006):

A natural test of the CC is the univariate Ljung-Box test of

H0,CC : r1 = ... = rK = 0

where rk denotes the k th autocorrelation:

LB(K ) = T (T + 2)K

∑k=1

br2kT k

d!T!∞

χ2 (K )






Denition (Hurlin and Tokpavi, 2006)

Let Θ = fθ1, .., θmg be a discrete set of m di¤erent coverage ratesand Hitt = [Hitt (θ1) : Hitt (θ2) : ... : Hitt (θm)]

0

Hitt (θi ) =

(1 θi if rt < VaR t jt1(θi )θi else

Under the null of CC (martingale di¤erence):

H0,CC : E[HittHit

0tk

= 0m 8k = 1, ...,K





Testing strategies

Testing strategies:










Testing strategies: independence tests (4/5)

LR tests

Christo¤ersen (1998) assumes that the violation process It (α) canbe represented as a Markov chaine with two states:

Π =

1 π01 π011 π11 π11

πij = Pr [ It (α) = j j It1 (α) = i ]

DenitionThe null of CC can be dened as follows:

H0,CC : Π = Πα =

1 α α1 α α






LR tests

Christo¤ersen (1998) assumes that the violation process It (α) canbe represented as a Markov chaine with two states:

Π =

1 π01 π011 π11 π11

πij = Pr [ It (α) = j j It1 (α) = i ]

DenitionThe null of IND can be dened as follows:

H0,IND : Π = Πβ =

1 β β1 β β






The corresponding LR statistics are dened by:

LRIND = 2 lnh(1H/T )TH (H/T )H

i+2 ln [(1 bπ01)n00 bπn0101 (1 bπ11)n10 bπn1111 ] d!

T!∞χ2 (1)

LRCC = 2 lnh(1 α)TH (α)H

i+2 ln [(1 bπ01)n00 bπn0101 (1 bπ11)n10 bπn1111 ] d!

T!∞χ2 (2)

By denition:LRCC = LRUC + LRIND
















Figure: Violations of Banks99% VaR






Regression based tests

Engle and Manganelli (2004) suggest another approach basedon a linear regression model. This model links current marginexceedances to past exceedances and/or past information.

Let Hit (α) = It (α) α be the demeaned process associatedwith It (α):

Hitt (α) =1 α if rt < VaR t jt1(α)α otherwise

.







Consider the following linear regression model:

Hitt (α) = δ+K

∑k=1

βkHittk (α) +K

∑k=1

γkztk + εt

where the ztk variables belong to the information set Ωt1(lagged P&L, squared past P&L, past margins, etc.)







The null hypothesis test of CC corresponds to testing the jointnullity of all the regression coe¢ cients:

H0,CC : δ = βk = γk = 0, 8k = 1, ...,K .

since under the null :

E [Hitt (α)] = E [It (α) α] = 0() Pr [It (α) = 1] = α






Denition (Engle and Manganelli, 2004)

Denote Ψ = (δ β1 ...βK γ1 ...γK )0 the vector of the 2K + 1

parameters in this model and Z the matrix of explanatory variablesof model, the Wald statistic, denoted DQCC , then veries:

DQCC =bΨ0Z 0Z bΨα (1 α)

d!T!∞

χ2 (2K + 1)

where bΨ is the OLS estimate of Ψ.







Extension: A natural extension of the test of Engle and Manganelli(2004) consists in considering a (probit or logit) binary modellinking current violations to past ones

Dumitrescu E., Hurlin C. and Pham V. (2012),Backtesting Value-at-Risk: From Dynamic Quantile toDynamic Binary Tests, Finance






Denition (Dumitrescu et al., 2012)

We consider a dichotomic model:

Pr [ It (α) = 1 j Ft1] = F (πt ) .

where F (.) denotes a c.d.f. and the index πt satises thefollowing autoregressive representation:

πt = c +q1

∑j=1

βjπtj +q2

∑j=1

δj Itj (α) +q3

∑j=1

γjxtj ,

where l(.) is a function of a nite number of lagged values ofobservables, and xt is a vector of explicative variables.







H0 : β = 0, δ = 0,γ = 0 and c = F1 (α) .

since under the null of CC:

Pr(It = 1 j Ft1) = F (F1(α)) = α.

The Dynamic Binary (DB) LR test statistic is:

DBLRCC = 2ln L(0,F1(α); It (α),Zt ) ln L(θ, c ; It (α),Zt )

d!

T!∞χ2 (dim(Zt ))





Testing strategies

Testing strategies:










Testing strategies: duration-based tests (5/5)

The UC, IND, and CC hypotheses also have some implicationson the time between two consecutive VaR margin exceedances.

Let us denote by dv the duration between two consecutiveVaR margin violations:

dv = tv tv1

where tv denotes the date of the v th exceedance.






Under CC hypothesis, the duration process dv has a geometricdistribution:

Pr [dv = k ] = α (1 α)k1 k 2 N.

This distribution characterizes the memory-free property ofthe violation process It (α) with E (dv ) = 1/α






Denition

Christo¤ersen and Pelletier (2004) use under the null hypothesisthe exponential distribution:

g (dv ; α) = α exp (αdv ) .

Under the alternative hypothesis, they postulate a Weibulldistribution for the duration variable:

h (dv ; a, b) = abbdb1v exph (adv )b

i.

H0,IND : b = 1 H0,CC : b = 1, a = α






Drawback: we have to postulate a distribution for the durationunder the alternative (misspecication of the VaR model).

Solution: Candelon et al. (2001) propose a J-test based onorthonormal polynomials associated to the geometric distribution.

Candelon B., Colletaz G., Hurlin C. et Tokpavi S. (2011),"Backtesting Value-at-Risk: a GMM duration-basedtest", Journal of Financial Econometrics,






Candelon et al. (2001)

In the case of continuous distributions, the Pearson distributions(Normal, Student, etc.) are associated to some particularorthonormal polynomials whose expectation is equal to zero.

These polynomials can be used as special moments to test fora distributional assumption (see. Bontemps and Meddahi,Journal of Econometrics, 2005).

In the discrete case, orthonormal polynomials are dened fordistributions belonging to the Ords family (Poisson, Pascal,hypergeometric, etc.).







DenitionThe orthonormal polynomials associated to a geometricdistribution with a success probability β are dened by thefollowing recursive relationship, 8d 2 N:

Mj+1 (d ; β) =(1 β) (2j + 1) + β (j d + 1)

(j + 1)p1 β

Mj (d ; β)

jj + 1

Mj1 (d ; β) ,

for any order j 2 N , with M1 (d ; β) = 0 and M0 (d ; β) = 1 and:

E [Mj (d ; β)] = 0 8j 2 N, 8d 2 N.Christophe Hurlin Backtesting






Example

We can show that if d follows a geometric distribution ofparameter β, then:

M1 (d ; β) = (1 βd) /p1 β

withE [M1 (d ; β)] = 0







Our duration-based backtest procedure exploits these momentconditions.

More precisely, let us dene fd1; ...; dNg a sequence of Ndurations between VaR violations, computed from thesequence of the hit variables fIt (α)gTt=1 .Under the CC assumption, the durations di , i = 1, ..,N, arei .i .d . geometric(α). Hence, the null of CC can be expressedas follows:

H0,CC : E [Mj (di ; α)] = 0, j = f1, .., pg ,

where p denotes the number of moment conditions.







DenitionThe null hypothesis of CC can be expressed as

H0,CC : E [M (di ; α)] = 0,

where M (di ; α) denotes a (p, 1) vector whose components are theorthonormal polynomials Mj (di ; α) , for j = 1, .., p. Under someregularity conditions:

JCC (p) =

1pN

N

∑i=1M (di ; α)

!| 1pN

N

∑i=1M (di ; α)

!d!

N!∞χ2 (p)







DenitionUnder UC, the mean of durations between two violations is equalto 1/α, and the null hypothesis is

H0,UC : E [M1 (di ; α)] = 0.

with a test statistic equal to

JUC =

1pN

N

∑i=1M1 (di ; α)

!2d!

N!∞χ2(1)




Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk

Backtesting

Recommandations





Backtesting

Recommandation 1: Test, test and test

Recommandation 2: Check the P&L data

Recommandation 3: The power of your tests may be low..

Recommandation 4: Take into account the estimation risk





Backtesting

Recommandation 1: Test, test and test

Each type of test (frequency, severity, independence,conditional coverage, multivariate test etc..) captures onetype of potential misspecication of the VaR model.

It is important to use a variety of tests





Backtesting





Backtesting





Backtesting

Recommandation 2: Check the P&L data

Frésard, L., C. Perignon, and W., Anders (2011), ThePernicious E¤ects of Contaminated Data in Risk Management,Journal of Banking and Finance.

1 A large fraction of US and international banks validate theirmarket risk model using P&L data that include fees andcommissions and intraday trading revenues.

2 Distinction between dirty P/L and hypothetical P/L (JP.Morgan, Romain Berry 2011).





Backtesting

Recommandation 3: The power of your tests may be low..

DenitionThe power of a backtesting test corresponds to its capacity todetect misspecied VaR model.

Pr [Rejection H0 j H1]

Example

Berkowitz, J., Christo¤ersen, P. F., and Pelletier, D., 2013,Evaluating Value-at-Risk Models with Desk-Level Data.Management Science.





Backtesting





Backtesting





Backtesting





Backtesting

Hurlin C. et Tokpavi S. (2008), Une Evaluation desProcédures de Backtesting : Tout va pour le Mieux dans leMeilleur des Mondes", Finance

Idea: we use 6 di¤erent methods (GARCH, RiskMetrics, HS,CaviaR, Hybride, Delta Normale) to forecast a VaR(5%) on thesame asset (GM, Nasdaq), and we apply the backtests (LR, DQ,Duration based tests) on a set of 500 samples (rolling window) ofT = 250 forecasts.





Backtesting

Example

LRCC tests: for 47% of the samples, we dont reject (at 5%) thenull for any of the six VaR forecats. In 71% of the samples, wereject at the most one VaR.

Example

DQCC tests: for 20% of the samples, we dont reject (at 5%) thenull for any of the six VaR forecats. In 51% of the samples, wereject at the most one VaR.





Backtesting

The power of a consistent test tends to 1 when the samplesize tends to ininity.

Recommandation: increase at the maximum the sample sizeof your backtest.. (T = 500, 750 or more.)





Backtesting

Recommandation 4: Take into account the estimation risk

The risk dynamic is usually represented by a parametric orsemi-parametric model, which has to be estimated in apreliminary step. However, the estimated counterparts of riskmeasures are subject to estimation uncertainty.

Replacing, in the theoretical formulas, the true parametervalue by an estimator induces a bias in the coverageprobabilities.





Backtesting





Backtesting

Escanciano and Olmo (2010, 2011) studied the e¤ects ofestimation risk on backtesting procedures. They showed howto correct the critical values in standard tests used to assessVaR models.

Escanciano, J.C. and J. Olmo (2010) Backtesting ParametricValue-at-Risk with Estimation Risk, Journal of Business andEconomics Statistics.

Escanciano, J.C. and J. Olmo (2011) Robust Backtesting Testsfor Value-at-Risk Models. Journal of Financial Econometrics.





Backtesting

Estimation Adjusted VaR

Gouriéroux and Zakoian (2013) a method to directly adjust theVaR to estimation risk ensuring the right conditional coverageprobability at order 1/T :

Prrt < EVaR t jt1 (α)

= α+ oP (1/T )

Gouriéroux C. and Zakoian J.M. (2013), Estimation AdjustedVaR, forthcoming in Econometric Theory.





Thank you for your attention


Documents

Backtesting Value-at-Risk Models - univ-orleans.fr€¦ · Introduction Backtesting Principles Testing strategies Recommandations Backtesting Value-at-Risk Models Christophe Hurlin