Risk Model Validationsubscriptions.risk.net/wp-content/uploads/2016/03/JRMV.pdf · portfolio of credit lines Marco Geidosch Incisive Media, Haymarket House, 28-29 Haymarket, London

Volume 9 Number 4December 2015

Validation

The Journal of

Risk Model

■ AERB: developing AIRB PIT–TTC PD models using external ratings Gaurav Chawla, Lawrence R. Forest Jr and Scott D. Aguais

■ A mean-reverting scenario design model to create lifetime forecasts and volatility assessments for retail loans Joseph L. Breeden and Sisi Liang

■ Downside risk measure performance in the presence of breaks in volatility Johannes Rohde

■ Liquidity stress testing: a model for a portfolio of credit lines Marco Geidosch

Incisive Media, Haymarket House, 28-29 Haymarket, London SW1Y 4RX

PEFC Certified

This book has been produced entirely from sustainable papers that are accredited as PEFC compliant.

www.pefc.org

The Jo

urn

al of R

isk Mo

del V

alidatio

n Volum

e 9 Num

ber 4 Decem

ber 2015

JoRMV_9_4_Dec15.indd 1 20/11/2015 16:14

Tria

l Cop

y For all subscription queries, please call:

UK/Europe: +44 (0) 207 316 9300

USA: +1 646 736 1850 ROW: +852 3411 4828

in numbers

140,000

Users

Page views

19,400+ on Regulation

6,900+ on Commodities

19,600+ on Risk Management

6,500+ on Asset Management

58,000+ articles stretching back 20 years

200+

New articles & technical papers

370,000

21,000+ on Derivatives £

Visit the world’s leading source of exclusive in-depth news & analysis on risk management, derivatives and complex fi nance now.

(each month)

(each month)

See what you’re missing

(each month)

RNET16-AD156x234-numbers.indd 1 21/03/2016 09:44

The Journal of Risk Model ValidationEDITORIAL BOARD

Editor-in-ChiefSteve Satchell University of Cambridge

Associate Editors

Moawia Alghalith University of the West Indies David Li Mitsubishi BankMohan Bhatia Wipro Ltd Christian Meyer DZ BANK AGStefan Blochwitz Deutsche Bundesbank Peter Miu McMaster UniversityJ. L. Breeden Prescient Models LLC Bogie Ozdemir Sun LifeWei Chen SAS Institute Inc. Financial GroupGeorge Christodoulakis Manchester Business Peter Quell DZ BANK AG

School Daniel Rösch University of HannoverMarcelo Cruz Harald Scheule University ofKlaus Duellmann European Central Bank Technology, SydneyDouglas Dwyer Moody’s Analytics Roger Stein Moody’s Investors ServiceChristopher C. Finger MSCI, RiskMetrics Lyn Thomas University of Southampton

SUBSCRIPTIONS

The Journal of Risk Model Validation (Print ISSN 1753-9579 j Online ISSN 1753-9587) is publishedquarterly by Incisive Risk Information Limited, Haymarket House, 28–29 Haymarket, LondonSW1Y 4RX, UK. Subscriptions are available on an annual basis, and the rates are set out in thetable below.

UK Europe USRisk.net Journals £1945 €2795 $3095Print only £735 €1035 $1215Risk.net Premium £2750 €3995 $4400

Academic discounts are available. Please enquire by using one of the contact methods below.

All prices include postage.All subscription orders, single/back issues orders, and changes of addressshould be sent to:

UK & Europe Office: Incisive Media (c/o CDS Global), Tower House, Sovereign Park,Market Harborough, Leicestershire, LE16 9EF, UK. Tel: 0870 787 6822 (UK),+44 (0)1858 438 421 (ROW); fax: +44 (0)1858 434958

US & Canada Office: Incisive Media, 55 Broad Street, 22nd Floor, New York, NY 10004, USA.Tel: +1 646 736 1888; fax: +1 646 390 6612

Asia & Pacific Office: Incisive Media, 20th Floor, Admiralty Centre, Tower 2,18 Harcourt Road, Admiralty, Hong Kong. Tel: +852 3411 4888; fax: +852 3411 4811

Website: www.risk.net/journal E-mail: [email protected]

To subscribe to a Risk Journal visit Risk.net/subscribe or email [email protected]

http://subscriptions.risk.net/subscribe/

mailto:corpsubs%40risk.net?subject=

The Journal of Risk Model ValidationGENERAL SUBMISSION GUIDELINES

Manuscripts and research papers submitted for consideration must be original workthat is not simultaneously under review for publication in another journal or otherpublication outlets. All articles submitted for consideration should follow strict aca-demic standards in both theoretical content and empirical results. Articles should beof interest to a broad audience of sophisticated practitioners and academics.

Submitted papers should follow Webster’s New Collegiate Dictionary for spelling,and The Chicago Manual of Style for punctuation and other points of style, apart froma few minor exceptions that can be found at www.risk.net/journal. Papers should besubmitted electronically via email to: [email protected]. Please clearlyindicate which journal you are submitting to.

You must submit two versions of your paper; a single LATEX version and a PDFfile. LATEX files need to have an explicitly coded bibliography included. All files mustbe clearly named and saved by author name and date of submission. All figures andtables must be included in the main PDF document and also submitted as separateeditable files and be clearly numbered.

All papers should include a title page as a separate document, and the full names,affiliations and email addresses of all authors should be included. A concise andfactual abstract of between 150 and 200 words is required and it should be includedin the main document. Five or six keywords should be included after the abstract.Submitted papers must also include an Acknowledgements section and a Declarationof Interest section. Authors should declare any funding for the article or conflicts ofinterest. Citations in the text must be written as (John 1999; Paul 2003; Peter and Paul2000) or (John et al 1993; Peter 2000).

The number of figures and tables included in a paper should be kept to a minimum.Figures and tables must be included in the main PDF document and also submittedas separate individual editable files. Figures will appear in color online, but willbe printed in black and white. Footnotes should be used sparingly. If footnotes arerequired then these should be included at the end of the page and should be no morethan two sentences. Appendixes will be published online as supplementary material.

Before submitting a paper, authors should consult the full author guidelines at:

http://www.risk.net/static/risk-journals-submission-guidelines

Queries may also be sent to:The Journal of Risk Model Validation, Incisive Media,Haymarket House, 28–29 Haymarket, London SW1Y 4RX, UKTel: +44 (0)20 7004 7531; Fax: +44 (0)20 7484 9758E-mail: [email protected]




The Journal of

Risk ModelValidation

The journalAs monetary institutions rely heavily on economic and financial models for a widearray of applications, model validation has become progressively inventive within thefield of risk. The Journal of Risk Model Validation focuses on the implementation andvalidation of risk models, and it aims to provide a greater understanding of key issuesincluding the empirical evaluation of existing models, pitfalls in model validation andthe development of new methods. We also publish papers on backtesting. Our mainfield of application is in credit risk modeling but we are happy to consider any issuesof risk model validation for any financial asset class.

The Journal of Risk Model Validation considers submissions in the form of researchpapers on, but not limited to, the following topics.

� Empirical model evaluation studies.

� Backtesting studies.

� Stress-testing studies.

� New methods of model validation/backtesting/stress testing.

� Best practices in model development, deployment, production and mainte-nance.

� Pitfalls in model validation techniques (all types of risk, forecasting, pricingand rating).




The Journal of Risk Model Validation Volume 9/Number 4

CONTENTS

Letter from the Editor-in-Chief vii

RESEARCH PAPERSAERB: developing AIRB PIT–TTC PD models using external ratings 1Gaurav Chawla, Lawrence R. Forest Jr and Scott D. Aguais

A mean-reverting scenario design model to create lifetime forecastsand volatility assessments for retail loans 19Joseph L. Breeden and Sisi Liang

Downside risk measure performance in the presence of breaks in volatility 31Johannes Rohde

Liquidity stress testing: a model for a portfolio of credit lines 69Marco Geidosch

Editor-in-Chief: Steve Satchell Subscription Sales Manager: Aaraa JavedPublisher: Nick Carver Global Head of Sales: Michael LloydJournals Manager: Dawn Hunter Information and Delegate Sales Director: Michelle GodwinEditorial Assistant: Carolyn Moclair Composition and copyediting: T&T Productions LtdMarketing Executive: Giulia Modeo Printed in UK by Printondemand-Worldwide

©Copyright Incisive Risk Information (IP) Limited, 2015. All rights reserved. No parts of this publicationmay be reproduced, stored in or introduced into any retrieval system, or transmitted, in any form or by anymeans, electronic, mechanical, photocopying, recording or otherwise without the prior written permission of thecopyright owners.

Marketing Manager: Rainy Gill




LETTER FROM THE EDITOR-IN-CHIEF

Steve SatchellTrinity College, University of Cambridge

This issue brings us four interesting papers, which I shall discuss below. As an aside,though, I was delighted to meet a loan book risk management team recently whoare avid readers of The Journal of Risk Model Validation. While respecting theiranonymity, I take on board their comments that a pressing issue for them is learninghow to react to the ever-changing regulatory framework. We would be delighted topublish such material but it is hard for the journal to provide timely information, notleast because of the built-in time lags between creation, production and disseminationof relevant material.

Our first paper, “AERB: developing AIRB PIT–TTC PD models using externalratings” by Gaurav Chawla, Lawrence R. Forest Jr and Scott D. Aguais, addressesthe criticism of the use of credit rating agency (CRA) ratings in the lending practicesof credit institutions since the financial crisis. It is believed that this criticism isdue to the fact that these ratings and observed risks have diverged. Some regulatorsdo not allow external ratings to be used as direct inputs into a credit institution’sinternal probability of default/ratings model, and hence the capital planning processas regulators prefer the use of internal assessments generally. However, regulatorsallow the use of CRAs’ long-run average default rates to benchmark the output ofan institution’s internal probability of default model. The authors propose a class of“agency replication” style models that make use of obligor information and CRAlong-term default rate information. They show how one can use this class of modelsto model portfolios such as large corporates, banks, insurance companies, etc. Theyalso present a range of applications of their methodology.

The issue’s second paper, “A mean-reverting scenario design model to create life-time forecasts and volatility assessments for retail loans” by Joseph L. Breeden andSisi Liang, discusses a forecasting model for loan prices over the life of a given loan.The approach taken is to use macroeconomic scenarios for the near term and thenassume the long-run average for future years. The authors create a loan-level forecast-ing model using an age–vintage–time structure for retail loans (in this case, a smallauto loan portfolio). The loan-level age–vintage–time model is similar in structure toan age–period–cohort model, but it is estimated at the loan level for greater robust-ness for small portfolios. They employ a concept called the environmental functionof time, which is correlated to macroeconomic factors and which allows for greatermodel stability. This approach is in line with the explicit goals of the new Finan-cial Accounting Standards Board loan loss accounting guidelines. In addition, the

www.risk.net/journal




authors’ model provides a simple mechanism to transition between point-in-time andthrough-the-cycle economic capital estimates using an internally consistent model.

The third paper in the issue, “Downside risk measure performance in the pres-ence of breaks in volatility” by Johannes Rohde, looks at downside risk measuresusing a framework based on the loss function for the comparative measurement of thesensitivity of quantile downside risk measures to breaks in the volatility or the distri-bution by extending the model comparison approach introduced by Lopez in 1998.The author finds that expected shortfall appears to be the superior measure becauseof its ability to identify breaks in the volatility. An empirical study – in which datafrom six stock market indexes is used – additionally demonstrates the applicabilityof this procedure and reconfirms the findings from the simulation study.

Our final paper, “Liquidity stress testing: a model for a portfolio of credit lines” byMarco Geidosch, discusses how cash outflows due to credit lines can be modeled in aliquidity stress test. The model is based on bootstrapping from a portfolio time seriesof daily credit line drawdowns. It is claimed that it does not rely on any distributionalassumptions or any complex parameter estimation, ie, model risk is low; and secondly,it is argued that it is intuitive and straightforward to implement. The author providessimulation results to support his claims. Returning to our earlier aside, we wouldwelcome regular commentary from practitioners related to issues of model risk andrisk measurement more generally.

Journal of Risk Model Validation 9(4)




CALL FOR PAPERS

Special Issue on Model Risk:Foundations, Quantification and Mitigation

Guest Edited by Christian Meyer and Peter Quell, DZ BANK AG

In December 2016, The Journal of Risk Model Validation will publish a special issueon the model risk of internal risk models with a focus on foundations, quantificationand mitigation. The journal is inviting submissions, in the form of research papers,on the following topics.

� Foundations.

– Definitions of model risk.

– Model risk through parameter uncertainty and model specification.

– Regulatory views on model risk.

– Internal models and standard approaches.

� Quantification.

– Pros and cons of model risk quantification.

– The role of benchmark models.

– To what extent could model risk be quantified in

� market risk,

� credit risk,

� operational risk and

� liquidity risk?

– Models for risk aggregation.

– Combining results from multiple risk models.

� Mitigation.

– The structure of comprehensive model risk management.

– Best-practice approaches.

– The role of stress tests in model risk assessment and risk model validation.

– Simple models versus comprehensive models.

– Conservatism as model risk mitigation.

www.risk.net/journal




Submission requirements

Manuscripts should be prepared for publication in accordance with our submis-sions guidelines, which can be found at www.risk.net/static/risk-journals-submission-guidelines.

All submissions will be subject to a peer review process by at least two independentpeer reviewers. Final decisions on paper acceptance will be taken by the Editor-in-Chief, Stephen Satchell.

Length

The journal has a strict length policy. Research papers should not exceed 8000 words(including references).

How to submit

Submissions should be sent via the online submission site: https://editorialexpress.com/risk.

Submission deadlines

Paper should be submitted by May 14, 2016. Publication date: December 23, 2016.

Journal of Risk Model Validation 9(4)




Journal of Risk Model Validation 9(4), 1–18

Research Paper

AERB: developing AIRB PIT–TTC PD modelsusing external ratings

Gaurav Chawla,1 Lawrence R. Forest Jr2 andScott D. Aguais3

123 Chelmsford Road, Shenfield CM15 8RB, UK; email: [email protected] & Associates (AAA) Ltd, 2080 Mackinnon Avenue, Cardiff-by-the-Sea, CA, USA;email: [email protected] & Associates (AAA) Ltd, 20–22 Wenlock Road, London N1 7GU, UK;email: [email protected]

(Received July 2, 2015; revised September 1, 2015; accepted September 15, 2015)

ABSTRACT

The use of credit rating agency (CRA) ratings in a credit institution’s lending practiceshas been directly criticized since the financial crisis, as these ratings and observedrisks have diverged. Some regulators do not allow the use of external ratings as directinputs into a credit institution’s internal probability of default (PD)/ratings model;hence, the capital planning process, as regulators generally prefer the use of inter-nal assessments. However, regulators allow for the use of a CRA’s long-run averagedefault rates to benchmark an institution’s internal PD model output.A recent study weconducted shows, however, that the indiscriminate use of such benchmarks can intro-duce significant biases in credit lending. Even with these criticisms and new regulatoryconstraints, one can still make use of the rich CRA data available to create regulatory-compliant PD models. In this paper, we propose a class of agency replication-stylemodels that make use of obligor information and CRA long-term DR information.Such models are extremely useful in cases where a credit institution has limited defaultsamples, so a purely internal default-based model could be potentially erroneous, andwhere, in contrast, agencies have plenty of data supporting the development of robust

Corresponding author: G. Chawla Print ISSN 1753-9579 j Online ISSN 1753-9587Copyright © 2015 Incisive Risk Information (IP) Limited

1




2 G. Chawla et al

models. In this paper, we show how one can use this class of models for modelingportfolios such as large corporates, banks and insurance companies. We discuss ourexperience developing approved, advanced internal ratings-based (AIRB) models,which we augment with external default data and, hence, colloquially call advancedexternal ratings-based (AERB) models. We show various simplifications of the for-mulation and show how they can be used in point-in-time/through-the-cycle-basedcredit rating systems. “AERB” is an acronym we started utilizing informally whiledeveloping PD models for regulatory approval under Basel II waivers. In contrastto AIRB models, AERB models are used when regulators, in our recent experience,require broader, external, long-run default calibrations to complement internal defaultdata when internal default data is limited.

Keywords: agency PIT model; agency replication model; point in time (PIT); through the cycle(TTC); low default portfolio (LDP).

1 OVERVIEW

Historically, credit rating agencies (CRAs) have provided credit assessments of insti-tutions in a manner that was conducive to the development of credit markets. Overtime, the big three CRAs – Standard & Poor’s (S&P), Moody’s and Fitch – weredubbed “nationally recognized statistical rating organizations” (NRSROs) (see LegalInformation Institute 2015). Although some other firms were later recognized asNRSROs, this certainly provided a barrier to entry into the big three and laid thefoundation for the systemic importance of ratings in credit decision making. Over theyears, CRA ratings were embedded in a credit institution’s lending practices, whichlater received regulatory approval under Basel I and Basel II.

However, the onset of the financial crisis changed this trend. The financial crisisinquiry report (Financial Crisis Inquiry Commission 2011) investigated the ratingprocess of CDOs and criticized the role of CRAs in the economic crisis. Over the pastfew years, CRAs have had billion-dollar settlements and faced additional scrutinyfrom government agencies, such as the Office of Credit Ratings at the Securities andExchange Commission (SEC) in the United States and the European Securities andMarkets Authority in the European Union.

Recently, regulatory authorities were tasked with decoupling credit institutions’lending practices from CRA ratings. Almost all regulatory authorities now call forthe internal assessment of obligors by the credit institution and ask for the collectionof credit-relevant information, eg, financial statement information, rather than rely-ing on CRA ratings. For example, in the United Kingdom, the Prudential RegulationAuthority (PRA) supervisory statement SS 11/13, Section 12.31 (Prudential Regula-tion Authority 2013), says that a model that has agency rating as its key driver without

Journal of Risk Model Validation www.risk.net/journal




AERB 3

any additional information does not meet the criteria of a valid internal ratings-based(IRB) model.

Thinking in terms of probability of default (PD) models, this clearly means thatusing CRA ratings as an explanatory factor in a PD model is not the best way for-ward in terms of regulatory approval. Note that a CRA rating is an amalgamation offinancial statement data and a CRA’s judgmental considerations (Standard & Poor’s2002; Moody’s 2008). The introduction of such financial statement data and a creditinstitution’s own judgmental factors in an internal model would supersede the needfor a CRA rating, ie, a model based on internally collected data would outperform onebased on a single CRA rating. Over the past decade, the use of automated financialdata feeds from data vendors, such as S&P Capital IQ and Bureau van Dijk (BvD), hasmade data collection easier, further reducing the cost of internal assessment basedon such data. In a typical advanced internal ratings-based (AIRB) approved bank,one would typically find automated data feeds for financial data, with credit officerscompleting the judgmental factors. The development of such an internal PD modelwith controlled cost and regulatory preference would mean that there is no need forCRA rating to be an explanatory variable in the internal PD model.

Let us have a look at another use of benchmarking in PD model development.AIRBPD models, typically calibrated to internal data with substantially different assump-tions in terms of their definition of default and modeling techniques, can result invariations in modeled PDs for the same entity, rated by different credit institutions.Such a variation has led regulatory authorities to benchmark every credit institution’sinternal PD model output against every other’s, as well as with CRA long-term defaultrates (DRs). For example, in the United Kingdom, the PRA performs a hypotheticalportfolio exercise (HPE) in which it compares each credit institution’s PDs with medi-ans from all reporting institutions as well as with S&P’s DRs.1 Such a benchmarkingexercise has its flaws, and we would like to point out two big shortcomings.

� The benchmark CRA DRs could be biased in terms of industry, time or rating.

� The internal model’s PDs could be point in time (PIT), through the cycle (TTC)or hybrid (they are mostly TTC, in our experience), and a comparison at oneinstance in time would be invalid.2

1 In this document, we refer to experienced default rates as DRs. These are annual default rates forS&P or Moody’s based on cohorts as of January 1. The ideal objective of any PD model should beto predict temporal and cross-sectional variation in DRs as closely as possible.2 We refer to a model as a PIT PD model if its output is purely point in time (assumed or quantifiedas pure PIT). We refer to a model as a TTC PD model if its output is purely through the cycle(assumed or quantified as pure TTC). Alternatively, we call a model a hybrid model if its output isneither fully PIT nor fully TTC. In our paper, we demonstrate that agency ratings in themselves arehybrid indicators of default.

www.risk.net/journal Journal of Risk Model Validation




4 G. Chawla et al

We have described these shortcomings in our related work on biased benchmarks(Forest et al 2015). But, overall, the case for the use of CRA ratings for benchmarkingis convincing due to the availability of a rich history of long-term default rates coveringmultiple recessions and a relation to any firm’s credit decision making.

With the general invalidity of CRA ratings as direct explanatory variables and thevalid use of agency DRs as benchmarks in mind, one can then ask the question ofhow and when CRA ratings can be used to develop valid AIRB PD models.

In this paper, we introduce a class of models called agency replication models.Here, an entity’s internal financial assessment and judgmental factors are regressedto corresponding CRA ratings’ PIT or TTC PDs (but not DRs). We show how wecan use this class of models for modeling portfolios such as large corporates, banksand insurance companies. For certain portfolios in which there is insufficient data,ie, for low default portfolios (LDP), this is clearly the preferred approach, becausethe use of strictly internal default experiences can give erroneous results. We discussour experience of AIRB approval of these models and, hence, colloquially call themadvanced external ratings-based (AERB) models. We also build these models withinour PIT–TTC framework, allowing for the dual use of PIT and TTC PDs.3 This dualusage allows direct and consistent support for both regulatory capital objectives requir-ing TTC model calibrations and stress testing, and International Financial ReportingStandard 9 (IFRS9), which in our opinion requires PIT model calibrations.

In Section 2, we explain our modeling choices and model specification. In Section 3,we discuss model diagnostics, benchmarking and monitoring, and provide an annualreview. We look at model usage in Section 4 and provide a summary of our findingsin Section 5.

2 MODELING CHOICES AND MODEL SPECIFICATION

Generally speaking, we can place all PD modeling approaches into two categories.

Direct calibration to default. Here, financial and judgmental input factors are used asexplanatory variables to explain direct default (binary) events. The most commonclass of such models is the Black–Scholes–Merton-style model (Black and Scholes1973; Merton 1974), where proxies of leverage and volatility are used in some formto predict defaults, eg, Moody’s KMV model. Alternatively, reduced-form modelsmake use of input data to predict defaults, eg, Kamakura models. Another example

3 We define PIT PDs as estimates that draw on up-to-date, comprehensive information on the relatedobligors, and that account fully for the future effects of accumulating, systematic and idiosyncraticrisk. PIT PDs are supposed to closely track the temporal fluctuations in the DRs of large portfolios.We define the PIT PD as the unconditional expectation of an entity’s PD. We define a TTC PD as theconditional expectation of an entity’s PD, assuming that credit conditions are close to the long-termaverage.





AERB 5

is using the smoothed default rate associated with an agency grade as an explanatoryvariable, together with the credit cycle index (CCI), to explain an agency’s binarydefault event. We call this the agency PIT model and have described it in detail inour previous study (Forest et al 2015).

Indirect calibration to default via grade replication. Here, financial and judgmentalinput factors are used as explanatory variables to explain default rates or defaultdistances associated with CRA ratings. One can think of this as two models linkedby an agency rating. In our experience, this is a very robust alternative to the directcalibration approach in wholesale credit portfolios, where internal default data isscarce but CRAs offer a wealth of long-term default data. We call these agencyreplication-style models.

The model developer first needs to develop the general development data set with allrelevant obligor information, including CRA-related information. Such data shouldbe at obligor-year level; Table 1 on the next page shows a typical data set.

Over the past decade, we have built agency replication-style models for severalportfolios, such as large corporates, banks, insurance companies, broker-dealers,sovereign and sub-sovereign entities. We have also got AIRB approval from the reg-ulator using our AERB approach. In this discussion, however, our comments andanalysis have been generalized, so nothing presented is specific to or confidential forthe approved models at the banks for which we have recently worked.

In almost all cases, we have had eight to ten factors in PD models. Table 2 on page 7summarizes the availability of data for the agency PIT model, agency replicationmodel and an alternative direct-to-default model. The richness of data is apparentfrom such a comparison, and, in our experience, agency replication-style models insuch cases are much more robust than the alternative direct-to-default model.

We have developed the agency replication class of models within the PIT and TTCdual PD/ratings approach, developed and presented in Aguais et al (2004, 2007),Forest et al (2013) and Chawla et al (2013). The PIT–TTC framework supports amore detailed analysis of cross-time variations in agency grades by controlling forsystematic credit conditions using the PIT–TTC framework’s CCIs. This is extremelyimportant due to the hybrid nature of agency ratings, as demonstrated by Forest et al(2015).

In developing an agency replication-style model, a model developer faces threechoices:

(i) the explanatory variable, ie, the obligor’s input variables (Xs);

(ii) the dependent variable, ie, the default rate or default distance (Y );

(iii) the link function.





6 G. Chawla et al

TAB

LE

1Ill

ustr

ativ

eD

ata

setf

orde

velo

ping

agen

cyre

plic

atio

n-st

yle

mod

el.

CR

Ara

tin

gan

das

soci

ated

DR

s,P

Ds

and

DD

s‚

…„ƒ

Ob

ligo

r’s

exp

lan

ato

ryC

RA

CR

Afa

cto

rsat

tim

et

Ag

ency

CR

Ara

tin

gra

tin

g‚

…„ƒ

rati

ng

rati

ng

TT

CP

D,

PIT

PD

,P

rofi

tab

ility

atti

me

smo

oth

edse

gm

ent

seg

men

tL

iqu

idit

y(%

)C

CI

tP

Ds

(%)

DD

spec

ific

(%)

DD

spec

ific

(%)

DD

210

C0.

5B

BB

0.21

2.85

90.

192.

899

0.17

2.92

92.

55

C0.

3B

B�

1.46

2.18

11.

132.

281

1.00

2.32

63

15C

0.4

A�

0.10

3.09

80.

093.

110

0.08

3.15

6

We

mak

eus

eof

the

term

dist

ance

tode

faul

t(d

efau

ltdi

stan

ceor

DD

)in

the

cont

ext

ofM

erto

n-st

yle

PD

mod

els.

Inth

eM

erto

nm

odel

,di

stan

ceto

defa

ult

ism

easu

red

asth

est

anda

rdde

viat

ion

ofas

sets

that

afir

mha

sto

lose

befo

reits

asse

tshi

tthe

defa

ultp

oint

.





AERB 7

TAB

LE

2C

ompa

rison

ofda

taav

aila

bilit

yfo

rva

rious

type

sof

mod

els.

Dat

afo

ral

tern

ativ

ed

irec

t-D

ata

use

dfo

rag

ency

Dat

au

sed

for

agen

cyP

ort

folio

to-d

efau

ltm

od

elP

ITm

od

elre

plic

atio

nm

od

el

Larg

eco

rpor

ates

�10

0in

tern

alde

faul

tsin

�10

year

sw

ithlo

wer

over

alld

efau

ltra

tew

hen

com

pare

dw

ithlo

ng-t

erm

agen

cyD

Rs

Ban

ks�

30in

tern

alde

faul

tsin

�10

year

s,bi

ased

agai

nstl

arge

bank

san

din

favo

rof

smal

lba

nks

whe

nco

mpa

red

with

agen

cyde

faul

tdat

a

Hun

dred

sof

defa

ults

over

30C

year

s,gl

obal

cove

rage

,hu

ndre

dsof

defa

ults

for

segm

ent-

spec

ific

(cor

pora

teve

rsus

finan

cial

inst

itutio

nsve

rsus

sove

reig

n)ca

libra

tions

Tho

usan

dsof

entit

yye

arda

tapo

ints

with

finan

cial

/jud

gmen

tal

fact

ors

and

corr

espo

ndin

gag

ency

ratin

gsle

adin

gto

robu

stne

ssof

agen

cyre

plic

atio

nm

odel

estim

ates

Insu

ranc

e<

5in

tern

alde

faul

tsin

�10

year

s

Bro

ker-

deal

ers

<10

inte

rnal

defa

ults

in�

10ye

ars

Sov

erei

gns

<5

inte

rnal

defa

ults

in�

10ye

ars

with

low

erov

eral

ldef

ault

rate

whe

nco

mpa

red

with

long

-ter

mag

ency

DR

s

Loca

laut

horit

ies,

stat

ego

vern

men

ts,c

ount

ies,

mun

icip

aliti

es,e

tc

<20

inte

rnal

defa

ults

in�

10ye

ars

and

mos

tlydi

scon

nect

edw

ithso

vere

ign

ratin

gs/P

Ds





8 G. Chawla et al

With regard to the choice of explanatory variables, one can make use of financialdata from external vendors and/or an internal ratings database and judgmental factorsfrom an internal ratings database. Such explanatory variables can be redefined basedon an expert’s preference or ongoing data availability. Naturally, such factors vary byasset class, and one can benchmark the choice with factor selection from other studiesin this area. We broadly agree with the choice of explanatory variables in other studiessuch as Altman and Rijken (2004), Kamstra et al (2001), Minardi et al (2007) andMizen and Tsoukas (2011).

The key value addition of this paper, however, is the choice of the dependentvariable. Most studies on agency replication-type models make use of ordinal ratings(AAA D 1, AAC D 2, : : : ), or similar logic, and fit an ordered logit regressionmodel.

We do not believe this is the best approach, because the notches are not equidistantin terms of default behavior (eg, there is no meaningful difference in default behaviorin AAC=AA=AA� space). It also completely ignores the variation of CRA defaultrates and assumes that a rating means the same DR behavior over time.

We recommend using the forecasted output of the agency PIT model, ie, the PITPDs and associated DD, as the dependent variables for the agency replication-stylemodels. Table 3 on the facing page clarifies the input, outputs and forecasts of thisway of modeling.

The exact model specification for the agency PIT model is described in our previousstudy (Forest et al 2015). In summary, the agency PIT model makes use of a smootheddefault rate associated with a CRA rating and CCI as explanatory variables to predictPIT PD associated with any associated grade. The agency PIT model is requiredbecause agency ratings are hybrid and not purely PIT or TTC in nature. Our previousstudy indicates that CRA ratings are 75–80% TTC in nature. When we use the outputof the agency PIT model as a dependent variable for the agency replication model,we make sure that we are predicting something that is completely PIT, as shownin (2.1).

Based on this understanding, we can now specify the agency replication model as

PIT_PDi;t D ˚

��DDi;t C bDDGAPI.i/;R.i/;t C �DDGAPI.i/;R.i/;tp

1 � �I.i/;R.i/

�;

PIT_DDi;t D DDi;t C bDDGAPI.i/;R.i/;t C �DDGAPI.i/;R.i/;tp1 � �I.i/;R.i/

;

DDi;t D ˇ0 C ˇkFki;t ; (2.1)

where PIT_PDi;t is the PIT PD for the i th entity at time t , PIT_DDi;t is the PIT DDfor the i th entity at time t , ˚ is the standard normal cumulative distribution function,





AERB 9

TAB

LE

3M

odel

spec

ifica

tion.

Cla

sso

fE

xpla

nat

ory

Dep

end

ent

Mo

del

Mo

del

dev

elo

pm

ent

mo

del

fact

ors

vari

able

ou

tpu

td

ata

set

Ref

eren

ce

Dire

ctto

defa

ultm

odel

,ca

lled

the

agen

cyP

ITm

odel

Sm

ooth

edD

Rco

rres

pond

ing

toag

ency

ratin

gan

dC

CIo

nly

Age

ncy

bina

ryde

faul

tind

icat

or(0

,1)

PIT

PD

and

DD

for

apa

rtic

ular

grad

ean

dtim

e,eg

,PIT

PD

for

BB

Bgr

ade

onJa

nuar

y1,

2015

D0.

21%

Age

ncy

ratin

gsan

dde

faul

tdat

ase

tco

verin

gde

cade

san

dhu

ndre

dsof

defa

ults

For

este

tal

2015

Age

ncy

repl

icat

ion-

styl

em

odel

Obl

igor

’sfin

anci

alan

dju

dgm

enta

lfa

ctor

s

Mod

elou

tput

ofag

ency

PIT

mod

elA

genc

yre

plic

atio

nP

ITan

dT

TC

PD

sC

o-ra

ted

data

set

betw

een

firm

’sor

data

vend

or’s

and

agen

cy-r

ated

wor

ldco

verin

gob

ligor

finan

cial

/judg

men

tal

fact

ors

and

agen

cy-r

atin

g-re

late

din

form

atio

n

Thi

sst

udy





10 G. Chawla et al

DDi;t is the company-specific DD for the i th entity at time t and DDGAPI.i/;R.i/;t

is the industry (I) and region (R) CCI at time t . DDGAP is a quantification ofcredit condition using the PIT–TTC dual-ratings approach. It measures how far anindustry’s or region’s credit conditions are from its long-run average. We also have�DDGAPI.i/;R.i/;tC1 as the change in industry (I) and region (R) CCI (from t tot C 1), b as the regression coefficient that denotes the degree of TTC-ness of DDi;t ,� as the correlation factor related to DDGAP, Fki;t as the kth factor value for the i thentity at time t , ˇ0 as the model intercept and ˇk as the regression coefficient for thekth factor.

We estimate the coefficients (ˇ0, ˇk and b) of the agency replication model byminimizing the sum of squares, the difference between the predicted values of PITDD from the agency replication model and the agency direct model:

min„ƒ‚…fˇ0;ˇk ;bg

SSE DX

i

.PIT_DDi;t � PIT_DDAi;t /

2; (2.2)

where PIT_DDi;t is the PIT DD for the i th entity at time t from the agency replicationmodel (as described in the previous equation), and PIT_DDA

i;t is the PIT DD for the i thentity at time t from the agency PIT model (as described in Forest et al (2015)). Themin function minimizes the sum of square errors, which are assumed to be normallydistributed.

In practice, however, we always see b D 1, ie, the firm’s financial and judgmentalfactors are TTC in nature, and CCIs explain all the default behavior; in which case,the entire formulation can be simplified to TTC PD, ie,

min„ƒ‚…fˇ0;ˇkg

SSE DX

i

.DDi;t � TTC_DDAi;t /

2; (2.3)

where DDi;t is the DD for the i th entity at time t from the agency replication model (asdescribed in the previous equation). This equals TTC_DDi;t when b D 1. TTC_DDA

i;t

is the PIT DD for the i th entity at time t from the agency PIT model (as described inForest et al (2015)).

To simplify this formulation even more, one can make use of an institution’s existingmodel’s scores as an explanatory variable and/or average TTC PDs per CRA rating asthe dependent variable. This reduces to creating a link function between the existingscores and the average TTC default distance per CRA rating. The use of simple logisticregression between existing scores and average agency TTC DDs would mean themodel is as simple as an estimation of two coefficients. However, such a simplificationworks only in certain assumptions; it does not provide a clean mechanism for deriving





AERB 11

dual PDs:

min„ƒ‚…fa0;a1g

SSE DX

i

.DDi;t � averaget .TTC_DDAi;t //

2;

DDi;t D ˚�1

�1

1 C exp.a0 C a1Si;t /

�; (2.4)

where DDi;t is the DD for the i th entity at time t from the agency replication model,as a function of the existing model’s scores, most probably a logistic regression.

We do not recommend further simplifying the agency replication-style models, asthis would lead to errors in calibration and interpretation. Some researchers and modeldevelopers do simplify the formulation even further by making use of default ratescorresponding to CRA ratings rather than PIT or TTC PDs, as shown in (2.5) below.In doing so, the agency PIT model formulation is not used at all, and one simplymakes use of the associated default rates behind them:

min„ƒ‚…fa0;a1g

SSE DX

i

.DDi;t � average.DRAi;t /t

/2;

DDi;t D ˚�1

�1

1 C exp.a0 C a1Si;t /

�; (2.5)

where DDi;t is the DD for the i th entity at time t from the agency replication model,as a function of the existing model’s scores, most probably a logistic regression.

In our experience, this always leads to a bias in estimates. For example, if one buildsthe model on the past six years of CRA default rate history, then there is no recessionin it. If one builds on the past ten to fifteen years of CRA default rate history, thenthis is biased against larger banks and in favor of larger corporates when comparedwith the longer CRA default rate history. A model making full use of CRA historyspanning decades cannot be built due to the lack of a commensurate long history offinancial assessments and judgmental factors as explanatory variables for the left-hand side of the equation. For this reason, we strongly recommend that one makesuse of the agency PIT model derived PIT and/or TTC PDs for the development ofagency replication-style models.

The most important decision that goes into the development of the agency replica-tion model is not the choice of input variables or the link function, but the choice of anappropriate segment in the agency direct model, ie, the calibration curve appropriatefor the portfolio. As we have shown in our previous research (Forest et al 2015), thecalibrations are statistically different for corporates and financial institutions and, toa certain degree, within financial institutions as well.

In Figure 1 on the next page, we show how the choice of the agency PIT model’ssegment-specific TTC PD curve leads to very different PDs when compared with





12 G. Chawla et al

FIGURE 1 Agency PIT model TTC PDs by segment.

100.00

10.00

1.00

0.10

0.01

0

AA

A

AA

+

AA

AA

–

CC

C/CA+ A A–

BB

B+

BB

B

BB

B–

BB

+

BB

BB

–

B+ B B–

CC

C+

%

Average smoothed DRsCorporate average TTC PDsFinancial institutions average TTC PDs

other segments and the average smoothed DRs. For details of the derivation of suchcurves, refer to Forest et al (2015).

3 MODEL DIAGNOSTICS, BENCHMARKING, MONITORING ANDREVIEW

After developing the leading agency replication model, the natural next step is to runmodel diagnostics and conduct sensitivity analysis. Since agency replication-stylemodels are based on thousands of data points over decades of history, they are fairlystable. We have seen R2 of 80% (r � 90%) for most models. The next step is tosee how the model performs on an internal default data set, but only if there aresufficient (at least more than twenty) defaults. A challenger internal direct-to-defaultmodel will most of the time outperform the agency replication-style model whentested on this small sample of default, because the direct-to-default model is builton it. However, direct-to-default models typically underperform on holdout samples.Hence, we recommend bootstrapping or conducting out-of-time/out-of-sample teststo compare the performance of agency replication-style models with the performanceof direct-to-default models.

In certain cases, the internal default data set could be enough to draw robust con-clusions. It could also be very different compared with the agency long-term defaultrates (eg, defaults in the banking sector in the past decade have been very differ-ent compared with the preceding two decades); hence, the agency replication modelwill definitely underperform. If the credit institution strongly believes that the defaultexperience of the last decade is relevant, and they want to develop models based on





AERB 13

such an experience, then such a situation can be easily remedied by selecting a timesegment in the agency PIT model calibration curve. We have proposed such a cal-ibration in our previous study (see Forest et al 2015). Alternatively, one could takethe output of the agency replication model default distances and, further calibratingthem to internal defaults using a parsimonious formulation, thereby maintain the rel-ative weights of explanatory variables and the rank ordering behavior of the agencyreplication model, but add an internal calibration overlay on top. The equation belowshows such an internal calibration add-on performed following the agency replicationstep:

max„ƒ‚…fa0;at g

LL DX

i

di;t ln.PIT_PDINTi;t / C .1 � di;t / ln.1 � PIT_PDINT

i;t /;

PIT_PDINTi;t D ˚.�PIT_DDINT

i;t /;

PIT_DDINTi;t D a0 C a1PIT_DDi;t ;

where PIT_DDi;t is the PIT DD for the i th entity at time t from the agency replicationmodel (as described in previous equations), and PIT_DDINT

i;t is the “internal” PIT DDfor the i th entity at time t , derived using the agency replication model PIT_DDi;t bymeans of a simple linear transformation in DD space using an intercept and slopecoefficient. a0 (the intercept) and a1 (the slope coefficient) are estimated empiricallyby maximizing the log likelihood based on internal defaults. We test the hypothesisthat a0 D 0 and a1 D 1 to check if internal calibration is any different from thatprescribed by the pure agency replication model. di;t is the default indicator for thei th entity at time t .

Agency replication-style models are fairly robust, and almost nothing changes overa quarter. An annual review might reveal changes in agency rating methodology orevolution in financial statement data (eg, the adoption of IFRS accounting standards),which might necessitate adjustments in the model; however, such changes are rare.In our experience of conducting annual reviews of such models, we have rarely madeany changes to them.

After model implementation, the best way to check the ongoing performance ofthe agency replication model is benchmarking the model output with agency ratings.There are likely to be �100 new ratings per quarter and commensurate externalratings for comparison. The agency replication model should be compared with theappropriate benchmark on which it was built. Any large deviations between internaland agency ratings should be studied. As a measure of success, one generally thinksthat a deviation of two or maybe three notches is acceptable for individual cases, andthere should be no significant net overall bias. One typically creates a co-rated matrixbetween internal ratings mapped to agency rating scale versus actual CRA ratingsand expects the diagonal to be dominant with two or three notch deviations around it.





14 G. Chawla et al

The typical expectation is shown in Table 4 on the facing page, where one expects themajority of the population to be along the light gray diagonal, with some individualratings along either side of the diagonal in a symmetric way, so that the net bias iszero.

However, in reality, this never happens, because of the design of agency replication-style models. The net bias is designed to be zero, but the perfect diagonal is not possibleto design. Instead, we often see majority entities along the dark gray diagonal, ie,the internal model overpredicts risk compared with CRA ratings at the highly ratedend and underpredicts risk compared with CRA ratings at the low-rated end. Modeldevelopers should not be surprised with this phenomena and should design tolerancelimits by rating level, eg, two notches in the BBB range and four notches at the AAAand CCC end.

We present here a simple mathematical proof of why this happens. Consider tworating systems,

DD1 D �1 C "1;

DD2 D �2 C "2;

"1 � N.0; �21 /;

"2 � N.0; �22 /; (3.1)

where DD1 and DD2 are two default distance-based rating assessments, � is themean DD and " is the variation, which has a very different interpretation dependingon whether the rating system is PIT, TTC or hybrid.

If the two systems are equally calibrated (eg, in the agency replication-style modelby design), then �1 D �2 D �. Further, assume that the two systems are correlatedwith a parameter �, which means we can define the bivariate normal distribution as

Pr.DD1=DD2 D x/ D N

�� C �1

�2

�.x � �/; .1 � �2/�21

�: (3.2)

A big driver of this distribution is the ratio of variances. In the case of the agencyreplication model, the variance of prediction is smaller when compared with thevariance in output of the agency PIT model, generally because of the estimationprocess as a regression equation even with high R2 � 80%. But, even if we assume�1 D �2 D � , the bivariate normal distribution reduces to

Pr.DD1=DD2 D x/ D N.� C �.x � �/; .1 � �2/�21 /

) E.DD1=DD2 D x/ D �x C �.1 � �/: (3.3)





AERB 15

TAB

LE

4C

o-ra

ted

mat

rixfo

rm

odel

benc

hmar

king

.

Act

ual

Cre

dit

inst

itu

tio

n’s

inte

rnal

rati

ng

map

ped

toS

&P

rati

ng

scal

eu

sin

gin

tern

alm

aste

rsc

ale

and

S&

PT

TC

PD

sS

&P

‚…„

ƒg

rad

eA

AA

AA

CA

AA

A�

AC

AA

�B

BB

CB

BB

BB

B�

BB

CB

BB

B�

BC

BB

�C

CC

CC

CC

–CA

AA

AA

+A

AA

A–

A+

A A–

BB

B+

BB

BB

BB

–B

B+

BB

BB

–B

+B B

–C

CC

+C

CC

–C





16 G. Chawla et al

Clearly,

E.DD1=DD2 D x/

8ˆ<ˆ:

D x when x D �;

> x when x < �;

< x when x > �:

(3.4)

This simply means that when compared with agency ratings, the agency replicationmodel will overpredict risk at the low-risk end and underpredict risk at the high-riskend. Simply speaking, the agency replication model cannot create as many AAA andCCC-C grades based purely on financial and judgmental data compared with actualCRA ratings. This effect reduces greatly if CRA judgmental factors are baked intothe agency replication model and the model is estimated effectively. Since a historicaldata set of such CRA judgmental factors is not possible, this would involve assigningsufficiently powerful coefficients to CRA judgmental factors in the agency replicationmodel, something that regulators would not approve of easily.

4 MODEL USAGE

The model as described should preferably be used in a PIT–TTC dual PD/ratingsframework. In our opinion, both European and US regulators dislike the use of agencyratings as a primary driver, preferring internal assessments of credit risk. However,the agency replication model structure is used to derive the relative and absoluterisk assessment due to its rich default history. We have developed several modelsusing this approach and obtained AIRB waivers (post-financial crisis); hence, this isa tested approach. The resulting models’ TTC PDs are used to drive the calculationof the regulatory capital risk-weighted assets and PIT PDs used for benchmarking.

In our experience, credit officers at the higher end of wholesale credit lending viewcredits in a “ratings” framework. This may or may not be enshrined in a credit institu-tion’s credit policies, but it is definitely embedded in credit practice and processes. Acredit institution’s credit-lending policy could be based on PIT or TTC ratings; eitherway, the agency replication model provides such outputs for credit decision making.In our experience, agency replication-style models serve as the primary models in acredit institution, with agency PIT-based models being used for the benchmarking ofoutput. This can be done either on a PIT basis or on a TTC basis, but it must be doneconsistently. For this reason, we highly recommend developing agency replicationand agency PIT-style models in a consistent PIT–TTC framework.

5 SUMMARY

The use of CRA ratings in a credit institution’s lending practices has been severelycriticized since the financial crisis. Recent studies also show that biases in benchmarks





AERB 17

can be created when using CRA long-term default history. Despite criticism andregulatory constraints, CRA ratings and default time series are a rich source of dataand a leading choice for building models when internal data is not sufficient to builddirect-to-default models. In this paper, we proposed a class of agency replication-style models that make use of an entity’s financial and judgmental information asthe explanatory variable and agency PIT model’s PIT and/or TTC PD output as thedependent variable.

We have provided a full formulation of such a model, built in a PIT–TTC frame-work, and we have motivated the reasoning behind developing it within the PIT–TTCframework. Developing an integrated, dual-ratings approach is the only way, in ourexperience, that multiple regulatory objectives for capital driven by TTC measures andstress testing, and IFRS9 driven by PIT measures, can be consistently and accuratelysupported.

We then demonstrated some commonly made simplifications to the model basedon reasonable assumptions. We showed how using agency default rate informationinstead of agency PIT model PIT/TTC PDs as the dependent variable can introducebiases. We shared our experience of building models for portfolios such as largecorporates, banks and insurance companies. Such models received AIRB approvaland, hence, we colloquially call them AERB models.

DECLARATION OF INTEREST

The authors report no conflicts of interest. The authors alone are responsible for thecontent and writing of the paper.

REFERENCES

Aguais, S. D., Forest, L. R. Jr., Wong, E. Y. L., and Diaz-Ledezma, D. (2004). Point-intime versus through-the-cycle ratings. In The Basel Handbook: A Guide for FinancialPractitioners, Ong, M. (ed). Risk Books, London.

Aguais, S. D., Forest, L. R. Jr., King, M., Lennon, M. C., and Lordkipanidze, B. (2007).Designing and implementing a Basel II compliant PIT–TTC ratings framework. In TheBasel Handbook II:A Guide for Financial Practitioners, Ong, M.(ed).Risk Books, London.

Altman, E. I., and Rijken, H. A. (2004). How rating agencies achieve rating stability. Journalof Banking and Finance 28, 2679–2714.

Black, F., and Scholes, M. S. (1973).The pricing of options and corporate liabilities. Journalof Political Economy 81(3), 637–654.

Chawla, G., Forest, L., and Aguais, S. (2013). Deriving point-in-time (PIT) and through-the-cycle (TTC) PDs. URL: https://en.wikipedia.org/wiki/Probability_of_default#Deriving_Point-in-Time.28PIT.29_and_Through-the-cycle.28TTC.29_PDs.

Financial Crisis Inquiry Commission (2011). The financial crisis inquiry report. Report,FCIC. URL: www.gpo.gov/fdsys/pkg/GPO-FCIC/pdf/GPO-FCIC.pdf.





18 G. Chawla et al

Forest, L., Chawla, G., and Aguais, S. D. (2013). Comment in response to “A methodologyfor point-in-time–through-the-cycle probability of default decomposition in risk classifi-cation systems” by M. Carlehed and A. Petrov. Journal of Risk Model Validation 7(4),73–78.

Forest, L., Chawla, G., and Aguais, S.D. (2015).Biased benchmarks.Journal of Risk ModelValidation 9(2), 1–1.

Kamstra, M., Kennedy, P., and Suan, T. K. (2001). Combining bond rating forecasts usinglogit. Financial Review 37, 75–96.

Legal Information Institute (2015). Application for registration as a nationally recog-nized statistical rating organization. Code of Federal Regulations 17 240.17g-1, CornellUniversity Law School. URL: www.law.cornell.edu/cfr/text/17/240.17g-1.

Merton, R. C. (1974). Theory of rational option pricing. Bell Journal of Economics andManagement Science 4(1), 141–183.

Minardi, A. M. A. F., Sanvicente, A. Z., and Artes, R. (2007). A methodology for estimatingcredit ratings and the cost of debt for business units and privately held companies.Research Paper, Credit Research Centre.

Mizen, P., and Tsoukas, S. (2011). Forecasting US bond default ratings allowing for pre-vious and initial state dependence in an ordered probit model. International Journal ofForecasting 28(1), 273–287.

Moody’s (2008). Moody’s senior ratings algorithm and estimated senior ratings. Moody’sGlobal Credit Policy Report, August, Moody’s Investor Service.

Prudential Regulation Authority (2013). Internal ratings based approaches. SupervisoryStatement 11/13, PRA. URL: www.bankofengland.co.uk/pra/Documents/publications/ss/2013/ss1113.pdf.

Standard & Poor’s (2002). Corporate ratings criteria. Report, S&P.






Research Paper

A mean-reverting scenario design model tocreate lifetime forecasts and volatilityassessments for retail loans

Joseph L. Breeden1 and Sisi Liang2

1Prescient Models LLC, 1600 Lena Street, Suite C24, Santa Fe, NM 87501, USA;email: [email protected] Platinion, 430 Park Avenue, New York, NY 10022, USA;email: [email protected]

(Received June 2, 2015; revised September 23, 2015; accepted September 29, 2015)

ABSTRACT

Lifetime loan forecasting has become essential to lender risk management and prof-itability. Loan-pricing models require forecasts over the life of the loan. Currentexpected credit loss (CECL) calculations proposed by the US Financial AccountingStandards Board (FASB) (2012) and included in International Financial ReportingStandard 9 (IFRS9) require lifetime forecasts. In both cases, we cannot create fore-casts that assume the current or historic environment persists for many years intothe future. Instead, a more reasonable approach is to use macroeconomic scenariosfor the near term and then relax onto the long-run average for future years. In thecurrent paper, we develop a modeling framework that can incorporate mean-revertingscenarios into any scenario-based forecasting model. Using prior economic condi-tions, we create an environmental index with which to calibrate a discrete versionof an Ornstein–Uhlenbeck (OU) mean-reverting model. OU models are best appliedto stationary processes, which is true for the environment function derived fromage–period–cohort-type (APC-type) models. The mean-reverting model is used totransition from the near-term macroeconomic scenario to the long-run average to

Corresponding author: J. L. Breeden Print ISSN 1753-9579 j Online ISSN 1753-9587Copyright © 2015 Incisive Risk Information (IP) Limited

19




20 J. L. Breeden and S. Liang

provide stable lifetime estimates for long-duration loans. We demonstrate this frame-work with a loan-level forecasting model using an age–vintage–time structure forretail loans, in this case, a small auto loan portfolio. The loan-level age–vintage–timemodel is similar in structure to an APC model, but it is estimated at the loan-level forgreater robustness on small portfolios. The environment function of time is correlatedto macroeconomic factors, and it is then extrapolated backward in time before theperformance data to stabilize the trend of the environment function. This frameworkis in line with the explicit goals of the new FASB loan-loss accounting guidelines. Inaddition, this model provides a simple mechanism to facilitate the transition betweenpoint-in-time and through-the-cycle economic capital estimates with an internallyconsistent model.

Keywords: forecasting; mean-reverting models; credit risk; age–period–cohort (APC) models;current expected credit loss (CECL); International Financial Reporting Standard 9 (IFRS9).

1 INTRODUCTION

Increasingly, retail lenders require accurate lifetime loss estimates that incorporatecurrent economic conditions, near-term economic scenarios and reasonable long-termextrapolations. The US Financial Accounting Standards Board (FASB) has proposedjust such a view as part of their new procedure for computing lifetime losses. Inaddition, when lenders price loans and estimate lifetime margin, a similar view isnecessary.

In Figure 1 on the facing page, we show the basic concept.For any model that includes macroeconomic factors, Ei .t/, we can aggregate those

factors into a single index, H.t/. Age–period–cohort (APC) models (Glenn 2005;Mason and Fienberg 1985) directly produce a function of time that captures the neteffect of macroeconomic conditions over time via a fixed effect in time rather than byexplicitly including macroeconomic factors. With either approach, we can take theestimated index H.t/ as a starting point for our analysis.

Due to the intrinsic autocorrelation structure of the economy, economists are betterthan random at predicting six to twelve months ahead. Assuming that a favoritescenario is taken by the lender for the near future, the goal is to create a model thatrelaxes from the macroeconomic effects produced by the economist’s scenario intothe long-run average effect via an appropriate model.

To create this relaxation onto long-run averages, we could assume a desired func-tional form. For example, an extrapolation based upon a hyperbolic tangent func-tion could provide the relaxation shown in Figure 1 on the facing page. Althoughwe can design aesthetically desirable extrapolations in this way, we do not obtain





A mean-reverting scenario design model for retail loans 21

FIGURE 1 A timeline of macroeconomic effects representing the assumed transitionsfrom history to a near-term macroeconomic scenario, eventually relaxing onto long-runconditions.

Mac

roec

onom

ic im

pact

s

History

Today

Predictable Unpredictable

12 24

the conditional distributions for future environments that are necessary for risk andcapital assessment.

By adopting a stochastic process for the extrapolation, we can obtain a reasonableextrapolation as well as an uncertainty interval. The Ornstein–Uhlenbeck (OU) pro-cess (Uhlenbeck and Ornstein 1930) is the only nontrivial stochastic process that isstationary, Gaussian and Markovian (Bibbona et al 2008). The Vasicek (1977) modelis an OU process, as are many other interest-rate models.

In this application, we adopt the OU process just to extrapolate macroeconomicimpacts. Any credit risk model with separable macroeconomic effects (Banerjee andCanals-Cerdá 2012; Bellotti and Crook 2009; Breeden et al 2008; Malik and Thomas2008; Rösch and Scheule 2007) can have those effects collected into an index. “Sep-arable” means that there are no cross-terms in the model between economic andcredit risk or loss-timing factors. Equivalently, economic factors would affect allloans proportionally within a given segment. If a regression structure is employed forincorporating macroeconomic effects, it would take the form shown in the followingequation, where all such economic factors are collected into a single index H.t/:

H.t/ DXiD1

NciEi .t/ C �t : (1.1)

This is equivalent to the environment function produced viaAPC, and it is conceptuallyequivalent to the systematic risk factor used in the derivation of the Vasicek formula:

logit.pi .a; v; t// D F.a/ C G.v/ C H.t/: (1.2)

Although the OU process can generate individual stochastic simulations, we areprimarily interested in the closed-form solutions for the expected mean and varianceof the process as a very computationally efficient way to meet the modeling needs of






lenders. This approach also has the added advantage of merging near-term point-in-time (PIT) estimates with long-run through-the-cycle (TTC) estimates, allowing usto create a single economic capital model that provides both answers.

2 MODELING APPROACH

For any credit risk model that incorporates a set of additive macroeconomic factors,such as that shown in (1.1), the following steps are appropriate.

Step 1 Create a forecast model that can take macroeconomic scenarios as inputs.

Step 2 Collect the macroeconomic effects of the credit risk model into a single index.

Step 3 Calibrate a mean-reverting model to the macroeconomic effects.

Step 4 Obtain a macroeconomic scenario from economist(s).

Step 5 Overlay the mean-reverting model onto the macroeconomic scenario.

The first two steps in this sequence are obvious, so we will focus on the remainingthree.

2.1 The Ornstein–Uhlenbeck process

The OU process is a continuous-time stochastic process, often described in the contextof Brownian motion:

dxt D �.� � xt / dt C � dWt : (2.1)

For a studied property xt , � is the long-run mean of the process, � is related to therelaxation time and � is related to the variance.

In discrete time, the OU process simplifies to a structured autoregressive order 1,AR(1), process. In the present context, xt is replaced with the environmental effectsH.t/ from (1.1):

�H.t/ D �.� � H.t//�t C �t ; (2.2)

where

� D d � �2

2�; �t � N.0; �/; (2.3)

and d is the drift term.Given this process, the expected mean and variance are

E.H.t// D .1 � e��.t�t0//� C e��.t�t0/H.t0/; (2.4)

Var.H.t// D �2

2�.1 � e�2�.t�t0//: (2.5)






In the limit, as t ! 1, this becomes

limt!1

E.H.t// D �; (2.6)

limt!1

Var.H.t// D �2

2�: (2.7)

Calibrating the discrete OU process to loan data is straightforward, except for thelength of history typically available. Under the Basel II minimum requirement of fiveyears of data to create probability of default (PD) models, in the majority of countriesat most one economic cycle would be present in the data. This is a dangerously shorttime interval from which to calibrate the mean and variance of the process.

The solution is simple, though it has some caveats. If we assume that the macro-economic effects are quantified as in (1.1) over the period of the loan-performancedata, typically five years or so, we still have the option to extrapolate (1.1) backwardover previous economic periods. Although loan performance data is rarely availablefor earlier periods, macroeconomic indexes are generally available for multiple priordecades, as shown in Figure 2 on the next page. The multi-decade extrapolation ofH.t/ is then used to calibrate the discrete OU process.

Long-run extrapolations of credit risk models calibrated over comparatively shortperiods can have trend instabilities. This instability arises naturally from the rela-tionship between the age of the loan a, the origination date of the loan v and theperformance observation date t , wherein a D t � v. This issue is discussed in detailby Breeden and Thomas (2016) for APC models, along with suggested solutions;however, the problem is a general one, as highlighted by Breeden et al (2015). Nev-ertheless, the good news for modeling loan performance data is that the nonlinearstructure in the time series is fully estimable (Holford 1983). So, as long as we havea full economic cycle to model and demonstrate that there is no long-term seculartrend, then trend ambiguity is not a problem.

For the rest of the analysis here, we assume that the trend extrapolation issue hasbeen properly addressed such that H.t/ is a stationary series. In the present context, weuse a simple linear detrending of the backward extrapolation to achieve stationarity.

Figure 2 on the next page provides a visualization of this backward extrapolation,where H.t/ is first calibrated against the loan performance data (solid line to theright) and then extrapolated backward through previous decades (long dashes). Overthe full extrapolation of H.t/, we compute the mean and deviation of H.t/ as HTTC

and �TTC, respectively. This provides part of the OU calibration, with

� D HTTC (2.8)

and

�2 D 43

ln.2/�2TTC: (2.9)






FIGURE 2 A backward extrapolation of the macroeconomic model, fitted to a sample ofloan portfolio performance.

σTTC

HTTC

A

The extrapolation is used to quantify TTC values of mean and deviation for macroeconomic effects. The verticalline separates the in-sample data (right) from the backward extrapolation (left). The line labeled “A” is H.t/ DPN

iD1 ci Ei .t/ C "t .

The last remaining parameter is the relaxation time, for which we have theexpression

t1=2 D ln.2/

�: (2.10)

This can be estimated from H.t/, but we can also draw on prior Monte Carlo studiesof retail lending models that show t1=2 to be on the order of 1.5–2 years (Breeden andIngram 2009). This should be viewed as a tunable parameter, and possibly estimable,given sufficiently long time series.

Once the calibration is known, the subsequent steps are simply procedural. Thepractitioner must select an economic scenario from their favorite economist orprovider, choose a transition point beyond which the scenario will no longer be takenas a given and extrapolate from that t0 using the OU model.

3 NUMERICAL EXAMPLE

To provide a numerical example, we considered a regional auto loan portfolio withapproximately US$400 million in direct and indirect loans. The data was initiallyanalyzed using an APC model, which has similar features to survival models (Hos-mer and Lemeshow 1999; Therneau and Grambsch 2000), and proportional hazardsmodels (Cox and Oakes 1984; Efron 2002). A loan-level APC model of default ratewas created by extending previous work by Breeden (2013).

As shown in (1.2), a lifecycle function with age of the loan F.a/, a credit riskfunction of vintage G.v/ and an environment function of calendar date H.t/ wereestimated. APC models are often estimated nonparametrically or via splines (Holford






FIGURE 3 PD lifecycle function with account age measured for a small US auto loanportfolio.

Age (monthly)

Pro

babi

lity

of d

efau

lt

0

0.001

0.002

0.003

0.004

0.005

0 12 24 36 48 60 72 84

The y-axis is scaled to the monthly PD rate.

FIGURE 4 PD credit risk function with vintage measured for a small US auto loan portfolio.

2000 2005 2010Vintage

Cha

nge

in lo

g od

ds

0.5

0

–0.5

The y-axis measures the change in log odds of default through calendar date.

2005). Bayesian estimation is also available (Schmid and Held 2007). The decompo-sition here was estimated via splines, as shown in Figure 3, Figure 4 and Figure 5 onthe next page.






FIGURE 5 PD environment function with time measured for a small US auto loan portfolio.

2007 2008 2009 2010 2011 2012 2013Time

Cha

nge

in lo

g od

ds1.0

0.5

0

–0.5

–1.0

The y-axis measures the change in log odds of default by vintage.

The functional form for an APC model of PD is

logit.pi .a; v; t// D F.a/ C G.v/ C H.t/: (3.1)

The functions in the APC model can be estimated either from vintage-aggregate orloan-level panel data. In vintage-aggregate data, for each vintage for each observationperiod, the total number of defaults and the previous number of active accounts arerecorded, giving us the conceptual definition of monthly PD as

PD.v; t/ D default accounts.v; t/

active accounts.v; t � 1/: (3.2)

For loan-level panel data, each account is one row of the panel for each month it wasactive, recording either 0 (non-default) or 1 (default). The loan no longer appearsafter it either defaults (the terminal event) or attrites (censored). This structure hasthe effect of measuring PD.t/, conditioned on the account being active at t � 1, andequivalent to the structure in (3.2).

The environment function H.t/ was fitted to macroeconomic data for unemploy-ment rate and changes in house prices, as shown in Table 1 on the facing page. Thetrend in this model was stabilized via the procedure described in Breeden and Thomas(2016) and extrapolated backward through to 1990, when all of the macroeconomicfactors first became available for the portfolio region.






TABLE 1 Regression output for fitting the environment function H.t/ to macroeconomicfactors.

StandardEstimate error t value Pr.>jtj/

(Intercept) 5.68220 0.51939 10.940 < 2e-16Unemp.MovingAvg.Lm2.W10 2.02001 0.20442 9.881 < 2e-16HPI.LogRatio.L9.W18 �2.60794 0.45811 �5.693 < 2.7e-08

In the model, Lm2.W10 means a lag of �2 months (into the future) and a window of ten months, L9.W18 is a lagof nine months and a window of eighteen months. In addition, residual standard error D 0.2575 on 340 degrees offreedom, multiple R-squared D 0.4055, adjusted R-squared D 0.4003, F -statistic D 77.32 on 2 and 340 degreesof freedom, and p-value < 2.2e-16.

FIGURE 6 Calibration of the OU process to the backward extrapolation of the macro-economic model, fitted to H.t/ for a small US auto loan portfolio.

1995 2000 2005 2010 2015 2020Time

History Scenario

Env

ironm

ent f

unct

ion

0.6

0.4

0.2

0

–0.2

–0.4

The OU model was calibrated as described above to produce the result in Figure 6.HTTC D �0:07 ˙ 0:035 is shown in Figure 6 as the central dashed line, and �TTC D0:30 is shown as the bounding dashed lines. The expected mean of the OU process isthe smooth extrapolation to the future, and the expected deviation of the OU processis given as the expanding dotted lines.

Figure 7 on the next page shows an expanded view of the OU process with the effectof the provided macroeconomic scenario for the first twelve months, from early 2013through to early 2014, after which the OU process takes over.






FIGURE 7 An expansion of Figure 6, highlighting the transition from a macroeconomicscenario to the OU process.

2011 2012 2013 2014 2015 2016 2017Time

History Scenario

HTTC

Env

ironm

ent f

unct

ion

0.6

0.4

0.2

0

–0.2

–0.4

The expected mean and deviations from the OU process can then be fed as scenariosinto the forecast model. For the purposes of computing economic capital, higherconfidence intervals can be used than the one-standard-deviation bands shown inFigure 7.

4 CONCLUSIONS

The application of an OU process for scenario design allows us to create scenario-based forecasts that relax from current conditions or the end of a given macroeconomicscenario into the long-run average environment over reasonable time scales. Thismeets the proposed FASB current expected credit loss (CECL) requirement of incor-porating current economic conditions into loss forecasts while relaxing onto long-runexpectations for lifetime forecasts.

From a loan-pricing perspective, this provides a mechanism for incorporating cur-rent conditions into pricing without the overly pessimistic or optimistic assumptionthat current conditions persist for the life of the loan. Sitting at the peak of a recession,we would be pricing for bad economic conditions in the near term, but a long-runimprovement to the mean. When sitting at the best point of an economic expansion,we could assume near-term continued good conditions but then deteriorate to thelong-run average.






The choice of an OU model was natural given the Basel II assumption that long-run PDs exist. The assumed existence of a long-run PD under Basel II implies thatPD is stationary. For a specific portfolio where volume and credit risk have changedsystematically through the history, the portfolio PD time series might not be stationary,but an equivalent and more robust assumption is to take the environment function H.t/

as stationary. Stationarity also follows from the precedent of using OU processes inmodeling macroeconomic variables.

Without using an OU mean-reverting process, most practitioners revert to discon-tinuous approaches. The first year might be modeled using a stress test-type modelwith an economic scenario, while “out years” are modeled with TTC models. Suchapproaches are commonly considered for IFRS9 and CECL, but the abrupt transitionbetween models is far from robust, with results that are very sensitive to the discon-nect between the two parts. Using a mean-reverting model offers a smooth transitionthat can carry a single model through the transition from PIT to TTC.

Since only the expected mean and deviation are required, implementation of adiscrete-time OU model is essentially trivial, once a stable measure of macroeconomiceffects to loan portfolio losses has been obtained.


The authors report no conflicts of interest. The authors alone are responsible for thecontent and writing of the paper.

REFERENCES

Banerjee, R., and Canals-Cerdá, J. J. (2012). Credit risk analysis of credit card portfoliosunder economic stress conditions. Technical Report, Research Department, FederalReserve Bank of Philadelphia.

Bellotti, T., and Crook, J. N. (2009). Credit scoring with macroeconomic variables usingsurvival analysis. Journal of the Operational Research Society 60(12), 1699–1707.

Bibbona, E., Panfilo, G., and Tavella, P. (2008). The Ornstein–Uhlenbeck process as amodel of a low pass filtered white noise. Metrologia 45, S117–S126.

Breeden, J. L. (2013). Incorporating lifecycle and environment in loan-level forecasts andstress tests. Credit Scoring and Credit Control XIII Conference Paper, Credit ResearchCentre.

Breeden, J. L., and Ingram, D. (2009). Monte Carlo scenario generation for retail loanportfolios. Journal of the Operational Research Society 61, 399–410.

Breeden, J. L., and Thomas, L. C. (2016). Solutions to specification errors in stress testingmodels. Journal of the Operational Research Society, forthcoming.

Breeden, J. L., Thomas, L. C., and McDonald, J. III (2008). Stress testing retail loanportfolios with dual-time dynamics. Journal of Risk Model Validation 2(2), 43–62.






Breeden, J. L., Bellotti, A., and Yablonski, A. (2015). Instabilities using Cox PH for forecast-ing or stress testing loan portfolios. Credit Scoring and Credit Control XIV ConferencePaper, Credit Research Centre.

Cox, D. R., and Oakes, D. O. (1984). Analysis of Survival Data. Chapman and Hall, London.Efron, B. (2002). The two-way proportional hazards model. Journal of the Royal Statistical

Society B 64, 899–909.Financial Accounting Standards Board (2012). Financial instruments: credit losses (sub-

topic 825-15). Exposure Draft, Financial Accounting Series, December, FinancialAccounting Foundation.

Glenn, N. D. (2005). Cohort Analysis, 2nd edn. Sage.Holford, T. R. (1983). The estimation of age, period and cohort effects for vital rates.

Biometrics 39, 311–324.Holford, T. R. (2005). Age–period–cohort analysis. In Encyclopedia of Statistics in Behav-

ioral Science. Wiley.Hosmer, D. W. Jr., and Lemeshow, S. (1999). Applied Survival Analysis: Regression

Modeling of Time to Event Data. Wiley Series in Probability and Statistics. Wiley.Malik, M., and Thomas, L. C. (2008). Modelling credit risk of portfolios of consumer loans.

Journal of the Operational Research Society 61(3), 411–420.Mason, W. M., and Fienberg, S. (1985). Cohort Analysis in Social Research: Beyond the

Identification Problem. Springer.Rösch, D., and Scheule, H. (2007). Stress-testing credit risk parameters: an application to

retail loan portfolios. Journal of Risk Model Validation 1(1), 55–75.Schmid, V. J., and Held, L. (2007). Bayesian age–period–cohort modeling and prediction

BAMP. Journal of Statistical Software 21(8), 1–15.Therneau, T. M., and Grambsch, P. M. (2000). Modeling Survival Data: Extending the Cox

Model. Springer.Uhlenbeck, G. E., and Ornstein, L. S. (1930). On the theory of Brownian motion. Physical

Review 38, 823–841.Vasicek, O. (1977). An equilibrium characterisation of the term structure. Journal of

Financial Economics 5(2), 177–188.





We also offer daily, weekly and live news updates

Visit Risk.net/alerts now and select your preferences

Asset Management

Commodities

Derivatives

Regulation

Risk Management

CHOOSE YOUR PREFERENCES

Get the information you needstraight to your inbox

RNET16-AD156x234-NEWSLETTER.indd 1 21/03/2016 09:27


Research Paper

Downside risk measure performance in thepresence of breaks in volatility

Johannes Rohde

Institute of Statistics, School of Economics and Management, Leibniz University of Hannover,Königsworther Platz 1, D-30167 Hannover, Germany; email: [email protected]

(Received June 16, 2015; revised September 8, 2015; accepted September 8, 2015)

ABSTRACT

The accurate evaluation of a risk measure employed by a financial institution is ofhigh importance in view of that institution’s minimum capital requirement. Having asensitive reaction to breaks in the volatility of the profit-and-loss process is a desir-able property of the underlying measure. This paper proposes a loss function-basedframework for the comparative measurement of the sensitivity of quantile downsiderisk measures to breaks in volatility or distribution. We do this by extending the modelcomparison approach introduced by Lopez in 1998. Value-at-risk and expected short-fall (ES) are contrasted over realistic evaluation horizons within a broad simulationstudy, in which numerous settings involving volatility breaks of different intensitiesand several data-generating processes are checked by employing a magnitude-typeloss function. As a result, it can generally be noted that ES appears to be the supe-rior measure in terms of its ability to identify breaks in the volatility. In addition, anempirical study, in which data from six stock market indexes are used, demonstratesthe applicability of this procedure and reconfirms the findings from the simulationstudy.

Keywords: value-at-risk (VaR); expected shortfall (ES); structural break; break in volatility; lossfunction; market risk.

Print ISSN 1753-9579 j Online ISSN 1753-9587Copyright © 2015 Incisive Risk Information (IP) Limited

31




32 J. Rohde

1 INTRODUCTION

During the past few decades, a growing awareness of the importance of accurate riskmanagement in financial institutions has evolved. Since the 1996 amendment to theBasel Accord on regulatory capital for market risk, banks have been demanded toimplement internal models for measuring market risk (Basel Committee on Bank-ing Supervision 1996, 1997).1 A main objective of Basel II (Basel Committee onBanking Supervision 2004) addresses the calculation of risk-sensitive minimum cap-ital requirements and the definition of standards for the quantitative measurement offinancial risk. In this context, value-at-risk (VaR) approaches are recommended asappropriate instruments for assessing the market risk exposure of a financial insti-tution, and they are widely used in financial risk management. However, the recentreviews of the Basel Accords redefine the capital rules for market risk and include theproposition to gradually replace VaR with expected shortfall (ES) by 2019 (see theconsultative documents of Basel III, issued by the BCBS (2012, 2013).

Hendricks and Hirtle (1997) point out that the benefits which arise from a model-based capital requirement are undermined by the use of incorrect models. This indi-cates that the evaluation of the accuracy of underlying risk models has to be of primaryconcern for banks and regulatory authorities. Backtesting frameworks represent thepreferred tool for evaluating the performance of risk measures, even though numer-ous tests suffer from a lack of statistical power when following the recommendationof the Basel Committee to adopt an evaluation horizon of one year. This constitutesa widely examined issue, which is described in the works of Lucas (2001), Camp-bell (2005), Nieppola (2009) and Røynstrand et al (2012), among others. In order toovercome this drawback, Lopez (1998) introduces the loss-function approach as analternative evaluation method that is not based on hypothesis testing but rather drawsupon forecast evaluation techniques. Campbell (2005) refers to the ability to target thespecific concerns of a financial institution by choosing a certain type of loss functionand emphasizes its usefulness for distinguishing between competing risk models.

Financial risk is often identified with the behavior of an asset’s volatility. Con-sequently, the evaluation and the accuracy of the risk model strongly depends onthe variance of the profit-and-loss (P&L) series of the financial institution. A lot ofevidence for occasional structural breaks in the volatility of financial time series isprovided by Lamoureux and Lastrapes (1990) and Amihud and Mendelson (1991),and more recently in the works of Diamandis (2008) and Eichengreen et al (2012),among others. Hence, a financial institution should preferably employ a risk measurethat is characterized by a reaction of sufficient sensitivity to the occurrence of a break

1 All remarks about the Basel Accords refer to the frameworks issued by the Basel Committee onBanking Supervision (BCBS).





Downside risk measure performance in the presence of breaks in volatility 33

in volatility in order to ensure an immediate adjustment of the underlying risk mea-sure. While a variety of research addresses the development of testing procedures withregard to the detection of a structural change (of unknown date) and the estimation ofthe break date (see Hansen (2001) and Perron (2006) for an overview of the testingand estimation methodology), the literature on the characteristics of risk measuresthus far lacks an analysis of their performance in the presence of a structural breakin volatility, or a substantiated recommendation on which measure to give priority insuch a case.

This paper provides a comparison of the two most-used downside risk measures,VaR and ES, with regard to their responsiveness to structural breaks in volatility anddistribution. The remainder of the paper is organized as follows. Section 2 providesa brief literature overview of the current status of research on risk measures and theprevious utilization of loss functions for risk evaluation. In Section 3, the most com-mon downside risk measures are reviewed, and whether they fulfill mathematical andpractical requirements on measuring financial risk is assessed. Section 4 introducesthe usage of loss functions for risk evaluation and describes the extension of the lossfunction-based framework suggested by Lopez (1998) for comparing the sensitivityof risk measures in response to a structural break in volatility, as well as in reactionto a change in distribution. In Section 5, the performance of VaR and ES for the mostcommon data-generating processes (DGPs) and realistic evaluation horizons is sur-veyed within a broad simulation study, which accounts for the direction and intensityof the volatility break. In Section 6, the simulation results are reconfirmed by applyingthe proposed evaluation technique to several stock index series. The conclusion of thework is provided in Section 7.

2 LITERATURE REVIEW

Even though VaR is the most commonly used risk measure in financial risk manage-ment, the suitability of VaR has been questioned since it became the benchmark toolfor assessing the exposure to market risk. Hendricks (1996) considers different VaRapproaches for simulated portfolios. While he can attest to an accurate performancefor all examined methods at the 95% level, an understatement of the actual risk can beobserved at the 99% level. This finding is endorsed by Bao et al (2006), who inves-tigate the predictive performance of VaR models in terms of several emerging Asianeconomies within the financial crisis of the late 1990s. Berkowitz and O’Brien (2002)present evidence of VaR model performance for large trading firms and conclude thatthe reportedVaR estimates do not appropriately indicate the firms’actual portfolio risk.Moreover, simple top-down autoregressive moving average–generalized autoregres-sive conditional heteroscedasticity (ARMA–GARCH) models appear to outperformbottom-up VaR approaches in terms of their forecasting performance, while VaR is





34 J. Rohde

not able to reflect volatility changes in the P&L series of the firms. Arising from allof these shortcomings in performance, and because of some further theoretical draw-backs (see Section 3.1 for a discussion), Acerbi and Tasche (2002) are surprised thatVaR has been adopted by essentially all banks and regulators. Recent research aboutthe improvement of the accuracy of VaR forecasts includes, for example, the paperby Halbleib and Pohlmeier (2012).

ES has frequently been considered as an alternative to VaR for evaluating marketrisk exposure; numerous academic contributions of the more recent past deal withthe comparison of VaR and ES by focussing on different aspects. Yamai and Yoshiba(2002) provide an overview of these studies and demonstrate that rational investorsare often misled by employing VaR. This can be mitigated by adopting ES as the mainrisk measure. However, an important requirement for its practicality is the availabilityof efficient methods for backtesting ES. The findings of Basu (2006), who examinesthe effect of stress scenarios on the performance of VaR and ES, indicate that theresponsiveness of VaR to shocks for historical simulations remains low. ES, mean-while, is more suitable for capturing the effect of stress. Chen (2014) evaluates theeffectiveness of the recent Basel reforms with regard to the regulatory reservationsarising from the use of VaR. While Chen criticizes ES for its lack of elicitability,and, hence, denies the reliability of the results that come from backtesting ES, Acerbiand Székely (2014) propose three methodologies for backtesting ES and claim thatelicitability is irrelevant for backtesting risk measures. Emmer et al (2014) supportthis result, even while conceding that more data is required for these procedures thanfor backtesting VaR in order to reach an equivalent level of certainty. An analogue tothe well-known conditional backtesting framework for VaR estimates is suggested byEscanciano and Du (2015) for the evaluation of ES forecasts.

Loss functions are a widely used tool for assessing the prediction performance ofcompeting models. After Lopez (1998) proposed three different types of loss func-tions and their utilization in measuring the accuracy of VaR estimates, this methodbecame an established procedure for the evaluation of risk measures as well. Gener-alizations of this conception are provided by the works of Lopez (2001) – in whicheconomic loss functions are incorporated into a volatility forecasting framework –and Caporin (2008), who introduces a new set of loss functions for the purpose ofcomparing VaR measures in the presence of long-memory effects. In further papers,loss-function techniques are applied for the evaluation of the forecasting performanceof several rival volatility models in VaR frameworks. These include González-Riveraet al (2004), who employ a goodness-of-fit loss function based on a VaR calcula-tion, and Amendola and Candila (2014), who suggest an asymmetric loss function forthis purpose. Degiannakis et al (2013) employ a quadratic loss function in order toexamine whether conditional volatility models accounting for long memory outper-form those implying short memory when forecasting VaR and ES. In a more recent






paper, Abad et al (2015) investigate whether the choice of a certain type of loss func-tion affects the comparison of VaR models by additionally accounting either for thefirm’s or the regulator’s point of view. Moreover, Campbell (2005) describes how lossfunction-based backtests can be conducted and remarks on the enhanced flexibilityof this approach.

3 MEASURING DOWNSIDE RISK

The intention of applying a risk measure to some random variable X modeling theP&L of a portfolio is to quantify its underlying risk and determine a minimum capitalrequirement to make the risky position acceptable to the regulatory authorities.

Following the axiomatic approach initiated byArtzner et al (1999), a risk measure issupposed to feature certain desirable properties in order to be suitable for measuringfinancial risk. To this purpose, consider a linear space H of measurable functionsX W ˝ ! R, where ˝ contains a fixed and finite set of possible future scenarios.Then, a mapping � W H ! R [ fC1g is called a coherent risk measure for H ifaxioms (i)–(iv) below are fulfilled.

(i) Monotonicity:

X1

almostsurely

6 X2I X1; X2 2 H ) �.X2/ 6 �.X1/:

(ii) Subadditivity:

X1; X2; X1 C X2 2 H ) �.X1 C X2/ 6 �.X1/ C �.X2/:

(iii) Positive homogeneity:

a 2 R>0I X; aX 2 H ) �.aX/ D a�.X/:

(iv) Translation invariance:

a 2 RI X 2 H ) �.X C a/ D �.X/ � a:

Note that for a D 0, axiom (iii) implies normalization for �, ie, �.0/ D 0.Föllmer and Schied (2002) propose a revision of the concept of coherent risk

measures by replacing (ii) and (iii) with a weaker axiom. A risk measure that satisfiesthe axioms (i) and (iv) belongs to the class of convex risk measures if, in addition, itfulfills the following axiom for X1; X2 2 H and � 2 .0I 1/.

(v) Convexity:

�.�X1 C .1 � �/X2/ 6 ��.X1/ C .1 � �/�.X2/:

Subject to the validity of (iii), axioms (ii) and (v) are equivalent (see Föllmer andSchied 2010).





36 J. Rohde

3.1 Value-at-risk

Let FX .�/ be the cumulative distribution function (cdf) of the P&L random variableX . Then, the VaR for an exogenously given confidence level 1 � ˛ is determined by

VaR˛.X/ D inffx 2 R W P.X > x/ 6 1 � ˛gD inffx 2 R W FX .x/ > ˛g D F �1

X .˛/; (3.1)

whereby ˛ WD P.X 6 VaR˛.X// holds, and the second equality only applies forparametricVaR approaches.VaR can easily be interpreted as the return that is exceededin 100.1 � ˛/% of all periods.2 The simplicity of interpretation is one of the mainreasons that VaR has evolved into an industry standard tool for financial institutions.Several techniques for the estimation of VaR exist, of which historical simulationprovides one of the simplest and most practical methods. This is because it doesnot require any distributional assumptions of X . For a data set of m observations,the historical VaR˛.X/ estimator is based on the sequence of past P&L realizationsfxtgm

tD1 and can be defined by

bVaR˛.X/ D q˛.fxtgmtD1/; (3.2)

where q˛.�/ denotes the quantile function for level ˛.Despite its popularity in application, VaR has a couple of shortcomings, which

include practical and intuitive issues as well as mathematical defects. First, VaRconsiders only a single quantile of the underlying probability distribution, while allrare events of the downside tail are disregarded, since the amount of the actual lossis not taken into account. Thus, a false sense of security could arise from the usageof VaR and lead to excessive risk-taking (see Einhorn and Brown 2008).

Further, as can easily be shown by a simple counterexample (see Artzner et al(1999) and Acerbi and Tasche (2001) for details), VaR fails to satisfy axiom (ii)of subadditivity; thus, it does not represent a coherent risk measure. However, thiscontradicts the principle of diversification, which is one of the key concepts of modernportfolio theory. It consists of the postulation that the risk of an aggregate positionshould not be higher than the sum of the risks of the single positions. In terms ofrisk management, the possibility of reducing risk (and thus capital requirements) bysplitting the risk up into its integral parts should be excluded by validity of (ii). Asa result of the aforementioned limitations of VaR, several alternative approaches torisk assessment have emerged.

2 In many practical applications, VaR is defined in terms of a loss distribution, ie, higher quantilesimply higher losses. In this paper, losses are given negative signs, eg, a 95% VaR refers to the 5%quantile of X .






3.2 Lower partial moments and expected shortfall

Risk measures that ensure the incorporation of the downside risk distribution providean alternative to the frequently employed VaR approaches. Lower partial moments(LPMs), for example, were (largely) introduced into financial economics by Fishburn(1977) Bawa (1978) and. They define a family of downside risk measures specifiedby order n 2 N0 and a target value � 2 R from which the negative deviations aregauged.

Let X be a continuous and integrable random variable measuring a portfolio’s P&L.Then, the general definition of LPM depending on n and � is given by

LPM.X I �; n/ D EŒmax.� � xI 0/n� DZ �

�1.� � X/n dFX ;

whereby the latter equality holds if FX represents a continuous distribution. LPMsdirectly refer to the deviance from the reference level � , and, unlike VaR, they are notrelated to a predetermined probability level. Depending on the problem under consid-eration, the reference level may be any suitable attractor, such as the expected returnon portfolios, the rate of inflation or simply the point separating profits and losses.However, in terms of financial risk management, the contemplated target frequently(and in the following) concerns VaR, ie, � � VaR˛.X/.

DefineLPM.X I VaR˛.X/; n/ DW LPMn;˛.X/

for simplification, and let 1fX6VaR˛.X/g be the indicator function for X falling shortof VaR˛.X/. In line with the work of Danielsson et al (2006), the relation

LPM1;˛.X/ D ˛ VaR˛.X/ � EŒX1fX6VaR˛.X/g�

holds when the LPM of order n D 1 is represented as a quantile of X .3 However,as emphasized by Barbosa and Ferreira (2004), LPMs do not belong to the class ofcoherent risk measures.

ES represents another downside risk measure, which constitutes a more establishedalternative in risk management than LPM measures. With respect to the target valueVaR˛.X/, its definition is given by

ES˛.X/ D EŒX j X 6 VaR˛.X/�

D 1

˛

Z ˛

0

VaR'.X/ d' D VaR˛.X/ � 1

˛LPM1;˛.X/: (3.3)

3 LPM1;˛.X/ can be interpreted as the downside expected value of X . For n D 0, the downsideprobability directly results, which shows the close relation to VaR. For n D 2 and � � EŒX�,LPM2;˛ equals the semivariance of X .





38 J. Rohde

The first equality marks the character as the conditional expectation of the 100 � ˛%worst losses, while the second targets the property of ES as the mean VaR over alllevels lower than ˛. The last equality refers to its close relation to the class of LPMand VaR, since ES results from the difference of the target value (VaR) and the scaledLPM1;˛.X/.4

Due to the fact that common values for ˛ are 5% or 1%, ES usually assumessubstantially larger values than LPM1. The representation of ES in terms of the actualP&L distribution indicates that the ES-related quantile is given by the differenceof F �1

X .˛/ and ES˛.X/. Hence, ES pays much more attention to the tail of thedistribution than VaR and LPM1. Next to the attribute that it constitutes possibly themost intuitive perception of risk, ES overcomes the theoretical drawbacks of VaR andthe class of LPM as it provides a coherent risk measure (see Artzner et al 1999) andfurther fulfills convexity (see Rockafellar and Uryasev 2000).

ES can easily be estimated by taking advantage of its relationship with the first-orderLPM. For a sample of size m, the estimator of LPMn;˛.X/ is given by

bLPMn;˛.X/ D 1

m

mXtD1

max.VaR˛.X/ � Xt I 0/n: (3.4)

Consequently, it follows that cES˛.X/ D bVaR˛.X/ � ˛�1 bLPM1;˛.X/.

4 THE COMPARISON OF RISK MEASURES USING LOSSFUNCTIONS

In addition to the more familiar strand of literature concerning backtesting methods,loss function approaches constitute a second group of procedures for evaluating riskmeasure estimates (see Caporin 2008), which provide us with the opportunity to com-pare risk measures across financial institutions. While most backtesting approachesin the end only count the number of shortfalls below VaR, the Basel Committee onBanking Supervision (1996) suggests that we attach importance to both the numberand the magnitude of violations within an institution’s risk evaluation.

The objective in using loss functions for risk evaluation is the minimization of costsutilizing a risk measure �. The value assigned by the loss function at a point in timeis commonly termed as the loss function’s score. From a regulatory point of view,lower scores are preferred over higher ones, ie, each shortfall below � increases thecumulative loss.

4 Again, the quantile representation refers to Danielsson et al (2006).






For some reference value � 2 � (fixed in the following), which we do not desire tobe underrun by an estimator Yi;t 2 Yi , define the mapping � W � � Yi 7! RC as theloss function that assigns a nonnegative-valued score at time t for some P&L processfYi;tg of type i . The actual type of the loss function is chosen subject to the matterof concern of the evaluating institution. Although many types can be constructed,Kiliç (2006) advocates the use of magnitude-type loss functions, since one immensesingle hit could already cause appreciable upheavals within the financial institution inquestion. Using the historical simulation approach for the evaluation of VaR models,Hendricks (1996) finds portfolio losses that exceed the corresponding VaR estimateby about 30% on average and extreme losses of a much higher intensity.

In his seminal paper, Lopez (1998) introduces the utilization of loss functionsfor the evaluation of VaR models and proposes different types of loss approaches.Considering the above-mentioned aspects, the quadratic loss approach is the focus ofthis work, whose score � Q.�I Yi;t / assigned at t is defined by

� Q.�; Yi;t / D(

1 C .Yi;t � �/2 if Yi;t < �;

0 if Yi;t > �:(4.1)

In case of an exception, the score comprises a fixed value of 1, although an addi-tional score imposed by � Q.�; Yi;t / increases quadratically with the magnitude ofthe occurred hit. Therefore, the quadratic loss approach might be suitable for financialrisk evaluation.

The cumulative loss of the entire evaluation period directly results from the sumof scores over all observations. Since the distribution of the observations at time t

depends on �.Yi;sI s < t/, assumptions on their dependence are to be made in orderto determine the actual score for the entire period. However, market risk amendmentsincluded in the Basel Accords (see Basel Committee on Banking Supervision 1997),which mandate an evaluation period of only 250 observations, entail that the assump-tion of independent and identically distributed (iid) observations is usually inevitable(see Lopez 1998; Dowd 2007).

On the basis of the work of Lopez (1998), a procedure for the comparison of thebehavior of VaR and ES, as defined by (3.1) and (3.3), in the presence of a break involatility can be developed as follows.

Consider the situation that a break in the volatility of the P&L process occurs atthe very beginning of the evaluation period. Then, it would be desirable for the finan-cial institution to identify the break and adjust the process as quickly as possible inorder to ensure a suitable evaluation of the underlying risk measure. In this regard,the capability of identification connotes that the break is reflected by the score ofthe loss function. To this purpose, the risk measures are evaluated for two differentsettings. On the one hand, the process that was imputed prior to the break is incor-rectly assumed to prevail within the evaluation period as well. On the other hand,





40 J. Rohde

the underlying process for evaluation is correctly adjusted for the change in volatil-ity. In order to construct a measure for the sensitivity of risk measures in responseto a structural break, consider the score assigned for the incorrect process to be thenumerator of a quotient and the score assigned for the properly adjusted process tobe the corresponding denominator.

The risk measures are distinguished by the quantiles of the mistakenly imputedprocess, which each mark the boundary between acceptable and undesirable risk. Forthis purpose, let X and Y be two independent random variables, whose unconditionalvariances are related by �2

Y D ��2X with � 2 RCnf1g.Apart from this, the distributions

of X and Y are identical. Assume that the observations of the P&L process prior to thebreak exhibit the distribution of X , while the observations after the break follow thedistribution of Y . Let F.�/ be the cdf of X . Then, the risk measures in question can berepresented by the quantile function of X and are defined by VaR˛.X/ WD F �1.˛/

and ES˛.X/ WD F �1. Q /, whereby ˛ > Q holds. Note that the quantiles of Y canbe converted into the respective cdf values of X , whose location with respect to thedensity of X is illustrated by Figure 1 on the facing page. The sensitivity of theunderlying risk measures is gauged by the quotient of the loss functions of X andY with reference values VaR˛.X/ and ES˛.X/, respectively. These quotients aredefined for the quadratic loss function by

� WD � Q.X; VaR˛.X//

� Q.Y; VaR˛.X//and Q� WD � Q.X; ES˛.X//

� Q.Y; ES˛.X//: (4.2)

If a risk measure shows an appropriate response to the occurrence of the break,the quotient should be greater than 1 if the volatility declines after the occurrenceof the break, while the quotient should display a value of less than 1 if the volatilityincreases. Therefore, the quotient itself serves as a measure .of the sensitivity of theunderlying risk measure to a structural break. High responsiveness for both directionsof the volatility change can be attested if the quotient value strongly deviates from 1.Obviously, the limit cases show �; Q� ! 0 if � ! 1, and �; Q� ! 1 if � ! 0.When comparing two risk measures, a higher sensitivity is adjudged for the riskmeasure that (depending on the direction of the break) features the quotient of themore extreme value. Thus, the risk measure of the smaller of both the quotient valuesshows a more preferable response to an increase in volatility, and vice versa. Sincethe quotient values only depend on a particular risk measure, this procedure is easilyexpandable to a comparison of more than two quantile risk measures. For a pairwisecomparison of, for example, three risk measures, a transitive inequality relation onthe real numbers applies.

Thus far, the switch in volatility was assumed to be directly caused by a break inthe second moment of the innovation process. In addition, the focus should be put onvolatility breaks induced by a change in the distribution of the innovation process and






FIGURE 1 Shifting of risk measure quantiles in the presence of a volatility break.

ES VaR 0x

(a)

ES VaR 0x

(b)

VaR and ES represented by quantiles F �1.˛/ and F �1. Q/ shift within the distribution of X when a break in volatilityof intensity � occurs.The quantiles of Y are notated in terms of X (red continuous lines). If � < 1, quantiles decreasefor X (as exemplified in (a)), while quantiles increase if � > 1 (as exemplified in (b)). Note that the quantiles of Y

can each be computed by multiplication of the initial quantile and ��0.5.

the comparative investigation of the ability of risk measures to discriminate betweenmodels of the same type, but with innovations drawn from distributions of differentfamilies. This aspect becomes practically relevant if, for example, a financial insti-tution evaluates its utilized risk measure by assuming the wrong distribution. Underconsideration of the most pertinent distributions in financial statistics, the Gaussiandistribution is supposed to be erroneously postulated instead of the Student t .

Maintaining the notation and the general setting of the previous subsection, considertwo random variables X � N.0I 1/, with cdf F.�/, and Y � t .�/. While the Gaussiandistribution prevails during the in-sample period and for the mistakenly perpetuatedmodel for evaluation, the Student t distribution holds true for the alternative modelduring the evaluation period. The sensitivity of the underlying risk measures to achange in distribution is measured by the quotients

�? WD � Q.X; VaR˛.X//

� Q.Y; VaR˛.X//and Q�? WD � Q.X; ES˛.X//

� Q.Y; ES˛.X//: (4.3)

Since VaR.Y / & VaR.X/ D 1 as � ! 1, and, thus, Y implicitly exhibits highervolatility than X for any choice of �, the risk measure that shows the smaller quotientalways outclasses the other. Similar to the illustration in part (b) of Figure 1, therespective quantiles of Y increase for any choice of �.





42 J. Rohde

5 VALUE-AT-RISK VERSUS EXPECTED SHORTFALL:A COMPARATIVE SIMULATION STUDY

5.1 Settings and data-generating process configurations

The aim in this section is to conduct a simulation study regarding the comparison ofVaR and ES (see (3.1) and (3.3)) according to the procedure described in Section 4.The quotient values of the loss functions, as defined by (4.2) and (4.3), respectively,are to be simulated over realistic evaluation horizons and under the assumption ofcertain settings. All examinations are performed in comparison to a reference timeseries model, denoted by fXt;ig (in the following, this is referred to as the “benchmarkmodel”), and premise on historical one-day VaR and ES estimates. The benchmarkmodel is valid during the in-sample period, from which the underlying risk measureis estimated. The estimated measures are appraised by a scenario analysis, whichcontrasts fXt;ig and the alternative model fYt;ig over the course of the evaluationperiod. Certain properties of the alternative model, which depend on the problemto be evaluated, distinguish the benchmark from the alternative model. The scoresimposed on the benchmark model over the evaluation period are each compared withthe scores generated by the alternative model. The index i of the process tags the typeof DGP.

When evaluating the performance for structural breaks in volatility, several scenar-ios for the alternative model are assumed. Different values of � indicate the extent andthe direction of the structural break. Thus, the alternative model features a volatilityamounting to the �-fold of the benchmark model. In order to evaluate the responseof the risk measures to volatility breaks of various intensity, volatility decreases of50%, 35%, 20% and 10%, as well as volatility increases of 10%, 20%, 35%, 50%,75% and 100%, with respect to the reference volatility of fXt;ig, are considered forthe alternative models fYt;ig. As defined in the previous section, a risk measure thatis better able to distinguish between the alternative model and the benchmark modelshould show the higher value of the quotients of loss functions (as defined by (4.2))if the alternative model features the lower volatility, and vice versa.

Within the performance study regarding a change in the distribution, the incrementsof the benchmark model are N.0I 1/, while Student t distributions with different num-bers of degrees of freedom (df) are assumed to generate the innovation process of thealternative model. Since all of them implicitly mark scenarios with higher volatilitiesthan the benchmark model, the risk measure that shows the smaller quotients of theloss functions (as defined by (4.3)) is always expected to be the superior measure.The numbers of df � are chosen in a way that generates increases of the unconditionalvariance of 10%, 20%, 35%, 50% and 75%, namely � 2 f22; 12; 7:71; 6; 4:67g.

The occurrence of scenarios without any violation of the risk measure depends onthe intensity of the volatility change and cannot be precluded, especially for small






out-of-sample lengths. In order to enable a functioning evaluation and comparison forthese special cases, some assumptions must be made. In the event that no exceedancetakes place for both the benchmark and the alternative model, a quotient value of 1 isassigned to the respective replication. A score of 1 plus a value reflecting the greatestfinite percentile of the distribution of the quotients is assigned for scenarios in whichno exceedance occurs for the alternative model. This avoids infinite values for a singlereplication and, thus, for the complete analysis. These substitution rules, however, donot affect the result of the comparison, since scenarios in which the rules effectivelyapply are characterized by very high values of the respective quotient. This leads tothe result that the value of such a quotient surpasses the other quotient’s value anyway.

The simulation studies are conducted by measuring loss using the quadraticapproach (see (4.1)) and 2500 replications each.5 The in-sample length n0 is chosen tocomprise 2000 data points, which approximately represents eight trading years, while,in line with the Basel Accords, the observation period n1 is suggested to be 250 (rep-resenting one year of trading). However, this recommendation is frequently objectedto by both theorists (see, for example, Best 2000; Pesaran and Zaffaroni 2004; Bamset al 2005) and economic authorities (such as the National Bank of Austria (1999)).In order to accommodate for diverse points of view and carve out the behavior of �

and Q�, different horizons for the observation period of n1 2 f100; 175; 250; 500g areimposed in the first instance.

A number of standard stochastic processes are assumed as possible DGPs, wherebyeach benchmark model is defined for t 2 f1; : : : ; n0g during the in-sample period andfor t 2 fn0 C 1; : : : ; n0 C n1g if the underlying process represents the benchmarkor the alternative model within the evaluation period. The innovations of all modelclasses are drawn from either the Gaussian or the Student t distribution. The followingmodel classes are assumed to be DGP i for both fXt;ig and fYt;ig (for simplicity, theDGPs are notated only in terms of the benchmark model).

DGP 1. White noise:

Xtiid� N.0; �2/ .DGP 1a/;

Xtiid� t .�/ .DGP 1b/:

DGP 2. ARMA(1,1): a simple linear model for the mean, given by

Xt D �Xt�1 C '"t�1 C "t ;

5 All the simulations presented in the following were also performed using the binomial loss approach(as presented by Lopez (1998)), which only takes the frequency of extreme losses into account.However, the results for the binomial approach almost equal those using the quadratic loss function;hence they are not reported here. The results are available upon request from the author.





44 J. Rohde

whereby the iid innovations "t are drawn from a Gaussian distribution with parameters.0I �2

" / (DGP 2a) and from a Student t .�/ distribution (DGP 2b), respectively.

DGP 3. GARCH(1,1), as proposed by Bollerslev (1986): for the mean Xt D "t and"t j �.Xt�1; Xt�2; : : : / � .0I �2

t /, the model of conditional volatility is defined by

"t D �t�t ;

�2t D ! C �"2

t�1 C ˇ�2t�1;

whereby the iid innovations �t are drawn from a Gaussian distribution (DGP 3a) andfrom a Student t .�/ distribution (DGP 3b), respectively.

When assuming a break in the unconditional volatility, the first unconditionalmoment needs to stay unaffected. This is guaranteed for all DGPs by an uncondi-tional expectation of 0. In order to avoid a change in the persistence of DGP 2 andDGP 3, the ARMA and GARCH coefficients stay unchanged by the volatility break.The volatility break in the ARMA process is implemented via a change in the errorvariance;6 for the GARCH process, a volatility change can easily be obtained by vary-ing the constant coefficient of the conditional variance equation.7 A standard deviationof 0:02 for the benchmark process is always implied by an appropriate choice of themodel parameters, which provides a realistic volatility level for financial log returns.TheARMA parameters of DGP 2 are assumed to be � D 0:7 and ' D 0:1. Parametersof � D 0:1 and ˇ D 0:7 for DGP 3 generate a heightened persistence of the GARCHmodels, while the constant of the conditional volatility of the benchmark models ischosen to be ! D 0:00008.8

5.2 Results: break in the volatility

The ability of VaR and ES to distinguish models of different volatilities is initiallycompared for a VaR exceedance level of ˛ D 0:05, where VaR is computed by (3.2),while ES is estimated by utilizing the LPM approach given by (3.4). The results ofthe simulation study can be found in Table 1 on page 46, where � and Q� measure theperformance of VaR and ES, respectively. The six parts of Table 1 contain the resultsfor all configurations of intensities of structural breaks (�), lengths of evaluation

6 The unconditional variance of an ARMA(1,1) is VaRŒXt � D ..1C'2 �2�'/=.1��2//�2" . Weak

stationarity is provided for j�j < 1.7 The unconditional variance of a GARCH(1,1) is given by VaRŒXt � D !=.1 � � � ˇ/. Weakstationarity is provided for j� C ˇj < 1.8 For all models featuring the Student t distribution, the volatility break needs to be implemented bya change in the degree of freedom � as the unconditional variance of a random variable T � t .�/

is given by VaR.T / D �=.� � 2/. Since VaR.T / & 1 as � ! 1, the Student t innovations needto be rescaled in order to ensure a standard deviation of 0:02.






periods (n1) and DGPs, as described in Section 5.1. In addition, the levels of thep-values for testing H0 W � > Q� if � < 1 and H0 W � 6 Q� if � > 1 are reported.

A whole string of general conclusions can be drawn. First of all, note that the lossquotient involving ES ( Q�) always holds a significantly larger value than the VaRquotient (�) for intensities � < 1 across all evaluation sample sizes and DGPs. Thisfinding attests ES predominance over VaR for volatility decreases, at least withinthe field of quantile risk measures. For the vast majority of cases, the superiority ofES can also be concluded for scenarios involving volatility increases. The samplelength n1 of the evaluation period, however, plays a more integral role here, since thedominance of ES in terms of distinguishing between the benchmark and the alternativemodel becomes more severe the longer the evaluation period lasts. While ES fails tooutperform VaR for small volatility heightenings, especially in small samples, ESprovides consistently and significantly better results than VaR for nearly all typesof DGP and intensities of volatility breaks in mid-sized and large sample horizons;this includes a period of 250 observations, as is recommended by the Basel Accords.Intuitively, the relative sensitivity in distinguishing processes with different volatilitiesimproves the more we assume extreme intensities of volatility breaks. This result isvalid for small sample sizes as well.

The assumption of the data to be generated as simple white noise comes closest tothe character of independent observations. This interlinks the introduced procedurewith the familiar backtesting approach of testing whether any two elements of ahit sequence are independent of each other (see, for example, Campbell 2005). Inaccordance with that, the Gaussian white noise (see part (a) of Table 1 on the nextpage) presents a very good performance. While ES only fails to outclass VaR in smallsample sizes for a volatility increase of 10%, ES provides preferable results throughoutfor other intensities for both small and large numbers of observations. Sample sizesof 250, however, are sufficient even for small volatility increases. If innovations aregenerated by a t -white noise (DGP 1b), ES cannot generally be identified as superiorfor small samples and volatility increases up to 20%. The relative performances of EScan keep pace with those of DGP 1a when employing mid-sized and large samples.

The findings for DGP 1 are widely confirmed even for the other assumed modelclasses. The ARMA class of models (DGP 2) shows a performance for Gaussianinnovations (DGP 2a) which equals that of white noise, with only a few exceptionsfor low volatility increases and small evaluation horizons. In contrast, if a superiorperformance of ES should be demonstrated in the analysis for Student t innovations(DGP 2b), sample sizes of 500 observations are strongly recommended to obtain asufficient responsiveness, especially if only small or mid-level breaks in volatilityemerge. Altogether, ARMA-t provides the least amount of evidence out of all theexamined DGPs for the superiority of ES over VaR regarding the reaction to volatility





46 J. Rohde

TAB

LE

1R

esul

tsof

the

sim

ulat

edlo

ssqu

otie

nts

ofV

aR(�

)an

dE

S(

Q �)

for

allc

ombi

natio

nsof

inte

nsiti

esof

vola

tility

brea

ks(�

)an

dev

alua

tion

horiz

ons

(n1),

assu

min

g˛

D5%

.[Ta

ble

cont

inue

son

next

five

page

s.]

(a)

DG

P1a

(5%

)

�‚

…„ƒ

0.5

0.65

0.8

0.9

1.1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒn

1�

Q �p

�Q �

p�

Q �p

�Q �

p�

Q �p

100

7.31

067.

7464

***

3.23

955.

4100

***

1.89

254.

0700

***

1.52

642.

8610

***

1.08

481.

1566

175

6.40

738.

7980

***

2.66

815.

8178

***

1.75

622.

5821

***

1.42

901.

7488

***

1.06

781.

0566

250

5.59

9010

.782

***

2.65

266.

4125

***

1.71

002.

3067

***

1.41

331.

6378

***

1.05

991.

0346

*50

04.

5858

12.1

89**

*2.

4471

4.00

20**

*1.

6586

2.01

18**

*1.

3767

1.48

77**

*1.

0564

1.01

22**

*

DG

P1a

(5%

)

�‚

…„ƒ

1.2

1.35

1.5

1.75

2‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�Q �

p�

Q �p

�Q �

p�

Q �p

�Q �

p

100

0.97

920.

9492

*0.

8541

0.74

48**

*0.

7657

0.63

37**

*0.

6691

0.53

08**

*0.

6045

0.47

43**

*17

50.

9615

0.89

79**

*0.

8456

0.76

74**

*0.

7653

0.65

20**

*0.

6749

0.54

14**

*0.

6749

0.54

14**

*25

00.

9552

0.86

37**

*0.

8513

0.74

16**

*0.

7686

0.63

92**

*0.

6794

0.54

67**

*0.

6172

0.48

93**

*50

00.

9622

0.88

38**

*0.

8607

0.75

26**

*0.

7819

0.66

44**

*0.

6691

0.57

70**

*0.

6394

0.51

26**

*






TAB

LE

1C

ontin

ued.

(b)

DG

P1b

(5%

)

�‚

…„ƒ

0.5

0.65

0.8

0.9

1.1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒn

1�

Q �p

�Q �

p�

Q �p

�Q �

p�

Q �p

100

2.15

735.

6718

***

1.63

545.

0585

***

1.40

162.

8868

***

1.30

912.

5355

***

1.21

501.

7207

175

2.06

886.

2213

***

1.53

783.

2636

***

1.34

101.

7324

***

1.26

181.

4820

***

1.18

191.

2343

250

1.93

236.

2341

***

1.52

742.

2890

***

1.32

261.

6020

***

1.24

731.

3741

***

1.15

521.

1669

500

1.83

744.

0202

***

1.46

022.

0075

***

1.32

111.

4939

***

1.23

891.

3121

***

1.15

181.

1397

DG

P1b

(5%

)

�‚

…„ƒ

1.2

1.35

1.5

1.75

2‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�Q �

p�

Q �p

�Q �

p�

Q �p

�Q �

p

100

1.18

231.

5175

1.12

451.

4080

1.12

061.

1190

1.08

821.

0949

1.03

671.

0528

175

1.13

091.

1321

1.11

921.

0595

***

1.09

541.

0288

***

1.04

920.

9793

***

1.02

720.

9791

***

250

1.12

881.

1051

1.10

231.

0603

**1.

0701

1.02

01**

*1.

0419

0.94

69**

*1.

0112

0.91

57**

*50

01.

1288

1.10

331.

0993

1.01

82**

*1.

0756

1.00

39**

*1.

0463

0.92

52**

*0.

9961

0.90

77**

*





48 J. Rohde

TAB

LE

1C

ontin

ued.

(c)

DG

P2a

(5%

)

�‚

…„ƒ

0.5

0.65

0.8

0.9

1.1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒn

1�

Q �p

�Q �

p�

Q �p

�Q �

p�

Q �p

100

12.4

5113

.906

***

7.02

959.

1720

***

2.88

267.

6616

***

1.93

516.

6179

***

1.24

543.

3440

175

9.36

9411

.411

***

3.81

558.

6495

***

1.94

776.

0081

***

1.59

662.

9443

***

1.13

171.

2370

250

7.58

3215

.308

***

3.13

308.

9290

***

1.90

393.

9196

***

1.46

552.

0343

***

1.10

091.

1046

500

5.24

249.

2209

***

2.53

145.

0726

***

1.71

742.

2114

***

1.41

571.

5782

***

1.08

361.

0337

***

DG

P2a

(5%

)

�‚

…„ƒ

1.2

1.35

1.5

1.75

2‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�Q �

p�

Q �p

�Q �

p�

Q �p

�Q �

p

100

1.11

431.

8018

0.93

731.

0426

0.82

320.

7793

*0.

7019

0.60

72**

*0.

6279

0.48

37**

*17

51.

0234

1.00

310.

8781

0.76

78**

*0.

7895

0.67

25**

*0.

6776

0.55

38**

*0.

6240

0.49

78**

*25

00.

9916

0.92

86**

*0.

8600

0.74

81**

*0.

7648

0.67

42**

*0.

6824

0.56

34**

*0.

6249

0.49

95**

*50

00.

9677

0.90

27**

*0.

8567

0.77

70**

*0.

7896

0.66

72**

*0.

6958

0.57

23**

*0.

6433

0.51

33**

*






TAB

LE

1C

ontin

ued.

(d)

DG

P2b

(5%

)

�‚

…„ƒ

0.5

0.65

0.8

0.9

1.1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒn

1�

Q �p

�Q �

p�

Q �p

�Q �

p�

Q �p

100

7.42

408.

2087

***

2.51

949.

2044

***

1.93

568.

4811

***

1.60

499.

1756

***

1.37

695.

3625

175

3.30

919.

6271

***

2.05

108.

0453

***

1.57

425.

0579

***

1.42

604.

7599

***

1.24

751.

9142

250

2.94

8912

.734

***

1.88

737.

4692

***

1.50

392.

5076

***

1.34

731.

8832

***

1.19

931.

3504

500

2.52

6814

.478

***

1.72

332.

9476

***

1.43

271.

7412

***

1.31

971.

4278

***

1.17

481.

1758

DG

P2b

(5%

)

�‚

…„ƒ

1.2

1.35

1.5

1.75

2‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�Q �

p�

Q �p

�Q �

p�

Q �p

�Q �

p

100

1.30

644.

4500

1.26

284.

8108

1.15

745.

7033

1.14

204.

0091

1.06

923.

6839

175

1.17

121.

5724

1.11

061.

3632

1.07

691.

1573

1.02

291.

0319

0.98

860.

9451

**25

01.

1495

1.27

501.

1026

1.09

581.

0487

1.02

511.

0065

0.92

85**

*0.

9724

0.90

06**

*50

01.

1321

1.11

511.

0773

1.01

87**

*1.

0457

0.94

56**

*0.

9957

0.88

77**

*0.

9768

0.84

27**

*





50 J. Rohde

TAB

LE

1C

ontin

ued.

(e)

DG

P3a

(5%

)

�‚

…„ƒ

0.5

0.65

0.8

0.9

1.1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒn

1�

Q �p

�Q �

p�

Q �p

�Q �

p�

Q �p

100

7.06

338.

0556

***

3.09

636.

1853

***

1.86

854.

1122

***

1.50

933.

9228

***

1.11

221.

1561

175

5.76

9510

.883

***

2.58

076.

3667

***

1.76

592.

7959

***

1.42

541.

7218

***

1.08

921.

0797

250

5.07

0110

.535

***

2.54

986.

5637

***

1.68

582.

2967

***

1.40

491.

6238

***

1.07

051.

0712

500

4.32

2311

.358

***

2.35

973.

8389

***

1.62

502.

0191

***

1.37

221.

5048

***

1.06

561.

0269

***

DG

P3a

(5%

)

�‚

…„ƒ

1.2

1.35

1.5

1.75

2‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�Q �

p�

Q �p

�Q �

p�

Q �p

�Q �

p

100

1.02

361.

0700

0.88

350.

7419

***

0.78

640.

6955

***

0.68

650.

5732

***

0.62

600.

5064

***

175

0.97

980.

9147

***

0.86

810.

7657

***

0.78

580.

6840

***

0.67

650.

5707

***

0.62

580.

4945

***

250

0.97

920.

8919

***

0.85

880.

7553

***

0.77

460.

6726

***

0.68

980.

5583

***

0.63

020.

5085

***

500

0.97

060.

9024

***

0.87

210.

7615

***

0.79

460.

6742

***

0.70

530.

5763

***

0.64

590.

5253

***






TAB

LE

1C

ontin

ued.

(f)

DG

P3b

(5%

)

�‚

…„ƒ

0.5

0.65

0.8

0.9

1.1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒn

1�

Q �p

�Q �

p�

Q �p

�Q �

p�

Q �p

100

2.65

114.

2271

***

1.89

813.

4844

***

1.53

592.

8503

***

1.39

162.

2819

***

1.16

711.

2963

175

2.46

914.

8620

***

1.85

852.

4144

***

1.52

391.

8133

***

1.33

421.

4435

***

1.14

071.

1880

250

2.39

973.

3991

***

1.82

212.

1697

***

1.48

021.

5588

***

1.34

971.

4527

***

1.14

501.

1468

500

2.31

802.

8070

***

1.77

172.

0295

***

1.47

431.

5355

***

1.33

811.

3820

**1.

1464

1.13

66

DG

P3b

(5%

)

�‚

…„ƒ

1.2

1.35

1.5

1.75

2‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�Q �

p�

Q �p

�Q �

p�

Q �p

�Q �

p

100

1.08

431.

2071

0.99

810.

9939

0.92

250.

8414

***

0.81

880.

7847

**0.

7621

0.67

75**

*17

51.

0668

1.04

910.

9896

0.90

40**

*0.

9065

0.86

25**

*0.

8262

0.75

05**

*0.

7637

0.66

89**

*25

01.

0723

1.04

47*

0.97

470.

9269

***

0.91

980.

8829

**0.

8370

0.75

69**

*0.

7748

0.69

28**

*50

01.

0802

1.05

46*

0.99

370.

9580

**0.

9273

0.86

37**

*0.

8574

0.78

41**

*0.

7938

0.70

90**

*

Par

ts(a

)th

roug

h(f

)ar

eta

gged

with

thei

rre

spec

tive

DG

Pi.

The

leve

loft

hep

-val

uefo

rte

stin

gH

0W�

>Q �

if�

<1

and

H0

W�6

Q �if

�>

1is

mar

ked

byth

efo

llow

ing

indi

catio

n:*:

p2

Œ0.1

I0.0

5/,*

*:p

2Œ0

.05I

0.01

/,**

*:p

2Œ0

.01I

0�.





52 J. Rohde

breaks, although appropriate reactions to volatility decreases, as well as to graveincreases, are still guaranteed if the evaluation horizon is long enough.

Even though theARMA and GARCH classes of models are characterized by a serialdependence of observations, the GARCH class features more unambiguous resultsand completely dwarfs the outcomes of ARMA in terms of relative sensitivity. Thisresult is not that surprising, however, as GARCH models are able to capture volatilityclustering. Assuming Gaussian innovations (DGP 3a), difficulties in distinguishingbetween the different models with a 10% volatility increase are observed in only a fewcases, for which even 250 observations are not sufficient to present ES as superior toVaR. Moreover, DGP 3b turns out to be a role model among all DGPs using Student t

innovations with regard to the predominance of ES, since even models capturinga volatility increase of only 20% can be discriminated easily from the benchmarkmodel.

Additionally, GARCH with Gaussian innovations as well as Gaussian white noiseyields excellent results, even beyond the comparison of VaR and ES. Almost entirelyin line with intuition, the values of both � and Q� cross the frontier of 1 from belowafter the intensities � switch from a volatility decrease to an increase. Again, minordeviations are observed in small samples, and for low volatility heightenings only.This finding indicates a strong ability to correctly distinguish between two processesof different volatilities, regardless of whether VaR or ES is employed. However, thedistance of the quotient of loss functions from the value of 1 (which can be interpretedas an utter inability to discriminate between the models) is always higher for ES.

All DGP classes show more explicit results if the innovations are drawn from aGaussian distribution.While evaluation horizons of at most 500 observations suffice todemonstrate the superiority of ES for these models, marginal volatility breaks cannotbe identified even in large samples if a Student t distribution is employed instead. Tosum up, our results give rise to the conclusion that ES generally predominates VaRin terms of its reaction to breaks in volatility, since the superiority of ES becomesmore obvious the larger the sample size. Moreover, we record that a small evaluationhorizon is to be avoided for the distinction of processes of similar volatility in orderto ensure the superiority of ES over VaR.

5.3 Results: change in the distribution

In order to compare the relative performance of VaR and ES in distinguishing betweentwo processes employing different innovation distributions, the VaR level is set to be˛ D 0:05, as in Section 5.2. According to (4.3), the study is carried out by simulatingthe quotients �? and Q�?, where the benchmark process utilizes N.0I 1/ innovations,while the alternative model employs Student t distributions with different numbers ofdf. The results can be found in Table 2 on page 54. The three parts of Table 2 contain






the performances for all configurations of t .�/ distributions, lengths of evaluationperiods (n1) and DGPs presented in Section 5.1. The values in parentheses report thelevels of the p-values for the t -statistic, testing H0 W �? 6 Q�?.

The principal finding lies in the fact that a diminishing number of df in the alternativemodel leads to a clearer predominance of ES over VaR in terms of its responsivenessto a change in distribution. The superiority becomes more obvious for a growingnumber of observations in the evaluation sample, while small samples are alreadysufficient for a low number of df, as these models yield the highest volatility andare easiest to distinguish from the benchmark model. Large sample sizes allow us todiscriminate between models of similar volatility in most of the examined cases. Thequotients show values of smaller than 1 with only a few exceptions. Longer samplehorizons and smaller values of � further underpin these results.

White noise and GARCH both exhibit an excellent ability to distinguish betweenthe models and yield highly significant results. The only exception exists for modelsof nearly equal volatilities. Especially for Q�?, values that are significantly lower than1 are observed; this is true even for the t .12/ alternative of relatively low volatility insmall samples.

Inexplicit results can be observed for the ARMA class in small samples. Whenevaluating over horizons of only 100 data points, only major changes in the distribution(ie, for models that feature a low number of df) can reliably be detected. For the t .22/

case, the ARMA model fails to indicate the superiority of ES over VaR regarding theidentification of the two processes, even in samples of 250 observations. Apart fromthis, even in theARMA case, the procedure shows highly significant results in samplesgreater than 100 observations as well as for volatility increases of at least 20%.

An evaluation sample of 250 observations, at least for ARMA models, is not suf-ficient to ensure the satisfactory sensitivity of both risk measures for similar models.However, evidence for the predominance of ES over VaR can be found for all DGPsand numbers of df considered, implying that the risk measure involving the lower oftwo quantiles is better able to discriminate between processes of similar volatility.

5.4 Robustness checks

In order to check the generality of the findings of the previous subsections, the sim-ulations are rerun for some parameter configurations that are different than thoseassumed so far.

Next to the 95% level, VaR estimates of a 99% confidence level are commonlyutilized for measuring the risk of financial institutions. In choosing ˛ D 0:01, nearlythe same tendencies can be observed, even though the conclusions drawn for ˛ D 0:05

only come into force for larger sample sizes (see Appendix A.1 online). However, thiscan simply be explained by a small number of violations for both the benchmark





54 J. Rohde

TAB

LE

2R

esul

tsof

the

sim

ulat

edlo

ssqu

otie

nts

ofV

aR(�

?)

and

ES

(Q �

?)

for

allc

ombi

natio

nsof

alte

rnat

ive

mod

els

(t.�

/)an

dev

alua

tion

horiz

ons

(n1)

assu

min

g˛

D5%

.[Ta

ble

cont

inue

son

next

page

.]

(a)

DG

P1

(5%

)

Alt

ern

ativ

ed

istr

ibu

tio

ns

‚…„

ƒt.

22/

t.12

/t.

7.71

/t.

6/t.

4.67

/‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�?

Q �?

p�

?Q �

?p

�?

Q �?

p�

?Q �

?p

�?

Q �?

p

100

1.04

181.

0474

0.90

880.

7861

***

0.75

640.

6295

***

0.66

480.

5229

***

0.54

510.

4296

***

175

1.01

840.

9586

***

0.87

270.

7482

***

0.71

900.

5988

***

0.62

670.

4968

***

0.53

250.

4194

***

250

0.99

930.

9276

***

0.85

860.

7412

***

0.70

990.

5791

***

0.62

490.

4893

***

0.51

760.

4048

***

500

0.99

340.

9026

***

0.85

380.

7362

***

0.70

970.

5799

***

0.61

740.

4865

***

0.51

020.

3986

***

(b)

DG

P2

(5%

)

Alt

ern

ativ

ed

istr

ibu

tio

ns

‚…„

ƒt.

22/

t.12

/t.

7.71

/t.

6/t.

4.67

/‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�?

Q �?

p�

?Q �

?p

�?

Q �?

p�

?Q �

?p

�?

Q �?

p

100

1.35

885.

6388

1.08

342.

1723

0.85

811.

0793

0.75

160.

7548

0.61

310.

5405

***

175

1.14

871.

3105

0.92

810.

9589

0.77

700.

6922

***

0.64

320.

5660

***

0.54

160.

4390

***

250

1.07

901.

0834

0.89

600.

8629

*0.

7378

0.64

62**

*0.

6325

0.55

90**

*0.

5296

0.43

16**

*50

01.

0201

0.99

27*

0.88

000.

7990

***

0.71

340.

6118

***

0.60

680.

5179

***

0.50

220.

4106

***






TAB

LE

2C

ontin

ued.

(c)

DG

P3

(5%

)

Alt

ern

ativ

ed

istr

ibu

tio

ns

‚…„

ƒt.

22/

t.12

/t.

7.71

/t.

6/t.

4.67

/‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

n1

�?

Q �?

p�

?Q �

?p

�?

Q �?

p�

?Q �

?p

�?

Q �?

p

100

1.09

611.

0827

0.93

970.

8184

***

0.82

020.

6677

***

0.77

200.

5628

***

0.67

370.

4856

***

175

1.03

830.

9917

**0.

9206

0.78

66**

*0.

8090

0.64

44**

*0.

7443

0.56

32**

*0.

6551

0.49

39**

*25

01.

0305

0.93

93**

*0.

9202

0.78

51**

*0.

8094

0.63

15**

*0.

7487

0.56

96**

*0.

6650

0.50

02**

*50

01.

0289

0.92

91**

*0.

9221

0.77

82**

*0.

8260

0.65

94**

*0.

7541

0.59

14**

*0.

6743

0.52

71**

*

The

leve

loft

here

spec

tive

p-v

alue

for

test

ing

H0

W�?

6Q �

?is

mar

ked

byth

efo

llow

ing

indi

catio

n:*:

p2

Œ0.1

I0.0

5/,*

*:p

2Œ0

.05I

0.01

/,**

*:p

2Œ0

.01I

0�.





56 J. Rohde

and the alternative process, which inhibits a quick convergence to the result expectedfrom the prior simulations due to a lack of suitable observations. (This is also a well-known problem in many backtesting frameworks.) While a sample size of at mostn1 D 500 is sufficient to demonstrate the superiority of ES for a 95% VaR confidencelevel, some DGPs demand larger evaluation horizons for ˛ D 0:01.9 This pointentails a minor difficulty in the introduced procedure. If the sensibility of the riskmeasures is evaluated for large volatility decreases (� D 0:5), the issue of too fewexceedances, especially of the alternative process, results in an inability to show thesuperiority of ES across all evaluation horizons. However, this problem seems morerelevant when applying Gaussian innovations; Student t innovations are characterizedby more extreme violations, which therefore lead to more suitable ES quotients.

The same tendency is observed for the results of a change in distribution (seeAppendixA.2 online), although to a less substantial extent. In very small samples, onlychanges to Student t distributions with a low number of df are reliably identified, whilesamples of 250 and 500 observations provide results that are as good as those using˛ D 0:05. Some major differences only arise for the evaluation of ARMA models,for which a sample size of n1 D 500 is recommended to ensure the predominance ofES over VaR.

Another check is provided for alternative choices of DGP volatility levels. While astandard deviation of 0:02 was assured within the preceding parts of the study, sim-ulations for two alternative levels offer a supplementary run for assuming a break involatility. One is a low-level standard deviation of 0:015 (see Appendix B.1 online),which corresponds to 56.25% of the initial variance, and the other is a high-levelstandard deviation of 0:04 (see Appendix B.2 online), which amounts to four timesthe initial variance. The simulation results for both the low- and high-level alternativevolatilities show no systematical different findings, and they comply with the out-comes discussed in Section 5.2 across all DGPs, intensities of breaks and evaluationhorizons. Thus, the relative risk measure performance appears to be independent ofthe actual variance level of the P&L process.

6 EMPIRICAL APPLICATION TO STOCK INDEXES

It remains to reconfirm the conclusions drawn from the simulation studies involvingbreaks in volatility through an application to empirical data sets. For this purpose, sixtime series of stock indexes are analyzed. They are DAX 30, EURO STOXX 50,FTSE 100, the Hang Seng Index, NIKKEI 225 and Standard and Poor’s 500(S&P 500). Each time series contains daily data from January 1990 up to and

9 Additional simulations are available for n1 D 1000 within this branch of the simulation study.






FIGURE 2 Plots of log returns of stock market indexes from January 1990 to March 2015.

1990 1996 2002 2008 2014 1990 1996 2002 2008 2014

1990 1996 2002 2008 2014 1990 1996 2002 2008 2014

1990 1996 2002 2008 2014 1990 1996 2002 2008 2014

Log

retu

rn

–0.1

0

0.1

Log

retu

rn

–0.050

0.05

Log

retu

rn

–0.1

0.1

0

Log

retu

rn

–0.1

0.1

0

0.1

Log

retu

rn–0.1

0.1

0

Log

retu

rn

–0.1

0.1

0

(a) (b)

(c) (d)

(e) (f)

(a) DAX 30. (b) EURO STOXX 50. (c) FTSE 100. (d) Hang Seng. (e) NIKKEI 225. (f) S&P 500.The estimated structuralbreaks in volatility are indicated by vertical red dashed lines.

including March 2015. After performing a log transformation of the return series,6585 observations are left for examination.

As a first step, the series are examined for structural breaks in the volatility byapplying the cumulative sum (CUSUM) of squares test, the version of Deng andPerron (2008). The null of the absence of a structural break is rejected if the teststatistic exceeds the 95% quantile of the limit distribution. By assuming a trimmingparameter of 0.15, breaks are restricted to occur only within the central 70% of theobservations. Thus, as is suggested by Bai and Perron (2006), a number of five breaksshould not be exceeded within each entire series. In order to generate subsamples oflengths that guarantee robust estimations, a minimal distance of 10% of the entiresample between two breaks is respected (see Pesaran and Timmerman 2002). Fourbreaks in volatility are each found for DAX 30, EURO STOXX 50, the Hang SengIndex and S&P 500, while the series of NIKKEI 225 and FTSE 100 contain three andfive breaks, respectively. The plots of the log returns along with the estimated breaksare presented in Figure 2.





58 J. Rohde

TABLE 3 Percentage changes in volatility of stock market indexes after the occurrence ofa structural break by length n1 of the evaluation horizon. [Table continues on next page.]

(a) DAX 30: mean D 0.00029(0.1014),variance D 0.00018

n1Break ‚ …„ ƒ

number 100 175 250 375 500 max EH

I �47.8 �51.9 �49.1 �50.9 �55.4 �44.5II 441.7 300.6 245.5 363.0 324.1 251.1III �41.2 �57.6 �57.0 �63.7 �69.8 �73.5IV 192.4 144.8 498.8 453.3 368.4 246.2

(b) EURO STOXX 50: mean D 0.00018(0.2654),variance D 0.00020


number 100 175 250 375 500 max EH

I �48.1 �50.1 �48.9 �52.8 �54.3 �29.7II 299.4 242.9 326.0 396.3 324.4 269.2III �48.6 �60.7 �62.7 �66.6 �71.6 �74.4IV 271.6 224.2 651.1 577.9 468.5 341.9

(c) FTSE 100: mean D 0.00016(0.2517),variance D 0.00012


number 100 175 250 375 500 max EH

I �51.1 �51.6 �50.9 �52.9 �50.7 �16.7II 423.9 324.3 246.1 201.6 207.1 169.0III �57.5 �64.7 �68.5 �72.0 �75.6 �78.8IV 145.8 70.1 63.8 113.0 201.0 534.4V �61.4 �61.8 �55.0 �56.3 �60.7 �58.0

A number of k breaks splits the series into k C 1 subperiods, so that VaR and ESare estimated from each of the first k subsamples. The risk measures estimated fromsubperiod j 2 f1; : : : ; kg are then evaluated within subperiod j C 1 over horizonsof n1 2 f100; 175; 250; 375; 500; max EHg, where max EH D 1000 is set, unless thelength of the evaluation subsample is smaller than 1000. In this case, max EH denotes






TABLE 3 Continued.

(d) Hang Seng: mean D 0.00033(0.1102),variance D 0.00025


number 100 175 250 375 500 max EH

I 670.7 570.0 457.8 407.5 323.2 198.0II �66.3 �73.6 �71.1 �72.8 �74.5 �74.1III �51.5 �55.1 �59.6 �59.2 33.5 154.8IV �41.3 �47.0 �50.2 �57.9 �60.7 �54.4

(e) NIKKEI 225: mean D �0.00011(0.5605),variance D 0.00022


number 100 175 250 375 500 max EH

I 133.9 73.7 64.6 60.2 36.7 20.0II �40.2 �37.9 �30.7 �35.0 �36.8 �46.7III 213.0 140.1 529.6 423.0 327.4 184.6

(f) S&P 500: mean D 0.00027(0.0544),variance D 0.00013


number 100 175 250 375 500 max EH

I 24.3 20.5 1.7 28.3 76.6 68.7II 248.6 169.3 135.2 101.6 121.4 102.5III �53.3 �60.4 �65.0 �68.2 �71.1 �75.8IV �41.4 �49.1 �30.0 �36.4 �44.3 �35.7

The Roman numbers indicate the chronological occurrence of the break in the respective series, while max EHdenotes the length of the evaluation subsample, not exceeding 1000.

the length of the respective subperiod. The percentage changes in the variances of theevaluation samples, each in relation to the previous subperiod, are stated in Table 3,with the breaks listed in chronological order.

Various intensities of changes in volatility can be observed with each series con-taining at least one huge volatility increase. These increases are caused by the Rus-sian financial crisis and the subsequent downfall of Long-Term Capital Management





60 J. Rohde

(LTCM) in 1998, the global financial crisis in late-summer 2007 or early 2008 or itsaftermath.10 After the period of high volatility beginning in 1998, a large volatilitydecrease is observed around spring 2003, when the markets calmed down after the2001 terrorist attacks, the economic crisis in Argentina and the accusation of account-ing fraud directed against Enron in 2002. Moreover, volatility changes of weakerintensity occur for many of the series. The percentage changes appear largely homo-geneous across the different out-of-sample sizes. A few exceptions can be observed,such as the breaks number I of NIKKEI 225 and S&P 500, for which the intensi-ties strongly depend on the evaluation horizon. For break number III of the HangSeng series, negative changes in volatility are present for short evaluation horizons,while the direction of the break switches for larger out-of-sample lengths. In addition,Table 3 on page 58 provides the means and sample variances of all the data. Note thatvariance levels are in the range of the low-level volatility assumed in the robustnesscheck conducted in Section 5.4.

In the next step, the different models presented in Section 5.2 are estimated foreach of the first k subsamples. From this, the best performing model is selected bythe criterion proposed in Schwarz (1978). The selected models along with the respec-tive in-sample lengths n0 are presented in Table 4 on the facing page. Unsurprisingly,within financial data analysis, GARCH-t models perform best for most of the sub-samples, with a few exceptions, ie, GARCH models with Gaussian innovations. Thelast subperiod of each series only serves for the evaluation of the last estimated riskmeasure, so no model needs to be estimated for subsample k C 1. The describedsubsampling approach is used in many empirical applications where time series areinvestigated for structural breaks, such as Granger and Hyung (2004), within anapplication to S&P 500 absolute stock returns, and Rapach and Strauss (2008), whoexamine the empirical relevance of structural breaks in the unconditional variance ofGARCH(1,1) models.

In accordance with the simulation studies presented above, the sensitivity of VaRand ES in response to a break in volatility is measured by the loss quotients givenby (4.2). The selected model (prevailing during the in-sample period) serves as thebenchmark model, which does not account for the break.11 The data of the respectivesubsample works as the alternative model of the application, which is confronted withsimulations of the correct DGP of the preceding subperiod. The simulations of thebenchmark model are carried out on the basis of 5000 replications, while a VaR levelof 95% is applied for the evaluation. The results of the application can be found inTable 5 on page 62, where the p-values of the respective one-sided t -statistic (eachtesting the predominance of ES under the alternative) are given in parentheses.

10 The estimates for the exact dates of the structural breaks are itemized in Appendix C.1 online.11 The estimated parameters of the selected models are listed in Appendix C.2 online.






TABLE 4 Selected models and in-sample lengths by subsample of the stock index series.

Subperiod‚ …„ ƒ1 2 3 4 5‚ …„ ƒ ‚ …„ ƒ ‚ …„ ƒ ‚ …„ ƒ ‚ …„ ƒ

Series Model n0 Model n0 Model n0 Model n0 Model n0

DAX 30 3b 1266 3b 702 3a 1541 3b 1197 — —EURO S 50 3b 1277 3b 749 3b 1463 3b 1215 — —FTSE 100 3b 1370 3b 854 3a 1242 3a 793 3b 828Hang S 3b 1987 3b 1093 3b 697 3b 1342 — —NIKKEI 225 3b 2035 3b 1332 3b 1330 — — — —S&P 500 3b 1553 3b 676 3b 1227 3b 1640 — —

EURO S 50 denotes EURO STOXX 50, and Hang S denotes Hang Seng.

The majority of our application results confirm the findings of the simulation stud-ies conducted in Section 5, attesting that ES has a superior ability to react to breaks involatility than VaR, even for real data sets. With some exceptions, which are mainlypresent in evaluation samples of n1 D 100, the comparison between both risk mea-sures indicates some highly significant differences for � and Q�, each in favor of thedirection to be expected. DAX break number IV and Hang Seng break number IIImark the only exceptions in samples of n1 D 500 for which ES is not preferredover VaR. Note that the latter break concerns the case in which the direction of thebreak switches shortly before the evaluation period ends. For the maximum evalu-ation sample size, ES is preferred over VaR with only a single exception (NIKKEIbreak number III). The results for breaks of large intensity, for which ES appears to besuperior in most cases, expand the findings of Basu (2006), who works out that ES isaffected by extreme shocks, while VaR remains very sticky. However, the analysis isvery satisfying even for breaks of weaker intensities, eg, for the small and mid-sizedevaluation samples of S&P 500 break number I. Apart from our comparative conclu-sions, we note that, in most of the cases we studied, both risk measures are able toidentify the breaks; this is demonstrated by the respective values of � and Q�. Thisresult confirms Lopez (1998), who attests that VaR has a strong ability to differentiatebetween true and false models when GARCH-t .6/ models are applied, and expandshis findings to ES.

7 CONCLUSION

The accurate evaluation of a risk measure employed by a financial institution is ofhigh importance in view of that institution’s capital requirement. Having the most sen-sitive response to breaks in the volatility of the P&L process is a desirable property





62 J. Rohde

TAB

LE

5R

esul

tsfo

rth

eev

alua

tion

ofV

aRan

dE

Sby

in-s

ampl

ele

ngth

n1

for

the

subs

ampl

esof

diffe

rent

stoc

kin

dexe

s.[T

able

cont

inue

son

next

two

page

s.]

(a)

DA

X30

100

175

250

375

500

max

EH

n1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

sub

per

iod

�Q

��

Q�

�Q

��

Q�

�Q

��

Q�

11.

6019

1.99

862.

6325

4.98

291.

9146

3.83

992.

4249

2.63

572.

5680

4.42

371.

8284

2.47

03(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)2

0.50

260.

3499

0.59

990.

4242

0.67

950.

5786

0.60

050.

4162

0.62

250.

5548

0.72

800.

5399

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

32.

3949

20.1

246

2.73

6910

.251

13.

0094

40.2

556

4.46

693.

2339

6.00

1213

.440

16.

1722

162.

701

(0.0

000)

(0.0

000)

(0.0

000)

(1.0

000)

(0.0

000)

(0.0

000)

40.

7283

0.64

110.

6182

0.54

890.

5011

0.36

530.

5173

0.38

130.

5659

0.57

780.

6487

0.62

87(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.623

1)(0

.015

9)

(b)

EU

RO

STO

XX

50

100

175

250

375

500

max

EH

n1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

sub

per

iod

�Q

��

Q�

�Q

��

Q�

�Q

��

Q�

13.

0726

3.25

023.

5192

5.16

412.

5272

1.21

923.

0922

3.58

322.

8583

3.01

521.

5038

1.70

55(0

.000

0)(0

.000

0)(1

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)2

0.66

500.

6200

0.67

610.

6493

0.57

630.

4165

0.59

240.

6305

0.65

620.

5905

0.74

760.

6977

(0.0

000)

(0.0

017)

(0.0

000)

(0.9

998)

(0.0

000)

(0.0

001)

35.

0131

10.0

123

8.71

1417

.700

16.

2924

40.2

533

6.35

3679

.199

58.

4197

91.9

534

7.05

9916

1.25

8(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)4

0.61

920.

2539

0.56

040.

3333

0.48

560.

3976

0.51

560.

3642

0.57

280.

4333

0.64

920.

5216

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)






TAB

LE

5C

ontin

ued.

(c)

FT

SE

100

100

175

250

375

500

max

EH

n1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

sub

per

iod

�Q

��

Q�

�Q

��

Q�

�Q

��

Q�

13.

2943

2.25

003.

6602

7.70

103.

1864

3.24

013.

3812

3.64

722.

8829

3.92

991.

4155

1.80

53(1

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)2

0.48

190.

3379

0.55

870.

4122

0.62

640.

5357

0.71

660.

5886

0.70

870.

6308

0.86

660.

7464

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

34.

6742

1.34

908.

1626

17.0

987

6.04

7130

.252

06.

1308

40.2

525

8.27

8657

.442

313

.017

613

8.78

4(1

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)4

0.63

440.

1674

0.85

800.

3611

0.93

410.

3321

0.82

730.

4881

0.68

860.

5328

0.60

530.

4317

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

55.

6437

20.2

520

9.33

6420

.248

22.

1007

35.6

527

2.00

2579

.771

63.

2389

160.

663

2.43

246.

4529

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(d)

Han

gS

eng

100

175

250

375

500

max

EH

n1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

sub

per

iod

�Q

��

Q�

�Q

��

Q�

�Q

��

Q�

10.

4820

0.30

270.

5424

0.32

510.

5302

0.35

260.

5407

0.50

790.

5855

0.54

630.

7272

0.67

73(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

1)(0

.000

0)2

5.54

2120

.276

09.

2770

59.2

046

6.60

1270

.227

79.

6138

70.2

937

12.5

195

127.

782

8.64

0793

.318

6(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)3

4.71

5713

.716

98.

1208

20.2

515

5.80

6376

.880

83.

6676

81.6

158

2.32

407.

7489

0.97

480.

6623

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(1.0

000)

(0.0

000)

45.

3499

1.25

014.

6449

5.53

764.

3006

6.84

594.

6061

9.32

033.

9414

16.1

485

1.80

023.

1314

(1.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

(0.0

000)





64 J. Rohde

TAB

LE

5C

ontin

ued.

(e)

NIK

KE

I225

100

175

250

375

500

max

EH

n1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

sub

per

iod

�Q

��

Q�

�Q

��

Q�

�Q

��

Q�

10.

6373

0.45

000.

7904

0.93

060.

5496

1.20

000.

6034

0.70

380.

7029

0.68

200.

8930

0.84

34(0

.000

0)(1

.000

0)(1

.000

0)(0

.499

0)(0

.039

8)(0

.000

0)2

2.66

851.

2501

3.18

224.

9738

1.63

070.

9166

1.77

912.

5476

1.86

162.

1657

2.16

844.

0735

(1.0

000)

(0.0

000)

(1.0

000)

(0.0

000)

(0.0

000)

(0.0

000)

30.

5288

0.43

160.

5924

0.34

090.

4964

0.39

020.

5223

0.42

840.

5796

0.42

510.

8094

0.80

83(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.490

6)

(f)

S&

P50

0

100

175

250

375

500

max

EH

n1

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

‚…„

ƒ‚

…„ƒ

sub

per

iod

�Q

��

Q�

�Q

��

Q�

�Q

��

Q�

10.

8278

0.50

000.

9971

0.91

681.

3222

0.58

341.

0388

0.70

170.

9650

0.65

641.

0317

0.63

10(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)2

0.60

760.

6068

0.66

840.

4993

0.71

950.

8055

0.81

200.

6661

0.77

320.

3690

0.87

700.

8384

(0.4

702)

(0.0

000)

(1.0

000)

(0.0

000)

(0.0

000)

(0.0

004)

35.

1230

30.2

504

9.02

1220

.247

312

.610

559

.985

018

.600

254

.762

924

.767

413

6.18

247

.804

521

5.66

0(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)4

1.64

600.

0144

1.70

1310

.001

61.

4065

1.75

011.

7045

3.66

511.

7577

6.16

381.

5247

2.93

61(1

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)(0

.000

0)

The

VaR

leve

lis

95%

.The

valu

esin

pare

nthe

ses

deno

teth

ep

-val

ues

for

the

resp

ectiv

et-

test

(the

dire

ctio

nsof

the

brea

ksca

nbe

take

nfr

omTa

ble

3on

page

58).






of the underlying measure. This paper proposes a loss function-based frameworkfor the comparative measurement of the responsiveness of quantile downside riskmeasures to breaks in the volatility or distribution. For this purpose, the model com-parison technique introduced by Lopez (1998) is exploited and extended. VaR and ESare contrasted over realistic evaluation horizons within a broad simulation study, inwhich numerous settings involving volatility breaks of different intensities and sev-eral DGPs are checked by employing a magnitude-type loss function. An empiricalstudy additionally demonstrates the applicability of the procedure using data from sixstock indexes.

Both the simulation study and the empirical application unambiguously indicatethe predominance of ES over VaR with regard to its ability to identify breaks involatility. While the superiority of ES is not clearly identifiable in small evaluationsamples for some DGPs, this result becomes more evident when evaluation horizonsare increased, and it applies for the usage of all DGPs, even though the quality ofperformance for GARCH and white noise clearly surpasses that for ARMA. Modelsthat involve Gaussian innovations provide better results in small samples than modelswhose innovations are drawn from a Student t distribution. It is only for breakswhich lead to a slightly increasing volatility that ES is not reliably identifiable as thesuperior risk measure, though this is ensured for volatility decreases of any intensityand increases of about 20–35% over sufficiently large evaluation periods. In contrast toseveral other applications and practical considerations, this paper suggests evaluationhorizons of at least 250 observations for the evaluation of risk measures. However,even the recommendation made by the BCBS does not seem to be sufficient fora limited set of scenarios in order to indicate a better performance of the superiorrisk measure. This outcome is even stronger for lower VaR exceedance levels. Theempirical application for breaks in the volatility using a subsampling approach widelyconfirms these results for the selected and estimated models. In the absence of asuitable test for a structural break in distribution, the corresponding outcomes of theMonte Carlo study remain to be validated for empirical data.

The results of this paper support the findings of prior research regarding the prop-erties of VaR and ES in several stress scenarios. Considering the fact that literature inthis particular field is still rare, this paper contributes to the expansion of knowledgeabout the characteristics of risk measures in the presence of structural breaks.


The author reports no conflicts of interest. The author alone is responsible for thecontent and writing of the paper.





66 J. Rohde

ACKNOWLEDGEMENTS

The author is grateful to Philip Bertram, Claudia Grote, Katharina Pape, PhilippSibbertsen as well as to two unknown referees for helpful comments and advice.

REFERENCES

Abad, P., Muela, S. B., and Martín, C. L. (2015).The role of the loss function in value-at-riskcomparisons. Journal of Risk Model Validation 9(1), 1–19.

Acerbi, C., and Székely, B. (2014). Backtesting expected shortfall. Risk 27(12), 76–81.Acerbi, C., and Tasche, D. (2002). Expected shortfall: a natural coherent alternative to value

at risk. Economic Notes 31(2), 379–388.Amendola, A., and Candila, V. (2014). Evaluation of volatility forecasts in a VaR framework.

In Mathematical and Statistical Methods for Actuarial Sciences and Finance, Perna, C.,and Sibillo, M. (eds), pp. 7–10. Springer.

Amihud, Y., and Mendelson, H. (1991). Volatility, efficiency, and trading: evidence from theJapanese stock market. Journal of Finance 46(5), 1765–1789.

Artzner, P., Delbaen, F., Eber, J. M., and Heath, D. (1999). Coherent measures of risk.Mathematical Finance 9(3), 203–228.

Bai, J., and Perron, P. (2006). Multiple structural change models: a simulation analy-sis. In Econometric Theory and Practice: Frontiers of Analysis and Applied Research,Corbae, D., Durlauf, S. N., and Hansen, B. E. (eds), pp. 212–237. Cambridge UniversityPress.

Bams, D., Lehnert, T., and Wolff, C. (2005). An evaluation framework for alternative VaRmodels. Journal of International Money and Finance 24(6), 944–958.

Bao, Y., Lee, T. H., and Saltoglu, B. (2006). Evaluating predictive performance of value-at-risk models in emerging markets: a reality check. Journal of Forecasting 25(2), 101–128.

Barbosa, A. M., and Ferreira, M. A. (2004). Beyond coherence and extreme losses: rootlower partial moment as a risk measure. SSRN Electronic Journal Paper 08(2004),DOI: 10.2139/ssrn.609221.

Basel Committee on Banking Supervision (1996). Supervisory framework for the use of“backtesting” in conjunction with the internal models approach to market risk capitalrequirements. Technical Note, Bank for International Settlements.

Basel Committee on Banking Supervision (1997). Modification of the Basel Capital Accordof July 1988, as amended in January 1996. Explanatory Note, Bank for InternationalSettlements.

Basel Committee on Banking Supervision (2004). International convergence of capitalmeasurement and capital standards. Report, Bank for International Settlements.

Basel Committee on Banking Supervision (2012). Fundamental review of the trading book.Consultative Document, Bank for International Settlements.

Basel Committee on Banking Supervision (2013). Fundamental review of the tradingbook: a revised market risk framework. Consultative Document, Bank for InternationalSettlements.

Basu, S. (2006). The impact of stress scenarios on VaR and expected shortfall. WorkingPaper 1091462, National Institute of Bank Management.






Bawa, V. S. (1978). Safety-first, stochastic dominance, and optimal portfolio choice. Journalof Financial and Quantitative Analysis 13(2), 255–271.

Berkowitz, J., and O’Brien, J. (2002). How accurate are value-at-risk models at commercialbanks? Journal of Finance 57(3), 1093–1111.

Best, P. (2000). Implementing Value at Risk. Wiley.Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of

Econometrics 31(3), 307–327.Campbell, S. D. (2005). A review of backtesting and backtesting procedures. Research

Paper 21, Divisions of Research and Statistics and Monetary Affairs, Federal ReserveBoard.

Caporin, M. (2008). Evaluating value-at-risk measures in the presence of long memoryconditional volatility. Journal of Risk 10(3), 79–110.

Chen, J. M. (2014). Measuring market risk under the Basel accords: VaR, stressed VaR,and expected shortfall. IEB International Journal of Finance 8, 184–201.

Danielsson, J., Jorgensen, B. N., Sarma, M., and de Vries, C. G. (2006). Compar-ing downside risk measures for heavy tailed distributions. Economics Letters 92(2),202–208.

Degiannakis, S., Floros, C., and Dent, P. (2013). Forecasting value-at-risk and expectedshortfall using fractionally integrated models of conditional volatility: international evi-dence. International Review of Financial Analysis 27, 21–33.

Deng, A., and Perron, P. (2008). The limit distribution of the CUSUM of squares test undergeneral mixing conditions. Econometric Theory 24(3), 809–822.

Diamandis, P. F. (2008). Financial liberalization and changes in the dynamic behaviour ofemerging market volatility: evidence from four Latin American equity markets. Researchin International Business and Finance 22(3), 362–377.

Dowd, K. (2007). Measuring Market Risk. Wiley.Eichengreen, B., Mody, A., Nedeljkovic, M., and Sarno, L. (2012). How the subprime crisis

went global: evidence from bank credit default swap spreads. Journal of InternationalMoney and Finance 31(5), 1299–1318.

Einhorn, D., and Brown, A. (2008). Private profits and socialized risk. Global Associationof Risk Professionals 42, 10–26.

Emmer, S., Kratz, M., and Tasche, D. (2014). What is the best risk measure in practice? Acomparison of standard measures. Preprint, ESSEC Business School, Paris.

Escanciano, J. C., and Du, Z. (2015). Backtesting expected shortfall: accounting for tail risk.Working Paper 001, Center for Applied Economics and Policy Research, EconomicsDepartment, Indiana University Bloomington.

Fishburn, P. C. (1977). Mean-risk analysis with risk associated with below-target returns.American Economic Review 67, 116–126.

Föllmer, H., and Schied, A. (2002). Convex measures of risk and trading constraints.Finance and Stochastics 6(4), 429–447.

Föllmer, H., and Schied, A. (2010). Convex risk measures. In Encyclopedia of QuantitativeFinance, Cont, R. (ed). Wiley.

González-Rivera, G., Lee, T. H., and Mishra, S. (2004). Forecasting volatility: a realitycheck based on option pricing, utility function, value-at-risk, and predictive likelihood.International Journal of Forecasting 20(4), 629–645.





68 J. Rohde

Granger, C.W., and Hyung, N. (2004). Occasional structural breaks and long memory withan application to the S&P 500 absolute stock returns.Journal of Empirical Finance 11(3),399–421.

Halbleib, R., and Pohlmeier, W. (2012). Improving the value at risk forecasts: theory andevidence from the financial crisis. Journal of Economic Dynamics and Control 36(8),1212–1228.

Hansen, B. E. (2001). The new econometrics of structural change: dating breaks in USlabor productivity. Journal of Economic Perspectives 15(4), 117–128.

Hendricks, D. (1996). Evaluation of value-at-risk models using historical data. FederalReserve Bank of New York Economic Policy Review 2(1), 39–67.

Hendricks, D., and Hirtle, B. (1997). Bank capital requirements for market risk: the internalmodels approach. Federal Reserve Bank of New York Economic Policy Review 3(4),1–12.

Kiliç, E. (2006). Violation duration as a better way of VaR model evaluation: evidence fromTurkish market portfolio. MPRA Paper 5610, University Library of Munich.

Lamoureux, C. G., and Lastrapes, W. D. (1990). Persistence in variance, structural change,and the GARCH model. Journal of Business and Economic Statistics 8(2), 225–234.

Lopez, J. A. (1998). Methods for evaluating value-at-risk estimates. Federal Reserve Bankof New York Economic Policy Review 4(3), 119–124.

Lopez, J. A. (2001). Evaluating the predictive accuracy of volatility models. Journal ofForecasting 20(2), 87–109.

Lucas, A. (2001). Evaluating the Basel guidelines for backtesting banks’ internal riskmanagement models. Journal of Money, Credit and Banking 33(3), 826–846.

National Bank of Austria (1999).Evaluation of value-at-risk models. In Guidelines on MarketRisk, Grau, W. (ed), Volume 3. National Bank of Austria.

Nieppola, O. (2009). Backtesting value-at-risk models. MA Thesis, Helsinki School ofEconomics.

Perron, P. (2006). Dealing with structural breaks. In Palgrave Handbook of Econometrics,Mills, T. C., and Patterson, K. (eds), Volume 1, Part IV, pp. 278–352. Palgrave Macmillan.

Pesaran, M., and Timmermann, A. (2002). Market timing and return prediction under modelinstability. Journal of Empirical Finance 9(5), 495–510.

Pesaran, M., and Zaffaroni, P. (2004). Model averaging and value-at-risk based evaluationof large multi asset volatility models for risk management.Research Paper, Money Macroand Finance Research Group Conference.

Rapach, D.E., and Strauss, J.K. (2008).Structural breaks and GARCH models of exchangerate volatility. Journal of Applied Econometrics 23(1), 65–90.

Rockafellar, R.T., and Uryasev, S. (2000). Optimization of conditional value-at-risk. Journalof Risk 2, 21–42.

Røynstrand, T., Nordbø, N. P., and Strat, V. K. (2012). Evaluating power of value-at-riskbacktests. MA Thesis, Norwegian University of Science and Technology.

Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics 6(2), 461–464.

Yamai,Y., andYoshiba, T. (2002). On the validity of value-at-risk: comparative analyses withexpected shortfall. Monetary and Economic Studies 20(1), 57–85.






Research Paper

Liquidity stress testing: a model for a portfolioof credit lines

Marco Geidosch

HypoVereinsbank, UniCredit Bank AG, Kardinal-Faulhaber-Straße 1, 80333 München, Germany;email: [email protected]

(Received October 6, 2015; accepted October 7, 2015)

ABSTRACT

In this paper, we demonstrate how cash outflows due to credit lines can be modeledin a liquidity stress test. Our model is based on bootstrapping from a portfolio timeseries of daily credit line drawdowns. Key features of our model are (i) that it does notrely on any distributional assumptions or any complex parameter estimation, ie, themodel risk is low; (ii) that it is intuitive and straightforward to implement; (iii) thatit is very flexible and allows the portfolio’s free amount to be reduced during thestress test horizon; and (iv) that it calculates an outflow profile with daily granularity.In a detailed simulation study, we demonstrate the reasonable behavior of the modeloutput toward changes in several input parameters.

Keywords: liquidity risk; stress testing; credit lines.

1 INTRODUCTION

Among the many lessons we have learned from the recent financial crises, there aresome that have had a fundamental and permanent impact on risk management in

Print ISSN 1753-9579 j Online ISSN 1753-9587Copyright © 2015 Incisive Risk Information (IP) Limited

69




70 M. Geidosch

financial institutions as well as on regulating authorities.1 One lesson is that, nowa-days, liquidity is perceived as a significant source of risk; another lesson is that stresstesting has become an important tool to overcome traditional risk models’weaknessesin anticipating future adverse events.2 Insofar, this paper touches on a quite importantand timely subject, as we combine liquidity risk and stress testing. In particular, wepresent an approach that practitioners can easily use in their liquidity stress tests tomodel stressed outflows due to credit lines.

What is liquidity stress testing? The general idea is quite simple. Starting with abank’s total amount of liquid assets (ie, cash and highly liquid securities), liquidityoutflows under stress conditions are modeled over a given time horizon (typically oneyear) in order to calculate a liquidity run-off profile, from which the bank’s survivalperiod can be derived. On the liquid asset side, the difficulty lies in finding reasonableliquidity criteria (eg, eligible as central bank collateral or accepted as collateral inrepurchase agreement (repo) markets) and defining appropriate stress haircuts. Withregard to stressed outflows, the challenge lies in implementing stress models for allproduct groups from which relevant liquidity outflows can occur. Prominent examplesinclude clients’withdrawals of sight deposits (ie, bank runs),3 the collapse of the repomarket,4 margin calls for derivatives and drawdowns of credit lines. The importanceof credit lines from a liquidity risk perspective is highlighted by looking at the balancesheets of some large German banks:5 Deutsche Bank, Commerzbank and UniCreditBank AG report €154 billion, €60 billion and €41 billion, respectively, in freecommitted credit line volume in their 2014 annual reports. It is at this point our papercomes in, as we present how cash outflows due to drawdowns of credit lines can bemodeled in times of stress.

Our framework uses a bootstrapping technique to sample from a portfolio’s historyof daily credit line drawdowns in order to produce a large set of potential daily outflow

1 For an overview of the financial crisis, see Hull (2009), Brunnermeier (2009) or Gorton and Metrick(2012a).2 Part of the regulators’ response to the crisis was the introduction of two liquidity risk metrics,namely the net stable funding ratio (NSFR) and the liquidity coverage ratio (LCR) (see BaselCommittee on Banking Supervision 2013a, 2014).3 Liang et al (2014) link bank runs to the institution’s default probability. Iyer and Puri (2012)analyze how different factors contributed to the bank run on an Indian bank. Shin (2008) offers aninteresting take on the run on Northern Rock.4 Acharya et al (2011) explain the freeze of the repo market, while Gorton and Metrick (2012b)investigate how the run on the repo market contributed to the panic of 2007–8.5 To be precise, the committed credit line volume is reported in the notes to the balance sheet.





Liquidity stress testing 71

profiles. Then, the model user has to apply a confidence level, which corresponds totheir risk appetite, to the set of simulated outflow profiles. Our model is straightforwardand intuitive; it does not rely on any distributional assumptions or complex parameterestimation, but instead uses institute-specific drawdown time series. The model istherefore exposed to a very low degree of model risk. At the same time, the model isvery flexible, allowing, for example, the portfolio’s free amount to be reduced withinthe stress test horizon, which may be required due to the cancelation of uncommittedcredit lines or expiring lines. In a detailed simulation study, we demonstrate the modeloutput’s reasonable behavior toward changes in various input factors. In Section 2,we describe our methodology, while the simulation study is presented in Section 3.Critical reflections on the model conclude the paper (Section 4). We complete theintroduction with a literature review.

Unsurprisingly, the literature on stress testing and liquidity risk has mushroomedsince the crisis. With regard to stress testing, macro-level or system-wide stress testinghas to be distinguished from micro-level or institute-specific stress testing. The formeris usually conducted by regulating authorities based on stress assumptions of balancesheet positions. The ultimate goal is to derive capital requirements for the participat-ing banks and to assess whether the financial system as a whole is stable. Morgan et al(2014) and Petrella and Resti (2013) used event study techniques to find that the 2009United States and 2011 European Union stress tests revealed valuable informationto the market. Borio et al (2014) holds a critical view on macro-level stress testing.Gauthier et al (2014) introduced liquidity risk to macro-level stress tests. For generalremarks and the history of stress testing, see Schuermann (2014). Micro-level stresstests, on the other hand, assess the effect of adverse scenarios on a single institu-tion. From a methodological point of view, there is considerably more freedom inspecifying stress scenarios as well as in using and calibrating the respective models.Micro-level stress tests usually focus on a single risk type, such as credit risk (seeRösch and Scheule 2007), market risk or liquidity risk. Basel Committee on BankingSupervision (2013b) gives a very good overview of institute-specific liquidity stresstesting.

Considerable progress has been made in specifying appropriate stress scenarios.Grundke (2011) and Breuer and Csiszar (2013) show how the usual limitation offocusing on handpicked, a priori scenarios can be overcome. Loukoianova et al (2012)present an intuitive measure for dealing with model error and parameter uncertaintyin stress tests. For more information on micro-level stress testing, we recommend thebooks of Rösch and Scheule (2008) and Rebonato (2010). The literature on creditlines focuses on factors that determine future drawdowns, pricing of credit lines orthe role of credit lines in corporate liquidity management (see Jimenez et al 2009;Stanhouse et al 2011; Sufi 2009).





72 M. Geidosch

2 METHODOLOGY

A reasonable approach to calculating outflows due to credit lines in times of stressshould consider four factors. First, the historic drawing behavior of the underlyingportfolio should be a central input parameter, as it captures institute-specific clientbehavior. In general, a credit portfolio is quite stable, which means that the over-all portfolio composition does not change too quickly to permit using the historicportfolio behavior as an indicator of the future. But we must always keep the usualdisclaimer in mind that future risk scenarios may significantly deviate from the past.Second, the current outstanding free amount (ie, the amount the client is contractuallyallowed to draw) is the decisive factor in future outflows. Obviously, the higher theportfolio’s free amount, the higher the risk of liquidity outflows. Third, some kind ofstress considerations should be included in the model. Fourth, an outflow profile overthe stress test horizon should be calculated on a daily basis, which means in particularthat a total outflow volume without time reference is not sufficient.

For the remainder of this paper, we fix the stress test horizon at one year, ie,257 business days. We use negative values to show cash outflows, while cash inflowscorrespond to positive values (ie, the redemption of drawn credit lines). The portfolio’sdrawn amount refers to the amount that has already been drawn by the client in thepast, ie, no future liquidity risk arises from this position. The drawn amount plus thefree amount equals the credit line’s notional amount.

In a nutshell, our model proceeds as follows. Starting from a portfolio’s time seriesof daily notional, drawn and free amounts, we use a bootstrapping technique to gen-erate a one-year outflow profile. More specifically, we draw 257 random sampleswith replacement from our daily time series of drawdowns in order to produce ahypothetical one-year outflow profile. We repeat this step 200 000 times and apply aconfidence level.6 Using bootstrapping and applying a confidence level incorporatespotential future drawing behavior and a stress consideration into the model. In orderto understand how the current free amount is considered, we have to concentrate onthe set of historic drawdowns from which the outflow profiles are bootstrapped. Infact, we do not draw random samples directly from the historic values but from scaledones. To see why, consider an actual aggregated portfolio drawdown of �200 millionon February 17, 2015, when the corresponding portfolio’s free amount was 50 billionon February 16, 2015.7 Picking the �200 million drawdown in the scope of the boot-strapping will only be appropriate for today’s liquidity risk position, when today’sportfolio free amount equals 50 billion. If the current free amount has increased to100 billion, the economic equivalent of the historic �200 million drawdown is a

6 We found that after 200 000 repetitions our outflow profile was sufficiently stable.7 As we are always referring to end-of-day data, the free amount of the previous day is the correctreference value for today’s drawdown.






�400 million drawdown. The converse argument holds when the current free amounthas been reduced to 25 billion. Therefore, we scale each historic drawdown with theratio of how today’s free amount has been changed compared with the correspondinghistorical free amount. Now we present the above approach in mathematical terms.

Let Fi be the portfolio’s free amount on day i ; day 0 denotes today, and stressoutflows start on day 1. Let DiD1;:::;N be the actual historic drawdowns of the portfolioat the end of day i , and let N be the number of observation days. Di is simplycalculated as the difference between the portfolio’s drawn amount on day i and theprevious day i � 1.

The historic drawdowns are scaled as follows:

sDi D Di

F0

Fi�1

: (2.1)

The scaling factor F0=Fi�1 ensures that the historic drawdown Di is scaled accordingto how the portfolio’s historic free amount Fi�1 has been changed in relation totoday’s free amount F0. Note that the free amount at the end of the previous day Fi�1

is relevant for the drawdown of day i .We now draw 257 random samples from the set sDiD1;:::;N , with replacement.

Remember the order of drawing, and the corresponding cumulative sum will give youa hypothetical drawdown path for one year (257 days). Repeat this step 200 000 timesand apply a confidence level q over the 200 000 outflow profiles for each of the 257days. We denote the resulting outflow path with O1

j D1;:::;257.There are two final issues that need to be considered. First, the outflow path

O1j D1;:::;257 assumes the portfolio’s free amount to be constant for the next 257 busi-

ness days, as O1j D1 is drawn under exactly the same circumstances as O1

j D257. Thisis problematic. For example, let us say that today’s free amount is 50 billion, andthat the calculated stress outflow after twenty days is O1

j D20 D 2 billion. Then, theportfolio free amount on day 21 should ceteris paribus be 48 billion, and the potentialfuture outflow risk should be slightly lower than that of the first twenty days. Weincorporate this effect into our model by adjusting the outflow profile O1

j D1;:::;257 inthe following way:

Oa1 D O1

1 .no scaling on day 1/;

Oaj D O1

j �1 C ŒO1j � O1

j �1�AFi for j D 2; : : : ; 257; (2.2)

where the adjustment factor is given by

AFj D 1 � 1

F0

j �1Xj D1

O1j : (2.3)

AFj is the ratio by which today’s free amount has been reduced due to drawdownuntil day j �1. Note that the adjustment factor AFj is only applied to the incremental





74 M. Geidosch

drawdown O1j � O1

j �1.8 Not using this adjustment step would imply that in times ofstress, for each drawn credit line, another committed line is given out to the customerwith an equivalent free amount to keep the total free amount constant; this is a highlyunrealistic scenario.

The second issue we need to consider is credit lines, which are not contractuallycommitted to the client but can be canceled by the bank without the client’s approval.A classic example is a retail overdraft facility on a current account. We abstract fromthe practical problem of determining a suitable point in time when the uncommittedcredit lines can be canceled, and assume the model user has specified day d � 6 257,at which the uncommitted credit line volume UC� will be canceled. For all daysj 6 d �, the final outflow O f

j equals Oaj from (2.2), as nothing happens for these

days. For all days j > d �, the final stress outflow is

O fj D Oa

j �1 C ŒOaj �1 � Oa

j �RFi ; (2.4)

where the reduction factor is given by RFj D 1 � .UC�=Fj �1/. This specifies the ratioby which the free amount of the previous day, Fj �1, is reduced by the cancelation of theuncommitted credit line volume UC�. Here is a summary of the above methodology.

Step 1 Calculate historic drawdowns Di at portfolio level.

Step 2 Scale historic drawdowns based on today’s portfolio free amount.

Step 3 Bootstrap 257 days from the scaled drawdowns of Step 2.

Step 4 Repeat Step 3 a sufficiently large number of times (eg, 200 000).

Step 5 Apply a confidence level on each day over the 200 000 outflow path fromStep 4.

Step 6 Adjust the outflow profile from Step 5 to include the free reduction due tostress outflow.

Step 7 Adjust the outflow profile from Step 6 to include the cancelation ofuncommitted credit lines.

From a practical point of view, choosing the confidence level in Step 5 is a ratherimportant decision and cannot be made in isolation, but it should be in line with theinstitute’s overall risk strategy. Step 7 can easily be adopted for credit lines, whichexpire during the stress test horizon.

8 Of course, the adjustment for reducing the free amount due to stress drawdowns could be incorpo-rated dynamically into the algorithm, ie, by updating the free amount after each bootstrapping step.This greatly increases the computation time for the bootstrapping approach without any significantbenefits.






3 SIMULATION STUDY

In a detailed simulation study, we demonstrate the model’s reasonable behavior byshowing the outflow’s reaction to changes in various input factors. We simulate a widerange of different daily time series that describe the possible drawdown histories of acredit line portfolio. The length of the time series is fixed at three years.9 We assumethat each drawing time series leads to a portfolio free credit line volume of 100 billionas of today, which ensures that the scenarios are comparable in terms of their futureliquidity outflow potential. To streamline the analysis at this point, we abstract from thescaling issue of (2.1) and assume that the simulated drawdown time series are alreadyscaled ones. The confidence level is fixed at 1%. We assume normal distributed dailydrawdowns and show how the outflow profile is affected by the time series’s meanand standard deviation. To investigate deviations from the normal assumption andthe impact of outliers, we add single days of extreme drawdowns to the time series.Further, we demonstrate the model’s flexibility by canceling the uncommitted creditline volume at different points in time during the stress test horizon, which is keptfixed at 257 business days throughout the simulation study. Finally, we show howthe stressed outflow increases if the assumption that the stressed drawdowns willnot reduce the free amount is relaxed (see (2.2)). In fact, relaxing this assumption ispretty unrealistic, as it implies that, for each drawdown, the bank hands out anotherfree credit line in order to keep the free volume constant during the stress test horizon.

Before we go into detail about the simulation study, we want to illustrate the generalbehavior of the bootstrapped outflow profiles via Figure 1 on the next page. Bothoutflow profiles assume a current free credit line portfolio amount of 100 billion,which (for better comparison) is kept constant for the next 257 business days, ie,unadjusted outflows are displayed. In part (a) of Figure 1, the outflow profiles arebootstrapped from a three-year daily drawdown time series, with mean �50 millionand standard deviation 500 million. In part (b), a time series with mean 50 millionand standard deviation 100 million is used. In both cases, 200 000 outflow profilesare bootstrapped, and the corresponding mean, 1% and 99% quantiles are displayed.What can be seen quite easily is that the outflow’s general direction is specified by thetime series daily average drawdowns. In part (a), in which the mean is equal to �50million the outflow profiles are clearly skewed toward outflows, while in part (b), inwhich the mean is equal to 50 million, inflows (ie, the redemption of drawn creditlines) prevail. The outflow profile’s dispersion is controlled by time series volatility,

9 Obviously the model output is invariant toward the length of the time series, given that the timeseries’s characteristics (mean and standard deviation) are constant.





76 M. Geidosch

FIGURE 1 Illustration of bootstrapped outflow profiles.O

utflo

w (

€ b

illio

n)

Out

flow

(€

bill

ion)

18,0

8,0

–2,0

–12,0

–22,0

–32,0

–42,0

18,0

8,0

–2,0

–12,0

–22,0

–32,0

–42,0

(a) (b)

1% quantileMean99% quantile

The mean, 1% and 99% quantiles of 200 000 bootstrapped outflow profiles. In (a) the underlying daily drawdowntime series has mean �50 million and standard deviation 500 million, while in (b) the mean is 50 million and thestandard deviation is 100 million. The gray line represents 257 stress test days.

as the gap between the 1 and 99% quantile is much wider in part (a) of Figure 1 thanin part (b) (standard deviation equalling 500 million versus 100 million). In fact, thisbehavior is pretty much in line with what common sense would have us expect.

In the following analysis, we assume the more realistic scenario: that the stresseddrawdowns reduce the available free credit line volume, ie, we display adjusted val-ues according to (2.2). To investigate the impact of the time series’s average dailydrawdown in more detail, we keep the time series volatility fixed at 500 million, wedo not cancel any uncommitted credit line volumes during the stress horizon, and wevary the average daily drawdown in 10 million steps between �100 million and C100million. As the general structure of the outflow profile does not change from that inFigure 1, it suffices to give only the final one-year outflow values in Table 1 on thefacing page.

As expected, the final outflow increases when the average daily drawdownsincrease. The maximum one-year outflow is �34.4 billion when the mean daily draw-down is �100 million. At the other end of the scale, there is an inflow of 7.7 billionafter one year, which corresponds to a daily drawdown average of C100 million.The impact of 10 million steps on the final outflow increases with the absolute levelof average drawdown. At a �100 million average daily drawdown, the impact of a10 million step on the final outflow is C1.5 billion, while at the other end of the rangethe impact of a 10 million step is almost doubled to C2.7 billion (see the last row ofTable 1 on the facing page).

Now we keep the average daily drawdown fixed at �50 million and vary the timeseries volatility in 50 million steps within the range of 100 million and 1000 million.We can see in Table 2 on page 78 that the final outflow increases when the time series






TAB

LE

1Im

pact

ofth

etim

ese

ries’

sav

erag

edr

awdo

wn

onth

eon

e-ye

arou

tflow

.

Ave

rag

ed

aily

dra

wd

ow

n(m

illio

ns)

�100

�90

�80

�70

�60

�50

�40

�30

�20

�10

0

One

-yea

rou

tflow

(bill

ions

)�3

4.4

�32.

9�3

1.4

�29.

7�2

8.1

�26.

3�2

4.5

�22.

7�2

0.7

�18.

7�1

6.6

Impa

ctof

10m

illio

nst

eps

(bill

ions

)—

1.5

1.5

1.6

1.7

1.7

1.8

1.9

1.9

2.0

2.1

Ave

rag

ed

aily

dra

wd

ow

n(m

illio

ns)

1020

3040

5060

7080

9010

0

One

-yea

rou

tflow

(bill

ions

)�1

4.5

�12.

3�1

0.0

�7.7

�5.3

�2.9

�0.3

2.3

4.9

7.7

Impa

ctof

10m

illio

nst

eps

(bill

ions

)2.

12.

22.

32.

32.

42.

52.

52.

62.

72.

7

The

calc

ulat

edou

tflow

afte

ron

eye

arfo

ran

aver

age

daily

draw

dow

nva

lue

that

varie

sbe

twee

n�1

00m

illio

nan

d10

0m

illio

n.T

hetim

ese

ries’

sst

anda

rdde

viat

ion

isfix

edat

500

mill

ion.

The

time

serie

sle

ngth

isth

ree

year

s.20

000

0ou

tflow

profi

les

are

sam

pled

and

the

1%qu

antil

eis

disp

laye

d.





78 M. Geidosch

TAB

LE

2Im

pact

ofth

etim

ese

ries’

sdr

awdo

wn

vola

tility

onth

eon

e-ye

arou

tflow

.

Dai

lyd

raw

do

wn

vola

tilit

y(m

illio

n)

100

150

200

250

300

350

400

450

500

550

One

-yea

rou

tflow

(bill

ions

)�1

5.2

�16.

7�1

8.1

�19.

6�2

1.0

�22.

4�2

3.7

�25.

1�2

6.3

�27.

6Im

pact

of50

mill

ion

step

s(b

illio

ns)

—�1

.5�1

.5�1

.4�1

.4�1

.4�1

.3�1

.3�1

.3�1

.2

Dai

lyd

raw

do

wn

vola

tilit

y(m

illio

n)

600

650

700

750

800

850

900

950

1.00

0

One

-yea

rou

tflow

(bill

ions

)�2

8.8

�30.

0�3

1.1

�32.

2�3

3.3

�34.

4�3

5.4

�36.

4�3

7.3

Impa

ctof

50m

illio

nst

eps

(bill

ions

)�1

.2�1

.2�1

.1�1

.1�1

.1�1

.0�1

.0�1

.0�0

.9

The

calc

ulat

edou

tflow

afte

rone

year

fora

daily

stan

dard

devi

atio

nth

atva

ries

betw

een

100

mill

ion

and

1000

mill

ion.

The

time

serie

s’s

mea

nda

ilydr

awdo

wn

isfix

edat

�50

mill

ion.

The

time

serie

sle

ngth

isth

ree

year

s.20

000

0ou

tflow

profi

les

are

sam

pled

and

the

1%qu

antil

eis

disp

laye

d.






TAB

LE

3Im

pact

ofou

tlier

son

the

one-

year

outfl

ow.

Nu

mb

ero

fo

utl

iers

01

23

45

67

89

10

One

-yea

rou

tflow

(bill

ions

)�2

6.3

�26.

7�2

7.1

�27.

5�2

8.0

�28.

2�2

8.7

�29.

1�2

9.4

�29.

8�3

0.1

Impa

ctof

outli

er(b

illio

ns)

—�0

.4�0

.4�0

.4�0

.4�0

.3�0

.5�0

.4�0

.3�0

.4�0

.3

Nu

mb

ero

fo

utl

iers

1112

1314

1516

1718

1920

One

-yea

rou

tflow

(bill

ions

)�3

0.4

�30.

8�3

1.1

�31.

5�3

1.8

�32.

1�3

2.4

�32.

8�3

3.0

�33.

4Im

pact

ofou

tlier

(bill

ions

)�0

.3�0

.4�0

.3�0

.4�0

.3�0

.4�0

.3�0

.4�0

.2�0

.3

The

effe

ctof

addi

ngou

tlier

sto

the

port

folio

time

serie

s.T

hetim

ese

ries

aver

age

daily

draw

dow

nan

dst

anda

rdde

viat

ion

isfix

edat

�50

mill

ion

and

500

mill

ion,

resp

ectiv

ely.

Up

totw

enty

days

ofex

trem

ecr

edit

line

draw

dow

ns(�

1.5

billi

on)

are

adde

dto

the

time

serie

s,an

dth

eco

rres

pond

ing

one-

year

outfl

owis

disp

laye

d.T

hetim

ese

ries

leng

this

thre

eye

ars.

200

000

outfl

owpr

ofile

sar

esa

mpl

edan

dth

e1%

quan

tile

isdi

spla

yed.





80 M. Geidosch

TABLE 4 Impact of canceling the uncommitted credit line volume at different points intime on the one-year outflow.

Canceling date (month) 0 1 2 3 4 5 6

One-year outflow (billions) �16.2 �18.6 �19.9 �20.8 �21.7 �22.5 �23.2One-month impact (billions) — �2.5 �1.3 �1.0 �0.9 �0.8 �0.7

Canceling date (month) 7 8 9 10 11 12

One-year outflow (billions) �23.8 �24.4 �25.0 �25.5 �26.0 �26.3One-month impact (billions) �0.6 �0.6 �0.5 �0.5 �0.5 �0.3

How the one-year outflow is increased when a third of the portfolio’s free amount is assumed to be uncommittedand canceled at various points in time. The time series average daily drawdown and standard deviation is fixed at�50 million and 500 million, respectively.The time series length is three years. 200 000 outflow profiles are sampledand the 1% quantile is displayed.

volatility increases. The minimum outflow after one year is �15.2 billion when thevolatility is at a minimum of 100 million. The outflow is increased to �37.3 billionwhen the volatility is increased to 1000 million. The volatility’s effect on the outflowprofile is more stable than that of the mean, but it is slightly reduced when the absolutevolatility level is higher. The sensitivity of 50 million volatility steps varies between�0.9 billion and �1.5 billion (see the last row of Table 2 on page 78). Comparing theeffect of the mean with that of the standard deviation, we can see that the model reactsa lot more sensitively to the time series mean than to the volatility (the impact of aten-million step in average drawdowns varies between 1.5 billion and 2.7 billion).

Next, we choose what we consider to be a reasonable baseline parameterization(�50 million average drawdown and 500 million volatility) and add �1.5 billiondrawdowns to the time series. The probability of occurrence of a �1.5 billion draw-down under the corresponding normal distribution is less than 0.2% and can thereforesafely be considered as a rather extreme event. Table 3 on the preceding page showsthe impact of adding up to twenty outliers to the daily three-year time series. Ourmodel reacts toward outliers in a rather stable way, ie, the occurrence of one outlierincreases the outflow at one year by around 0.4 billion. We believe that the observedimpact of adding an outlier to the time series is quite reasonable, as the model strikesa good balance between sensitivity and robustness toward outliers.

One of the advantages of our model is the ability to capture changes in the freeamount during the stress test horizon. A flexible free volume may be required bythe model user due to credit lines which expire during the stress test horizon ordue to credit lines which can be canceled by the bank (for example retail overdraftfacilities on current accounts are usually uncommitted credit lines). To demonstrate






the behavior of the model, we again use the baseline parameterization from above(�50 million average drawdown and 500 million volatility). We assume that a third oftoday’s free amount of 100 billion is not contractually committed to the customer butcan be canceled by the bank. We investigate the impact of canceling the uncommittedvolume at different points in time, starting at day 0 and proceeding with monthlyfrequency. Obviously, canceling a third of the portfolio free amount has a remarkableimpact on the one-year outflow profile, but the timing is also crucial, as can be seen inTable 4 on the facing page. The earlier the uncommitted credit line volume is canceled,the greater the positive effect on the bank’s liquidity situation, which is pretty muchin line with common sense. Canceling the uncommitted credit line volume at day 0(for whatever reason) leads to a one-year outflow of �16.2 billion. Canceling theuncommitted volume only one month later is quite expensive, as the outflow after oneyear has increased by �2.5 billion to �18.6 billion. As can be seen in Table 4, thepositive impact on the liquidity situation is gradually reduced with every month theuncommitted volume is not cut. It is up to the model user to specify a canceling datethat is appropriate for the portfolio under consideration, but Table 4 clearly revealsthat “time is money”.

We conclude our simulation study by showing the effect of relaxing the outflowadjustment of (2.2). Remember, we introduced this adjustment because it is an unre-alistic assumption that the portfolio free amount stays constant for the stress horizoneven after significant stress outflow has occurred. Figure 2 on the next page uses thebaseline parameterization (mean �50 million, standard deviation 500 million) andshows the impact of the adjustment. At the beginning of the stress horizon, let us saythat until day 50 the difference is immaterial; this is because the simulated drawdownneeds some time to grow until its impact on the portfolio free amount is recognized.Ultimately, the difference becomes quite large, with 4.8 billion after one year (�26.3billion versus �31.1 billion).

4 CRITICAL REFLECTION ON THE MODEL

“Never trust a model unconditionally!” is another lesson we have learned from thefinancial crisis, which is why we want to give some concluding critical reflections onour model. The crucial impact factor, as we have seen in the simulation study, is theactual drawdown time series of the underlying portfolio. Therefore, even by usingbootstrapping and applying an extreme confidence level, the model’s output is alwaysconditioned on drawing patterns that have been observed in the past. The model cannotgenerate any new drawing profiles that are structurally different from the past. It istherefore highly important for the model user to critically scrutinize the time series’sability to reflect all possible future stress scenarios. The length of the time series is an





82 M. Geidosch

FIGURE 2 Effect of adjusting the stressed outflows.

0

–5

–10

–15

–20

–25

–30

–35

Out

flow

(€

bill

ion)

257 days

–26.3 billion

–31.1 billion

Free amount constant at 100 billionFree amount reduced according to drawdowns

Outflow profiles for which the underlying time series’s mean and standard deviations are fixed at �50 million and500 million, respectively. For the blue line, the portfolio’s free amount is kept constant for the stress horizon, whilefor the red line the free amount is reduced according to calculated drawdowns.

important issue here. Are times of severe stress conditions covered by the time series?Is a client’s extreme withdrawing behavior included in the time series? We suggestasking some common sense questions such as “What is the impact of tomorrow’sdrawdown of the top ten committed lines?”, as benchmarks to the modeled outflow.The liquidity outflows of committed lines, as specified by the LCR may also serveas a reasonable yardstick. Ultimately, the model user must feel comfortable with themodel output and is free to add any heuristic conservative elements to the calculatedoutflow profile.


The author reports no conflicts of interest. The author alone is responsible for thecontent and writing of the paper. The views expressed in this paper are the author’spersonal opinions and do not necessarily reflect the views of their affiliated institution.

ACKNOWLEDGEMENTS

I thank my colleague Torsten Wiesmeier for putting a lot of effort into providing anexcellent database, on which this model has been developed.

REFERENCES

Acharya, V., Gale, D., and Yorulmazer, T. (2011). Rollover risk and market freezes. Journalof Finance 66(4), 1177–1209.






Basel Committee on Banking Supervision (2013a). Basel III: the liquidity coverage ratioand liquidity risk monitoring tools. Report, Bank for International Settlements.

Basel Committee on Banking Supervision (2013b). Liquidity stress testing: a surveyof theory, empirics and current industry and supervisory practices. Report, Bank forInternational Settlements.

Basel Committee on Banking Supervision (2014). Basel III: the net stable funding ratio.Report, Bank for International Settlements.

Borio, C., Drehmann, M., and Tsatsaronis, K. (2014). Stress-testing macro stress testing:does it live up to expectations? Journal of Financial Stability 12, 3–15.

Breuer, T., and Csiszar, I. (2013). Systematic stress tests with entropic plausibility con-straints. Journal of Banking and Finance 37(5), 1552–1559.

Brunnermeier, M.K. (2009).Deciphering the liquidity and credit crunch 2007–2008.Journalof Economic Perspectives 23(1), 77–100.

Gauthier, C., Souissi, M., and Liu, X. (2014). Introducing funding liquidity risk in a macrostress-testing framework. International Journal of Central Banking 10(4), 105–141.

Gorton, G., and Metrick, A. (2012a). Getting up to speed on the financial crisis: a one-weekend-reader’s guide. Journal of Economic Literature 50(1), 128–150.

Gorton, G., and Metrick, A. (2012b). Securitized banking and the run on repo. Journal ofFinancial Economics 104(3), 425–451.

Grundke, P. (2011).Reverse stress tests with bottom-up approaches.Journal of Risk ModelValidation 5(1), 71–90.

Hull, J. (2009). The credit crunch of 2007: what went wrong? Why? What lessons can belearned? Journal of Credit Risk 5(2), 3–18.

Iyer, R., and Puri, M. (2012). Understanding bank runs: the importance of depositor–bankrelationships and networks. American Economic Review 102(4), 1414–45.

Jimenez, G., Lopez, J. A., and Saurina, J. (2009). Empirical analysis of corporate creditlines. Review of Financial Studies 22(12), 5068–5098.

Liang, G., Lutkebohmert, E., and Xiao, Y. (2014). A multiperiod bank run model for liquidityrisk. Review of Finance 18(2), 803–842.

Loukoianova, E., Schmieder, C., Kinda, T., Taleb, N. N., and Canetti, E. (2012). A newheuristic measure of fragility and tail risks: application to stress testing. Working Paper,International Monetary Fund.

Morgan, D. P., Peristiani, S., and Savino, V. (2014).The information value of the stress test.Journal of Money, Credit, and Banking 46(7), 1479–1500.

Petrella, G., and Resti, A. (2013). Supervisors as information producers: do stress testsreduce bank opaqueness? Journal of Banking and Finance 37(12), 5406–5420.

Rebonato, R. (2010). Coherent Stress Testing: A Bayesian Approach to the Analysis ofFinancial Stress. Wiley.

Rösch, D., and Scheule, H. (2007). Stress-testing credit risk parameters: an application toretail loan portfolios. Journal of Risk Model Validation 1(1), 55–75.

Rösch, D., and Scheule, H. (2008). Stress Testing for Financial Institutions: Applications,Regulations and Techniques. Risk Books, London.

Schuermann, T. (2014). Stress testing banks. International Journal of Forecasting 30(3),717–728.

Shin, H. S. (2008). Reflections on modern bank runs: a case study of Northern Rock.Working Paper, Princeton University.





84 M. Geidosch

Stanhouse, B., Schwarzkopf, A., and Ingram, M. (2011). A computational approach topricing a bank credit line. Journal of Banking and Finance 35(6), 1341–1351.

Sufi, A. (2009). Bank lines of credit in corporate finance: an empirical analysis. Review ofFinancial Studies 22(3), 1057–1088.





The Journal of Risk Model ValidationINDEX OF PAPERS

To order back issues, please contact:Head Office: Incisive Media (c/o CDS Global), Tower House,Sovereign Park, Market Harborough, Leicestershire, LE16 9EF, UK;Tel: +44 (0)870 787 6822; fax: +44 (0)870 607 0106;e-mail: [email protected] or subscribe online atwww.risk.net/static/risk-journals-subscribe

Volume 9/Number 1

RESEARCH PAPERSThe role of the loss function in value-at-risk comparisons 1–19Pilar Abad, Sonia Benito Muela and Carmen López Martín

Backtesting general spectral risk measures with application toexpected shortfall 21–31Nick Costanzino and Mike Curran

Country risk index and sovereign ratings: do they foresee financial crises? 33–55Nerea San-Martín-Albizuri and Arturo Rodríguez-Castellanos

The effect of introducing economic variables into credit scorecards:an example from invoice discounting 57–78Jie Zhang and Lyn C. Thomas

Volume 9/Number 2

RESEARCH PAPERSBiased benchmarks 1–11Lawrence R. Forest Jr., Gaurav Chawla and Scott D. Aguais

Backtesting Solvency II value-at-risk models using a rolling horizon 13–31Miriam Loois

Stress testing and modeling of rating migration under the Vasicek modelframework: empirical approaches and technical implementation 33–47Bill Huajian Yang and Zunwei Du

Commodity value-at-risk modeling: comparing RiskMetrics,historic simulation and quantile regression 49–78Marie Steen, Sjur Westgaard and Ole Gjølberg




Volume 9/Number 3

RESEARCH PAPERSRisk model validation for BRICS countries: a value-at-risk, expectedshortfall and extreme value theory approach 1–22Jean Paul Chung Wing and Preethee Nunkoo Gonpot

Loss given default modeling: an application to data from a Polish bank 23–40Marek Karwanski, Michał Gostkowski and Piotr Jałowiecki

Stress testing and model validation: application of the Bayesian approachto a credit risk portfolio 41–70Michael Jacobs Jr., Ahmet K. Karagozoglu and Frank J. Sensenbrenner

Comprehensive Capital Analysis and Review stress tests: is regressionthe only tool for loss projection? 71–99Pawel Siarka and Lina Chan

Volume 9/Number 4

RESEARCH PAPERSAERB: developing AIRB PIT–TTC PD models using external ratings 1–18Gaurav Chawla, Lawrence R. Forest Jr and Scott D. Aguais

A mean-reverting scenario design model to create lifetime forecastsand volatility assessments for retail loans 19–30Joseph L. Breeden and Sisi Liang

Downside risk measure performance in the presence of breaks involatility 31–68Johannes Rohde

Liquidity stress testing: a model for a portfolio of credit lines 69–84Marco Geidosch


