Upload
michael-jacobs
View
265
Download
0
Tags:
Embed Size (px)
Citation preview
Validation of Economic Capital Models: State of the
Practice, Supervisory Expectations and Results
from a Bank StudyMichael Jacobs, Ph.D., CFA
Senior Economist / Credit Risk Analysis Division
U.S. Office of the Comptroller of the Currency
Risk Conference on Economic Capital, February 2010The views expressed herein are those of the authos and do not necessarily represent the views of the Office of the Comptroller of the Currency or the Department of the Treasury.
Outline• Introduction, Background and Motivation• Fitness for Use of Economic Capital (EC) Models• Providing Confidence Regarding EC Model Assumptions• Assessing the Value of Validation Methodologies
– Qualitative Approaches• Use Testing• Data Quality Analysis
– Quantitative Approaches• Validating of Inputs and Parameters• Model Replication and Benchmarking• Stress Testing
• Technical Challenges in Testing the Accuracy of EC Models– The Tails of the Loss Distribution– Example: Alternative Models for Risk Aggregation
• Effective Reporting of EC Model Outputs – Avoidance of Misuse and Misunderstanding of the EC Model
Introduction, Background and Motivation
The validation of EC models is at a very preliminary stage• EC models can be complex, having many components, and it may not
be immediately obvious that a such model works satisfactorily• Models may embody assumptions about relationships amongst or
behavior of variables that may not always hold (e.g., stress) • Validation can provide a degree of confidence that assumptions are
appropriate, increasing the confidence of users in the model outputs• Additionally, validation can be also useful in identifying the limitations of
EC models (i.e., where embedded assumptions do not fit reality)• There exists a wide range of validation techniques, each providing
evidence regarding only some of the desirable properties of a model • Such techniques are powerful in some areas (risk sensitivity) but not in
others (accuracy - overall / absolute or In the tail of the distribution)– ..
Introduction, Background and Motivation (continued)
• Used in combination, particularly with good controls and governance, a range of validation techniques can provide more substantial evidence for or against the performance of the model
• There appears to be scope for the industry to improve the validation practices that shed light on the overall calibration of models, particularly in cases where assessment of overall capital is an important application of the model
• .
Fitness for Purpose of Economic Capital Models
• In some cases the term validation is used exclusively to refer to statistical ex post validation (e.g., backtesting of a VaR)
• In other cases it is seen as a broader but still quantitative process that also incorporates evidence from the model development stage
• Herein, “validation” is meant broadly, meaning all the processes that provide evidence-based assessment of a model's fitness for purpose
• This assessment might extend to the management and systems environment within which the model is operated
• It is advisable that validation processes are designed alongside development of the models, rather than chronologically
• This interpretation of validation is consistent with the Basel Committee (2005) in relation to the Basel II Framework– However, that was phrased in terms of the IRB parameters & developed in the context of
assessment of risk estimates for use in minimum capital requirements– Validation of EC differs to an IRB model as the output is a distribution rather than a single
predicted forecast against which actual outcomes may be compared
Fitness for Purpose of EC Models (continued)
• EC are conceptually similar to VaR models, but several differences force validation methods to differ in practice from those used in VaR– Long time horizon, high confidence levels, and the scarcity of data
• Full internal EC models are not used for Pillar 1 minimum capital requirements, so fitness for purpose needs to cover a range of uses– Most of which and perhaps all these uses are internal to the firm in question
• Note that EC models and regulatory capital serve different objectives & may reasonably differ in some details of implementation
• BCBS’s validation principle 1 refers to predictive ability of credit rating systems, an emphasis on performance of model forecasts
• The natural evolution of this principle for EC is that validation is concerned with the predictive properties of those models– I.e., embody forward-looking estimates of risk & their validation involves assessing those
estimates, so this related principle remains appropriate.
• Broadly interpreted validation processes set out herein in different ways all provide insight into the predictive ability of EC model
Providing Confidence Regarding EC Model Assumptions
• Properties of an EC model that can be assessed using powerful tools, and hence that are capable of robust assessment, include:– Integrity of model implementation – Grounded in historical experience– Sensitivity to risk & to external environment – Good marginal properties– Rank ordering & relative quantification.
• Properties for which only weaker processes are available include: – Conceptual soundness – Degree to which forward-looking– Absolute risk quantification
• It is important to stress the power of individual tests & acknowledge that views as to strength and weakness are likely to differ.
Providing Confidence Regarding EC Model Assumptions (cont.)
• There is great difficulty in validating conceptual soundness of an EC model due to many untestable or hard-to-test assumptions made: – Family of statistical distributions for risk factors– Economic processes driving default or loss– Dependency structure among defaults or losses– Likely behavior of management or economic agents how these vary over time.
• Some EC models are of risk aggregation models where estimates for individual categories are combined to generate a single risk figure– There may be no best or unique way to do this aggregation
• Since many of these assumptions may be untestable, it may be impossible to be certain that a model is conceptually sound
• While the conceptual underpinnings may appear coherent and plausible, they may in practice be no more than untested hypotheses
• Opinions may reasonably differ about the strength or weakness of any particular process in respect of any given property
Validation of EC Models: Introduction to Range of Practice
• While we will describe the types of validation processes that are in use or could be used, note that the list is not comprehensive
• We do not suggest that all techniques should or could be used by banks
• We wish to demonstrate that there is a wide range of techniques potentially covered by our broad definition of validation
• This is creating a layered approach, the more (fewer) of which that can be provided, the more (less) comfort that validation is able to provide evidence for or against the performance of the model
• Each validation process provides evidence for (or against) only some of the desirable properties of a model
• The list presented below moves from the more qualitative to the more quantitative validation processes, and the extent of use is briefly discussed
Validation of EC Models: Range of Practice in Qualitative Approaches
• The philosophy of the use test as incorporated into the Basel II Framework: if a bank is actually using its risk measurement systems for internal purposes, then we can place more reliance on it• Applying the use test successfully will entail gaining a careful understanding of
which model properties are being used and which are not• Banks tend to subject their models to some form of qualitative review
process, which could entail: • Review of documentation or development work• Dialogue with model developers or model managers• Review and derivation of any formulae or algorithms• Comparison to other firms or with publicly available information
• Qualitative review is best able to answer questions such as: • Does the model work in theory? • Does the model incorporate the right risk drivers? • Is any theory underpinning it conceptually well-founded? • Is the mathematics of the model right?
Range of Practice in Qualitative Approaches to Validation (continued)
• Extensive systems implementation testing is standard for production-level risk measurement systems prior to implementation• Such as user acceptance testing, checking of model code etc.• These processes could be viewed as part of the overall validation effort, since
they would assist in evaluating whether the model is implemented with integrity• Management oversight is the involvement of senior management in
the validation process• E.g., reviewing output from the model & using the results in business decisions. • Senior management knowing how the model is used & outputs are interpreted, • This should take account of the specific implementation framework adopted and
the assumptions underlying the model and its parameterization.• Data quality checks refer to the processes designed to provide
assurance of the completeness, accuracy and appropriateness of data used to develop, validate and operate the model. • E.g., Review of: data collection and storage, data cleaning of errors, extent of
proxy data, processes that need to be followed to convert raw data into suitable model inputs, and verification of transaction data such as exposure levels
• While not traditionally viewed by the industry as a form of validation, increasingly forming a major part of regulatory thinking
Range of Practice in Qualitative Approaches to Validation (concluded)
• As all models rest on premises of various kinds, varying in the degree to which obvious, we have examination of assumptions
• Certain aspects of an EC model are 'built-in' and cannot be altered without fundamentally changing the model.
• To illustrate, these assumptions could be about: • Fixed model parameters (PDs, correlations or recovery rates)• Distributional assumptions (shape of tail distributions)• Behavior of senior management or of customers
• Some banks go through a deliberate process of detailing the assumptions underpinning their models, including examination of:• Impact on model outputs• Limitations that the assumptions place on model usage and applicability.
Range of Practice in Quantitative Approaches to Validation: Inputs
• A complete validation of an EC model would involve the inputs and parameters, both those that may be statistically estimated • Examples of estimated (assumed) parameters are the main IRB parameters
such PD or LGD) (is PD in a low default portfolio) • Techniques could include assessing parameters against:
• Historical data through replication of estimators• Outcomes over time through backtesting• Market-implied parameters such as implied volatility or implied correlation• Materiality of model output to input and parameters through sensitivity testing
• This testing of input parameters could complement examination of assumptions & sensitivity testing described previously• However, that checking of model inputs is unlikely to be fully satisfactory since,
every model is based on underlying assumptions• The more sophisticated the model, the more susceptible to model
error, so checking input parameters will not help here • However, model accuracy and appropriateness can be assessed, at least to
some degree, using the processes described in this section
Range of Practice in Quantitative Validation: Model Replication
• Model replication is useful technique that attempts to replicate EC model results obtained by the bank
• This could use independently developed algorithms or data sources, but in practice replication might leverage a bank’s existing processes • E.g., run a the bank's algorithms on a different data-set or vice versa, but once
the either of these have been validated and are reliable • This technique and the questions that often arise in implementing
replication can help identify if: • Definitions & algorithms the bank claims to use correctly are understood by staff
develop, maintain, operate and validate the model • The bank is using in practice the modeling framework that it purports to • Computer code is correct, efficient and well-documented • Data used in this validation are those used by the bank to obtain its results
• However, this technique is rarely sufficient to validate models, and in practice there is little evidence of it being used by banks for either validation or to explore the degree of accuracy of their models
• Note that replication simply by re-running a set of algorithms to produce an identical set of results would not be sufficient model validation due diligence
Range of Practice in Quantitative Validation: Benchmarking
• Benchmarking and hypothetical portfolio testing is examination of whether the model produces results comparable to a standard reference model or comparing models on a set of reference portfolio • E.g., benchmarking could be a comparison of an in-house EC model to other
well-known or vendor models (after standardization of parameters) • Benchmarking is among the most commonly used forms of
quantitative validation• From a supervisory perspective, this permits comparison of several
banks' models against the same reference model and identification of models that produce outliers
• Hypothetical portfolio testing means comparison of models against the same reference portfolio, internal or external to the bank • Capable of addressing similar questions to benchmarking by different means. • The technique is a powerful one and can be adapted to analyse many of the
preferred model properties such as rank-ordering and relative risk quantification • A limitation of benchmarking is it only provides relative assessments
and provides little assurance that any model accurately reflects reality or about the absolute levels of capital
Range of Practice in Quantitative Validation: Benchmarking (continued)
• Therefore, as a validation technique, benchmarking is limited to providing comparison of one model against another or one calibration to others, but not testing against ‘reality’.
• It is therefore difficult to assess the degree of comfort provided by such benchmarking methods, as they may only be capable of providing broad comparisons confirming that input parameters or model outputs are broadly comparable
• There may be good reasons why models produce outliers in benchmarking, all of which complicate interpretation of the results:• May be designed to perform well under differing circumstances• May be more or less conservatively parameterized• May differ in their economic foundations
• Comparisons of internal EC are made with varied alternatives:• Industry survey results• Rating agency or industry-wide models• Consultancy marketed models • Academic papers• Regulatory capital models
Range of Practice in Quantitative Validation: Backtesting
• Backtesting addresses the question of how well the model forecasts the distribution of outcomes.
• There are many forms of this that entail some degree of comparison of outcomes to forecasts, and there is a wide literature on the subject.
• However, weak power of backtesting tests for models of risk that quantify high quantiles has been noted • E.g., for portfolio credit models see BCBS (1999)
• Variations to the basic backtesting approach which can increase the power of the tests have been suggested in the literature: • Backtesting more frequently over shorter holding periods (e.g., in market risk
using a one-day standard versus the 10-day regulatory capital standard• Using cross-sectional data on a range of reference portfolios• Using information in forecasts of the full distribution• Testing expected values of distributions as opposed to high quantiles
• Backtesting is useful principally for models whose outputs are a quantifiable metric with which to compare an outcome
• However, some risk measurement systems in use have outputs cannot be interpreted in this way and cannot be backtested
Range of Practice in Quantitative Validation: Backtesting (continued)
• Such risk measurement approaches not amenable to outcomes-based validation might nevertheless be valuable tools for banks • E.g., rating systems, sensitivity tests and aggregated stress losses. • The role of backtesting for such models, if used, would need elaboration
• In practice, backtesting is not yet a key component of banks' validation practices for economic capital purposes
• Stress testing covers both stressing of the model and comparison of model outputs to stress losses
• The outputs of the model might be examined under conditions where model inputs and model assumptions might be stressed
• This process can reveal model limitations or highlight capital constraints that might only become apparent under stress
• Stress testing of regulatory capital models, particularly IRB models, is undertaken by banks but there is more limited evidence of stress testing of economic capital models
Range of Practice in Quantitative Validation: Stress Testing
• Stress testing covers both stressing of the model and comparison of model outputs to stress losses
• The outputs of the model might be examined under conditions where model inputs and model assumptions might be stressed
• This process can reveal model limitations or highlight capital constraints that might only become apparent under stress
• Stress testing of regulatory capital models, particularly IRB models, is undertaken by banks but there is more limited evidence of stress testing of economic capital models
• Through a complementary programme of stress testing, the bank may be able to quantify the likely losses that the firm would confront under a range of stress events
• Comparison of stress losses against model-based capital estimates may provide a modest degree of comfort of the absolute level of capital.
• Banks report some use of this stress testing technique to validate the approximate level of model output.
Range of Practice in Validation: Additional Considerations
• We have not mentioned internal audit, but validation of the overall implementation framework and process should also be subject to independent and periodic review
• This work should be made by parties within the banking organization that are independent of those accountable for the design and implementation of the validation process
• A possibility is that internal audit would be in charge of undertaking this review process, and as such it could be viewed as comprising a part of the management oversight process listed above
• The list of validation tools also does not address the issue of adequate internal standards relevant for validation
• Examples of such standards include:• A description of the issues that need to be addressed as part of
validation • The standards that capital models are expected to achieve • A series of quantitative thresholds that models need to meet• Warning indicators for particular monitoring metrics • Assessment against model development standards
Technical Challenges in Assessing the Adequacy of an EC Model
• A fundamental difficulty faced in EC modeling is the lack of data to estimate high quantiles in the tails of the loss distribution leads to a very high degree of uncertainty
• One approach to dealing with this problem is Bayesian techniques, which combine expert assessments with available data (Kiefer, 2008)• However, this is computationally demanding, and also requires the
elicitation of a prior distribution from an expect, which is very involved• As we don’t see this used in practice currently, we will not further pursue
this approach here • As noted previously, traditional backtesting procedures as applied in
market risk VaR models are impractical in an EC model setting• An alternative approach is to try to assess the accuracy of the EC
output by approximating a statistical measure of uncertainty • But thin data in the tails implies that such confidence bounds are likely
to be very wide • ehf
EC Model Validation Example: Alternative Risk Aggregation Models
• “Models for Risk Aggregation and Sensitivity Analysis: An Application to Bank Economic Capital”, by Hulusi Inanoglu and Michael Jacobs, OCC & Federal Reserve BOG, Working Paper
• Develops proxies for 5 risk types (credit, market, operational, trading and interest income) from historical quarterly call report data for 5 largest banks as of 4Q08
• Compares EC output of different copula models for combining these according to absolute levels and variability• Use a non-parametric bootstrap to assess accuracy of output to
estimation error in inputs (parameters of margins and correlations)• While not a study of EC model validation per se, this illustrates
several quantitative techniques discussed herein:• Benchmarking / hypothetical portfolio analysis of alternative models• Sensitivity analysis to inputs• Testing accuracy of EC (model output) quantile estimates
• Conclusion: a non-parametric model (empirical copula) is more conservative than common copulas (e.g., Gaussian) and also less variable in the resampling experiment (more stable or accurarate)
Validation Example: Alternative Risk Aggregation Models – Risk Proxy Data Summary (Largest Banks As Of 4Q08)
Minimum5th Percentile
25th Percentile Mean Median
75th Percentile
95th Percentile Maxmimum
Standard Deviation Skewness Kurtosis
Gross Chargeoffs2 1.92 3.29 5.24 7.89 6.66 9.88 13.60 31.16 4.51 2.4931 9.0434Non-Interest Expense3 6.20 8.55 12.85 18.47 17.55 24.20 28.64 33.10 6.60 0.1041 -0.9597Net Trading Revenue4 -7.20 -1.79 -0.62 0.01 -0.13 0.25 1.30 16.13 2.14 4.5929 35.3144Liquidity Gap5 -159.68 -112.11 -66.48 -20.10 -20.50 26.51 75.92 375.83 72.07 1.5928 7.8773Interest Rate Gap6
-171.72 -89.86 -57.76 -2.62 7.34 59.23 85.11 153.01 64.80 -0.0682 -0.6063Gross Chargeoffs 0.79 1.07 1.38 1.96 1.74 2.32 3.59 4.53 0.82 1.2623 1.3433Non-Interest Expense 2.15 2.80 3.47 4.29 4.08 5.01 5.96 7.00 1.06 0.3278 -0.3711Net Trading Revenue -1.59 -0.96 -0.32 -0.03 -0.09 0.21 0.87 3.65 0.63 1.9651 11.1842Liquidity Gap -82.87 -26.97 -12.15 2.28 -0.76 13.25 38.46 88.94 24.92 0.4594 2.2183Interest Rate Gap -30.86 -16.66 -8.67 -0.13 0.92 8.46 14.76 25.60 10.92 -0.1630 -0.3518Gross Chargeoffs 0.85 1.21 1.52 2.10 1.91 2.43 3.91 5.81 0.87 1.6790 3.8429Non-Interest Expense 2.81 3.19 3.71 4.14 4.05 4.53 5.20 5.98 0.62 0.3662 0.1090Net Trading Revenue -2.43 -0.36 -0.08 0.00 -0.01 0.05 0.25 4.38 0.61 2.8475 31.4616Liquidity Gap -65.05 -43.17 -20.41 -6.63 -6.85 5.92 35.65 84.95 24.59 0.5063 1.3789Interest Rate Gap -34.99 -17.06 -9.37 -0.42 0.54 9.03 15.35 31.71 12.12 -0.1040 -0.0776Gross Chargeoffs 0.17 0.48 0.91 1.58 1.11 2.19 3.55 4.96 1.01 1.0753 0.2524Non-Interest Expense 0.69 1.67 2.21 3.04 2.58 3.72 5.21 6.43 1.20 0.7202 -0.1864Net Trading Revenue -3.87 -0.39 -0.16 0.03 -0.02 0.06 0.43 9.09 1.04 6.2887 60.3677Liquidity Gap -34.38 -20.14 -10.08 -0.12 -1.93 7.55 20.76 90.05 17.96 2.2629 9.0858Interest Rate Gap -15.34 -11.93 -6.42 -0.58 -0.91 5.83 11.39 14.60 7.32 -0.0244 -0.9156Gross Chargeoffs 0.45 0.60 0.83 1.15 1.00 1.33 2.21 3.50 0.52 1.7393 4.0686Non-Interest Expense 1.94 2.30 2.69 2.99 2.90 3.23 4.03 6.63 0.59 2.6639 14.1990Net Trading Revenue -0.56 -0.12 -0.03 0.00 -0.01 0.02 0.12 0.75 0.13 2.0058 16.6127Liquidity Gap -41.58 -29.25 -12.26 -5.28 -4.91 0.29 19.90 30.69 13.30 0.0705 0.5398Interest Rate Gap -24.80 -12.17 -6.31 -0.24 0.86 6.97 11.49 22.39 8.48 -0.1075 0.0474Gross Chargeoffs 0.10 0.17 0.21 0.33 0.27 0.38 0.68 1.33 0.20 2.4750 7.7287Non-Interest Expense 0.42 0.57 0.75 0.84 0.84 0.93 1.07 1.23 0.15 0.0660 0.4106Net Trading Revenue -0.21 -0.03 -0.01 0.00 0.00 0.00 0.04 0.21 0.04 0.2049 22.0198Liquidity Gap -17.36 -10.17 -5.46 -1.59 -1.85 1.78 8.07 23.62 6.35 0.7620 2.3140Interest Rate Gap -7.58 -3.76 -1.98 -0.02 0.58 2.43 3.52 7.26 2.80 -0.1077 -0.3254
2 - Gross charge-offs (GCO) is our proxy measure credit risk (CR).3 - Other non-interest expense (ONIE) is our proxy measure of operational risk (OR).4 -The deviation to the trailing 4-quarter average in net-trading revenues (NTR-4QD) is our proxy measure of market risk (MR).5 -The deviation to the trailing 4-quarter average of the liquidity gap, defined as total loans minus total deposits, our proxy measure of liquidity risk (LG-4QD).6 -The deviation to the trailing 4-quarter average of the interest rate gap, defined as total interest expense minus total interest income, our proxy measure of interest rate risk (IRG-4QD).
Table 1.3: Summary Statistics on Risk Measures for Top 200 and 5 Largest Banks by Asset Size (Call Report Data 1984-2008)1
Top
200
B
anks
JPM
C
1 - Dollar amounts expressed in billions.
Ban
k of
A
mer
ica
Citi
grou
pW
ells
Far
goP
NC
-2 0 2 4
x 107
0
1
2x 10
-7 Credit Risk: GCO
-2 0 2 4 6
x 107
0
5x 10
-8 Op Risk: ONIE
-1 0 1 2
x 107
0
0.5
1x 10
-6Market Risk:NTR-4QD
-5 0 5
x 108
0
0.5
1x 10
-8 Liquidity Risk: LG-4QD
-4 -2 0 2 4
x 108
0
5x 10
-9Interest Rate Risk:
IRG-4QD
-5 0 5 10
x 108
0
2
4x 10
-9Total Risk: Sum of
Cr.,Ops.,Mkt.,Liqu.&Int.
Historical Loss Distributions (200 Largest Banks 1984-2008)
-2 0 2 4 6
x 106
0
0.5
1x 10
-6 Credit Risk: GCO
-5 0 5 10
x 106
0
5x 10
-7 Op Risk: ONIE
-5 0 5 10
x 106
0
2
4x 10
-6Market Risk:NTR-4QD
-1 0 1 2
x 108
0
2
4x 10
-8 Liquidity Risk: LG-4QD
-4 -2 0 2 4
x 107
0
5x 10
-8Interest Rate Risk:
IRG-4QD
-1 0 1 2
x 108
0
2
4x 10
-8Total Risk: Sum of
Cr.,Ops.,Mkt.,Liqu.&Int.
Historical Loss Distributions (Citibank 1984-2008)
-2 0 2 4 6
x 106
0
0.5
1x 10
-6 Credit Risk: GCO
0 5 10
x 106
0
2
4x 10
-7Op Risk: ONIE
-4 -2 0 2
x 106
0
0.5
1x 10
-6Market Risk:NTR-4QD
-2 -1 0 1 2
x 108
0
1
2x 10
-8 Liquidity Risk: LG-4QD
-5 0 5
x 107
0
2
4x 10
-8Interest Rate Risk:
IRG-4QD
-2 -1 0 1 2
x 108
0
1
2x 10
-8Total Risk: Sum of
Cr.,Ops.,Mkt.,Liqu.&Int.
Historical Loss Distributions (JP Morgan Chase 1984-2008)
0 2 4 6 8
x 106
0
0.5
1x 10
-6 Credit Risk: GCO
2 4 6 8
x 106
0
0.5
1x 10
-6 Op Risk: ONIE
-5 0 5
x 106
0
2
4x 10
-6 Market Risk:NTR-4QD
-1 0 1 2
x 108
0
1
2x 10
-8 Liquidity Risk: LG-4QD
-10 -5 0 5
x 107
0
2
4x 10
-8Interest Rate Risk:
IRG-4QD
-2 -1 0 1 2
x 108
0
1
2x 10
-8Total Risk: Sum of
Cr.,Ops.,Mkt.,Liqu.&Int.
Historical Loss Distributions (Bank of America 1984-2008)
-2 0 2 4
x 106
0
1
2x 10
-6 Credit Risk: GCO
0 2 4 6 8
x 106
0
1
2x 10
-6 Op Risk: ONIE
-1 -0.5 0 0.5 1
x 106
0
0.5
1x 10
-5Market Risk:NTR-4QD
-10 -5 0 5
x 107
0
2
4x 10
-8Liquidity Risk: LG-4QD
-4 -2 0 2 4
x 107
0
2
4x 10
-8Interest Rate Risk:
IRG-4QD
-1 0 1 2
x 108
0
2
4x 10
-8Total Risk: Sum of
Cr.,Ops.,Mkt.,Liqu.&Int.
Historical Loss Distributions (Wells Fargo 1984-2008)
-5 0 5 10 15
x 105
0
5x 10
-6 Credit Risk: GCO
0 5 10 15
x 105
0
2
4x 10
-6 Op Risk: ONIE
-4 -2 0 2 4
x 105
0
5x 10
-5 Market Risk:NTR-4QD
-4 -2 0 2 4
x 107
0
0.5
1x 10
-7 Liquidity Risk: LG-4QD
-2 -1 0 1 2
x 107
0
1
2x 10
-7Interest Rate Risk:
IRG-4QD
-1 0 1 2
x 108
0
2
4x 10
-8Total Risk: Sum of
Cr.,Ops.,Mkt.,Liqu.&Int.
Historical Loss Distributions (PNC 1984-2008)
Validation Example: Alternative Risk Aggregation Models – Distributions of Risk
Proxies (Largest Banks As Of 4Q08)
Validation Example: Alternative Risk Aggregation Models – Correlations of Risk
Proxies (Largest Banks As Of 4Q08) )
-2 0 2
x 108
-5 0 5
x 108
-2 0 2
x 107
0 2 4
x 107
0 2 4
x 107
-2
0
2
x 108
-5
0
5
x 108
-2
0
2
x 107
0
2
4
x 107
Pairwise Scattergraph & Pearson Correlations of 5 Risk TypesTop 200 Banks (Call Report Data 1984-2008)
0
2
4
x 107
Credit
Liqu.
Operat.
Market
Int.Rt.
corr(cr,ops)= 0.6517
corr(mkt,liqu)= 0.1127
corr(int,liqu)= 0.1897
corr(cr,mkt)= 0.2241
corr(ops,liqu)= 0.1533
corr(mkt,int)= 0.2478
corr(cr,liqu)= 0.5343
corr(ops,int)= -0.1174
corr(ops,mkt)= 0.1989
corr(cr,int)= -0.1328
-5 0 5
x 107
-1 0 1
x 108
-5 0 5
x 106
0 5 10
x 106
0 2 4
x 106
-5
0
5
x 107
Pairwise Scattergraph & Pearson Correlations of 5 Risk TypesJP Morgan Chase (Call Report Data 1984-2008)
-1
0
1
x 108
-5
0
5
x 106
0
2
4
x 106
0
5
10
x 106
Credit
Operat.
Liqu.
Int.Rt.
Market
corr(cr,ops)= -0.0577
corr(ops,mkt)= 0.1092
corr(mkt,liqu)= 0.0156
corr(int,liqu)= 0.1996
corr(cr,mkt)= 0.1973
corr(ops,liqu)= 0.0737
corr(mkt,int)=-0.2792
corr(ops,int)= -0.1425
corr(cr,liqu)= 0.1907
corr(cr,int)= -0.0782
-5 0 5
x 107
-1 0 1
x 108
-5 0 5
x 106
2 4 6
x 106
0 5 10
x 106
-5
0
5
x 107
-1
0
1
x 108
-5
0
5
x 106
2
4
6
x 106
0
5
10
x 106
PairPairwise Scatterplot & Pearson Correlations of 5 Risk Types
Bank of America (Call Report Data 1984-2008)
Credit
Liqu.
Int.Rt.
Ops.
Market
corr(cr,ops)= -0.0434
corr(ops,mkt)= 0.1201
corr(mkt,liqu)= -0.1823
corr(int,liqu)= 0.0917
corr(cr,mkt)= 0.0529
corr(ops,liqu)= -0.0855
corr(mkt,int)=-0.1670
corr(cr,liqu)= 0.4787
corr(ops,int)= -0.2349
corr(cr,int)= -0.1809
-2 0 2
x 107
-1 0 1
x 108
-1 0 1
x 107
0 5 10
x 106
0 2 4
x 106
-2
0
2
x 107
-1
0
1
x 108
-1
0
1
x 107
PairPairwise Scatterplot & Pearson Correlations of 5 Risk Types
Citibank (Call Report Data 1984-2008)
0
2
4
x 106
2
4
6
x 106
Credit
Ops.
Market
Liqu.
Int.Rt.
corr(cr,ops)= 0.7665
corr(ops,mkt)= 0.1346
corr(mkt,liqu)= 0.0629
corr(int,liqu)= 0.1238
corr(cr,mkt)= 0.1640
corr(ops,liqu)= 0.1176
corr(cr,liqu)= 0.3147
corr(cr,int)= -0.0878
corr(ops,int)= -0.0879
corr(mkt,int)= -0.1917
-5 0 5
x 107
-5 0 5
x 107
-1 0 1
x 106
0 5 10
x 106
0 2 4
x 106
-5
0
5
x 107
-5
0
5
x 107
-1
0
1
x 106
0
5
10
x 106
0
2
4
x 106
PairPairwise Scatterplot & Pearson Correlations of 5 Risk Types
Wells Fargo (Call Report Data 1984-2008)
Ops.
Credit
Market
Liqu.
Int.Rt.
corr(cr,ops)= 0.1007
corr(ops,mkt)= -0.0428
corr(mkt,liqu)= -0.0094
corr(int,liqu)= 0.0914
corr(cr,mkt)= 0.1842
corr(ops,liqu)= -0.0485
corr(mkt,int)= -0.1779
corr(cr,liqu)= 0.0230
corr(ops,int)= -0.1588
corr(cr,int)= -0.1431
-1 0 1
x 107
-5 0 5
x 107
-5 0 5
x 105
0 1 2
x 106
0 1 2
x 106
-1
0
1
x 107
-5
0
5
x 107
-5
0
5
x 105
0
1
2
x 106
Pairwise Scatterplot & Pearson Correlations of 5 Risk TypesPNC (Call Report Data 1984-2008)
0
1
2
x 106
Credit
Ops.
Market
Liqu.
Int.Rt.
corr(cr,ops)= 0.2887
corr(ops,mkt)= -0.0931
corr(mkt,liqu)= -0.0321
corr(int,liqu)= 0.1286
corr(cr,mkt)= 0.1113
corr(ops,liqu)= -0.1022
corr(mkt,int)= 0.0338
corr(cr,liqu)= 0.2095
corr(ops,int)= -0.1568
corr(cr,int)= -0.1313
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Frank Copula
Top
200
JPM
C
Bof
A
Citi
Wel
ls
PN
C
0.00E+00
1.00E+08
2.00E+08
3.00E+08
4.00E+08
5.00E+08
6.00E+08
7.00E+08
8.00E+08
9.00E+08
1.00E+09
$ VaR
Figure 6.1: 99.97th Percentile Value-at-Risk for 5 Risk Types: Credit, Operational, Market, Liquidity and Interest
Rate (200 Largest Banks: Call Report Data 1984-2008)
Gaussian Copula
Normal Approx.
Empirical Copula T-Copula
Gumbel Copula
Clayton Copula
Frank Copula
Top 200 7.64E+08 6.88E+08 8.59E+08 8.12E+08 9.30E+08 7.28E+08 7.52E+08
JPMC 2.30E+08 1.87E+08 3.92E+08 2.38E+08 2.47E+08 2.19E+08 2.32E+08
BofA 1.94E+08 1.82E+08 2.05E+08 2.00E+08 2.07E+08 1.82E+08 2.03E+08
Citi 1.62E+08 1.32E+08 2.77E+08 1.72E+08 2.00E+08 1.49E+08 1.60E+08
Wells 1.63E+08 1.04E+08 1.87E+08 1.71E+08 1.99E+08 1.52E+08 1.58E+08
PNC 4.79E+07 4.66E+07 5.78E+07 5.01E+07 5.23E+07 4.66E+07 4.71E+07
Validation Example: Alternative Risk Aggregation Models – Absolute EC
Comparison (Largest Banks As of 4Q08)
2e+08 4e+08 6e+08 8e+08 1e+09
0.0e
+00
5.0e
-10
1.0e
-09
1.5e
-09
Gaussian Copula Simulated vs. Normal Approximation and Empicial Copula Annual Loss Distribution
GausCopVaR-99.7%=7.64e+8K=$764B,NormApproxVaR-99.7%=6.95e+8K=$695B,EmpCopVaR-99.7%=8.59e+8K=$859B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): 200 Large Banks (1984-2008)
Dens
ity
Normal ApproximationGaussian CopulaEmpirical Copula
1e+08 2e+08 3e+08 4e+08
0e+0
01e
-09
2e-0
93e
-09
4e-0
95e
-09
6e-0
97e
-09
Gaussian Copula Simulated vs. Normal Approximation and Empirical Copula Annual Loss Distribution
GausCopVaR-99.7%=2.30e+8K=$230B,NormApproxVaR-99.7%=1.88e+8K=$188B,EmpCopVaR-99.7%=3.92e+8K=$392B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): JP Morgan Chase (1984-2008)
Dens
ity
Normal Approximation
Gaussian Copula
Empirical Copula
0e+00 1e+08 2e+08 3e+08 4e+08
0e+0
01e
-09
2e-0
93e
-09
4e-0
95e
-09
6e-0
97e
-09
Gaussian Copula Simulated vs. Normal Approximation and Empirical Copula Annual Loss Distribution
GausCopVaR-99.7%=1.94e+8K=$94B,NormApproxVaR-99.7%=1.82e+8K=$182B,EmpCopVaR-99.7%=2.05e+8K=$205B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): Bank of America (1984-2008)
Dens
ity
Normal ApproximationGaussian CopulaEmpirical Copula
0.0e+00 5.0e+07 1.0e+08 1.5e+08 2.0e+08 2.5e+08 3.0e+08
0e+0
02e
-09
4e-0
96e
-09
8e-0
9
Gaussian Copula Simulated vs. Normal Approximation and Empirical Copula Annual Loss Distribution
GausCopVaR-99.7%=1.62e+8K=$162B,NormApproxVaR-99.7%=1.32e+8K=$132B,EmpCopVaR-99.7%=2.77e+8K=$277B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): Citibank (1984-2008)
Den
sity
Normal ApproximationGaussian CopulaEmpirical Copula
0.0e+00 5.0e+07 1.0e+08 1.5e+08 2.0e+08 2.5e+08 3.0e+08
0e+0
02e
-09
4e-0
96e
-09
8e-0
9
Gaussian Copula Simulated vs. Normal Approximation and Empirical Copula Annual Loss Distribution
GausCopVaR-99.7%=1.63e+8K=$163B,NormApproxVaR-99.7%=1.04e+8K=$104B,EmpCopVaR-99.7%=1.87e+8K=$187B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): Wells Fargo (1984-2008)
Den
sity
Normal ApproximationGaussian CopulaEmpirical Copula
0e+00 2e+07 4e+07 6e+07 8e+07
0.0e
+00
5.0e
-09
1.0e
-08
1.5e
-08
2.0e
-08
2.5e
-08
Gaussian Copula Simulated vs. Normal Approximation and Empirical Copula Annual Loss Distribution
GausCopVaR-99.7%=4.79e+7K=$47.9B,NormApproxVaR-99.7%=4.65e+7K=$46.5B,EmpCopVaR-99.7%=5.78e+7K=$57.8B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): PNC (1984-2008)
Den
sity
Normal ApproximationGaussian CopulaEmpirical Copula
Validation Example: Alternative Risk Aggregation Models – Absolute EC
Comparison (Largest Banks As of 4Q08)
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Frank CopulaTop200
JPMCBofA
CitiWells
PNC
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
VaR/BVA
Figure 6.2: 99.97th Percentile VaR as a Proportion of BVA for 5 Risk Types: Credit, Operational, Market, Liquidity and
Interest Rate (200 Largest Banks: Call Report Data 1984-2008)
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Frank Copula
Top 200 7.10% 6.40% 7.98% 7.55% 8.64% 6.76% 6.99%
JPMC 12.43% 10.12% 21.18% 12.88% 13.35% 11.85% 12.54%
BofA 11.43% 10.72% 12.05% 11.75% 12.19% 10.72% 11.93%
Citi 12.28% 10.03% 21.03% 13.07% 15.19% 11.29% 12.13%
Wells 13.18% 8.45% 15.11% 13.86% 16.11% 12.28% 12.81%
PNC 16.52% 16.06% 19.94% 17.30% 18.04% 16.06% 16.25%
Validation Example: Alternative Risk Aggregation Models – Relative EC
Comparison (Largest Banks As of 4Q08)
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Frank CopulaTop200
JPMC BofACiti
WellsPNC
0.00%
50.00%
100.00%
150.00%
200.00%
250.00%
300.00%
% DivPer
Figure 6.3: 99.97th Percentile VaR Diversification Benefit for 5 Risk Types: Credit, Operational, Market, Liquidity and Interest Rate (200 Largest Banks: Call Report Data
1984-2008) Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Frank Copula
Top 200 42.05% 39.60% 196.34% 31.47% 14.51% 46.38% 41.69%
JPMC 28.27% 31.17% 127.18% 32.60% 20.18% 35.42% 27.74%
BofA 38.31% 36.32% 252.32% 37.50% 25.22% 45.34% 30.13%
Citi 35.23% 37.91% 182.83% 27.53% 11.22% 46.05% 36.41%
Wells 33.97% 41.03% 189.86% 27.82% 10.35% 44.76% 39.28%
PNC 32.16% 31.12% 242.61% 35.21% 21.09% 36.39% 36.09%
99.97% 99.97%
99.97%
99.97%%
VaR VaRFullyDiversified PerfectCorrelationVaR
VaRFullyDiversified
VaR VaRDiversification Benefit
VaR
Validation Example: Alternative Risk Aggregation Models – % Diversification Comparison (Largest Banks As of 4Q08)
Gaussian Copula
T-Copula
Gumbel Copula
Clayton Copula
Frank Copula
Top200
JPMCBofA
CitiWells
PNC
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
% DivPer
Figure 6.4: Genest et al (2009) Copula GOF Tests for 5 Risk Types: Credit, Operational, Market, Liquidity and Interest
Rate (200 Largest Banks: Call Report Data 1984-2008)Gaussian
Copula T-
CopulaGumbel Copula
Clayton Copula
Frank Copula
Top 200 3.55% 6.54% 0.05% 0.25% 1.15%
JPMC 20.53% 7.24% 0.05% 61.69% 37.23%
BofA 50.10% 6.04% 42.91% 60.79% 39.01%
Citi 32.82% 6.04% 30.12% 12.04% 6.94%
Wells 28.72% 57.19% 30.02% 8.44% 8.14%
PNC 47.70% 35.35% 82.07% 10.94% 6.51%
Validation Example: Alternative Risk Aggregation Models – Goodness of Fit
Comparison (Largest Banks As of 4Q08)
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Top
200
JPM
C
Bof
A
Citi
Wel
ls
PN
C
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
CV
Figure 7.1: Numerical Coefficients of Variation in Bootstrap of Correlations for 99.97th Percentile Value-at-Risk for 5
Risk Types: Credit, Operational, Market, Liquidity and Interest Rate (200 Largest Banks: Call Report Data 1984-
2008) Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Top 200 7.08% 28.20% 8.10% 8.49% 10.56% 6.55%
JPMC 7.78% 30.34% 6.42% 7.51% 7.53% 7.00%
BofA 8.85% 27.86% 13.57% 11.09% 10.80% 6.50%
Citi 7.95% 45.26% 9.08% 12.00% 9.57% 6.52%
Wells 8.10% 26.98% 8.91% 16.42% 9.25% 6.00%
PNC 8.96% 32.85% 12.78% 10.89% 9.98% 5.90%
95% 5%99.97%
95%
99.97% 99.97%
99.97%Q Q
SMPL
BTSTRP BTSTRPVaRVaR VaR
NCVVaR
Validation Example: Alternative Risk Aggregation Models – EC Variability
Comparison (Largest Banks As of 4Q08)
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Top
200
JPM
C
Bof
A
Citi
Wel
ls
PN
C
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
CV
Figure 7.2: Numerical Coefficients of Variation in Bootstrap of Margins for 99.97th Percentile Value-at-Risk for 5 Risk Types: Credit, Operational, Market, Liquidity and Interest
Rate (200 Largest Banks: Call Report Data 1984-2008)
Gaussian Copula
Normal Approx.
Empirical Copula
T-Copula
Gumbel Copula
Clayton Copula
Top 200 35.37% 28.20% 8.10% 43.61% 39.32% 44.29%
JPMC 41.38% 30.34% 6.42% 46.64% 33.51% 49.89%
BofA 48.21% 27.86% 13.57% 56.14% 37.79% 25.16%
Citi 38.41% 45.26% 9.08% 62.23% 52.66% 54.38%
Wells 40.00% 26.98% 8.91% 45.99% 45.74% 62.14%
PNC 44.82% 32.85% 12.78% 47.44% 44.94% 69.55%
Validation Example: Alternative Risk Aggregation Models – EC Variability
Comparison (Largest Banks As of 4Q08)
Summary of Contributions and Major Findings
• We have compared alternative risk aggregation methodologies used in practice: the VCA, well-known GCS & less-well known ECS
• First major exercise involved fitting the models, describing & comparing VaR & PDBs across banks & models, GOF statistics
• The second part involved measuring the statistical variation in VaR & PBD
• Across models & banks ECS & AGCS produce the highest absolute magnitudes of VaR vs. either GCS, STCS or other Archimadean copulas – ECS – a variant of legacy “historical simulation” in market risk practice – in many cases the most conservative (a surprise according to asymptotic theory) – VCA consistently produces the lowest VaR number (disturbing in that several practitioners for the lack of theory or supervisory guidance adopt this
• But PDB tended to be largest for the ECS vs.GCS or VCA, while AGCS produces the lowest, a point of caution if banks choose this route
• Failed to find business mix to exert a directionally consistent impact on risk
Summary of Contributions and Major Findings (continued)
• In application of a blanket GOF tests (Genest et al, 2009) found mixed results: about ½ the cases parametric copula models fail to fit the data, confidence levels tended to be modest, so clearly needs more study
• Bootstrapping experiment revealed the variability of the VaR or PDB itself to be significantly lowest (highest) for the ECS (VCA) relative to other copulas
• Contribution of the sampling error in the parameters of the marginal distributions is an order of magnitude greater than the correlations
• Results constituted a sensitivity analysis that argues for practitioners to err on the side of conservatism in considering a non-parametric copula alternative in order to quantify integrated risk • Standard copula formulations produced a wide divergence in measured VaR, diversification benefits as well as the sampling variation in both of these across different measurement frameworks and types of institutions