Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
© 2013 Experian Ltd. All rights reserved.
No part of this copyrighted work may be reproduced, modified, or distributed in any form or manner without the prior written permission of Experian.
Experian Public.
Prediction errors in credit loss forecasting models based on macroeconomic data Eric McVittie Experian Decision Analytics
Credit Scoring & Credit Control XIII
August 2013
University of Edinburgh Business School
2 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Abstract
Loss forecasting models including aggregated economic variables
may generate substantial and persistent forecast biases when fitted
on limited historical data. Recent evidence from the UK, for example,
suggests a bias towards over-prediction of credit losses from models
estimated on short historical data periods including the 2008-2009
recession, when such models are applied to more recent economic
conditions. This paper considers possible explanations for this
pattern, and the potential for alternative approaches that are less
prone to forecast errors of this type.
3 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Outline
Evidence of recent deterioration in forecast accuracy for portfolio loss / pd models including macroeconomic variables:
► Market, bureau & client data
► Variety of modelling methodologies
Here concentrate on:
► Market card delinquency data
► Unemployment measures
Look at some candidate explanations.
Illustrate some issues around use of time series methods and aggregated economic variables in loss models.
Contents:
► An evaluation framework borrowed from Hendry (1995) Dynamic Econometrics, OUP
► Some evidence – lots of pictures
► Many questions
► Very few firm answers
► Some tentative suggestions
20
06
Q1
20
06
Q3
20
07
Q1
20
07
Q3
20
08
Q1
20
08
Q3
20
09
Q1
20
09
Q3
20
10
Q1
20
10
Q3
20
11
Q1
20
11
Q3
Actual
Predicted
4 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Model Building & Forecasting
Robust forecasts require models that replicate ‘key’ features of the (unknown) DGP
Information is inevitably lost in moving from the DGP via observed data to empirical models
Models provide better representations of the DGP the less information they lose and the more they retain
Model design criteria should select models which minimize information loss
Behaviour Measure-
ment
Data Generation Process (DGP)
Observed
Data
Empirical
Models Predictions
/ Forecasts
Not known or
directly observed
5 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Model Building & Forecasting: Information Losses
Measurement System Economic
activity
①
Data Transformations & Aggregation ②
Observed Data
Parameters of
Interest Data Partition ③
Marginalization ④
Lag Truncation ⑤
Functional Form ⑥
Derived Empirical Models
Forecasts
Forecast
Assumptions
Information
losses
6 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Model Building & Forecasting: Design Criteria
Domain Criteria / Objectives AlternativesMeasurement Data Accuracy Volatile, imprecise or Inaccurate data
Past / in-
sample
Homoskedastic, innovation errors Residual autocorrelation;
heteroskedasticity
Present Weakly exogenous conditional
variables for parameters of interest
Invalid conditioning
Future / Out of
sample
Constant, invariant parameters of
interest
Parameter non-constancy; predictive
failure
Theory consistent / structural
relationships
Implausible coefficients
Rival models Encompasses rival models / Forecasts
dominate rival models
Relative poor fit / significant
additional variables / alternative or
additional variables valuable in
forecasting / relatively poor forecasts
Design criteria require minimum information losses at all stages of reduction
relative to potential rival models.
Generally highly demanding even when data availability is good.
7 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Forecast errors: Hypotheses
(1) The models were never ‘right’
► Invalid / excessive marginalization
Variable Selection:
► Wrong drivers ► Missing variables?
● Economic drivers ● Quality of stock, lender
behaviour ● Interactions
Functional misspecification
(2) The models were ‘right’ but the world has changed
► Model parameters not constant / structural
Fundamental ‘regime shift’ resulting in permanent changes to relationships between economic factors and credit losses
Continuously evolving relationships between economic factors and credit losses, resulting in gradual deterioration of calibration of models estimated on historical data.
Some variants of (2) are observationally equivalent to some variants of (1) – e.g. apparent
regime shifts (in terms of observed variables) may be due to changes in unobserved
(missing) variables.
8 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013
Prediction errors in macroeconomic credit loss forecasting models
Thinking about economic risk: Variable Selection
Search space is large even if restricted to macroeconomic (aggregate) variables:
► Many potentially-relevant variables ► Functional forms? ► Lag structures?
What about disaggregated data (e.g. for geographical areas)?
Finding good models hampered by poor data availability:
► Limited historical time series for consistent credit loss data
► Missing variables related to credit quality, lending practices, etc.
Weak tests for valid reductions.
High risk of building models based on spurious correlations.
Grid Search / Data Mining
Theory, priors & ‘expert’ judgement
Statistical Data Reduction Methods
Macroeconomic factors proxy
more direct influences on
borrowers – proxy quality varies.
Aggregated time series approaches lead to large information losses especially given
short historical time series and force additional losses due to excess marginalization
and lag truncation
Data limitations weaken tests to enforce design criteria even if attempts are made to
apply them rigorously.
9 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
Look at models of UK market delinquency rate measure (derived from BoE data) over period 1992-2011
Focus on unemployment-related variables :
► In naive models including only unemployment measures ► Conditioned on other candidate drivers of card delinquency ► How do models perform in most recent time periods? ► Stability of coefficient estimates over sub-periods of the full sample
Advantages of modelling ‘market’ data:
► Longer (apparently) consistent history ► Better matching of borrower population to macroeconomic aggregates? (although
even here population for macro variables may not match borrower population) ► I can show you the results!
Disadvantages:
► More missing variable issues – not possible to condition on quality of stock or allow for changes in lender behaviour
► Rules out anything other than simple time series modelling approaches
Analysis and fitted models are purely illustrative– useful case study to consider some of the issues involved linking credit outcomes to economic variables as basis for loss forecasting.
10 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
-2
-1
0
1
2
3
1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
Credit Card Delinquency Rate
ILO Unemployment Rate
Claimant Unemployment Rate
Normalized series - levels
Card
Delinquency
Rate
ILO Unemployment Rate
Covariance -0.375
Correlation -0.242
t-Statistic -2.298
Probability 0.024
Claimant Rate
Covariance -0.808
Correlation -0.400
t-Statistic -4.020
Probability 0.000
Covariance Analysis
11 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
-1.5
-1
-0.5
0
0.5
1
1997Q1 2000Q1 2003Q1 2006Q1 2009Q1
5 year
7 year
10 year
15 year
Mo
vin
g C
orr
elat
ion
Co
effi
cien
t
End of rolling sample period
Moving correlations between ILO rate & card delinquency rate
Card
Delinquency
Rate
ILO Unemployment Rate
Covariance -0.375
Correlation -0.242
t-Statistic -2.298
Probability 0.024
Claimant Rate
Covariance -0.808
Correlation -0.400
t-Statistic -4.020
Probability 0.000
Full Sample
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
12 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Normalized series – delinquency rate levels, quarterly
changes in unemployment rates (smoothed)
Covariance Analysis
-3
-2
-1
0
1
2
3
4
1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
Credit Card Delinquency Rate
Quarterly Change in ILO Rate (Smoothed)
Quarterly Change in Claimant Rate (Smoothed)
Card
Delinquency
Rate
Quarterly Change: ILO Rate
Covariance 0.136
Correlation 0.755
t-Statistic 10.545
Probability 0.000
Quarterlly Change: Claimant Rate
Covariance 0.145
Correlation 0.676
t-Statistic 8.417
Probability 0.000
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
13 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Prediction errors in macroeconomic credit loss forecasting models
Credit Card Delinquency & Unemployment
Full Sample Card
Delinquency
Rate
Quarterly Change: ILO Rate
Covariance 0.136
Correlation 0.755
t-Statistic 10.545
Probability 0.000
Quarterlly Change: Claimant Rate
Covariance 0.145
Correlation 0.676
t-Statistic 8.417
Probability 0.000
-0.2
0
0.2
0.4
0.6
0.8
1
1997Q1 2000Q1 2003Q1 2006Q1 2009Q1
5 year
7 year
10 year
15 year
Mo
vin
g C
orr
elat
ion
Co
effi
cien
t
End of rolling sample period
Moving correlations between quarterly changes in ILO rate & card delinquency rate
14 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Extended model of credit card delinquency rate, includes:
► Quarterly change in ILO rate
► Annual growth of household real disposable income
► Quarterly change in interest rates
What can we learn from residuals?
E.g. Tendency to under-predict delinquency rate in post-recession periods (after 1992 and 2009)?
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2
1
2
3
4
5
6
1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
Residual Actual Fitted
Strong autocorrelation in model residuals – could be addressed using
alternative specification in full sample but may reflect more
fundamental specification issues (wrong or missing variables).
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
15 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
2.0
2.5
3.0
3.5
4.0
4.5
92 94 96 98 00 02 04 06 08 10
Recursive C(1) Estimates
± 2 S.E.
-1
0
1
2
3
4
5
6
92 94 96 98 00 02 04 06 08 10
Recursive C(2) Estimates
± 2 S.E.
-.2
-.1
.0
.1
.2
.3
92 94 96 98 00 02 04 06 08 10
Recursive C(3) Estimates
± 2 S.E.
-0.4
0.0
0.4
0.8
1.2
92 94 96 98 00 02 04 06 08 10
Recursive C(4) Estimates
± 2 S.E.
Constant Change in ILO Rate
Income Growth Interest rate
Recursive regression coefficients (progressively extending estimation sample to include later dates)
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
16 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Moving regression coefficients between changes in ILO rate & card delinquency rate
Coefficient 2.737
Std. Error 0.359
t-Statistic 7.621
Prob. 0.000
-1
-0.5
0
0.5
1
1.5
2
2.5
3
1997Q1 2000Q1 2003Q1 2006Q1 2009Q1
5 year
7 year
10 year
Mo
vin
g R
egre
ssio
n C
oef
fici
ent
End of rolling sample period
Full Sample
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
17 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Predictive accuracy for model estimated on sub-sample: 2003Q1 – 2009Q4
-1
0
1
2
3
4
5
6
2004 2005 2006 2007 2008 2009 2010 2011
Actual Predicted Residual
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
18 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Predictive accuracy for model estimated on alternative 10 year sub-samples
-2
-1.5
-1
-0.5
0
0.5
1
all 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
2009
2010
2011
Fore
cast
Erro
r fo
r C
CD
Del
inq
Rat
e
Model estimated on 10 year sample ending:
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
19 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Some conclusions on aggregate analysis:
► Estimated relationships between credit outcomes and unemployment (and other factors) vary greatly depending on the estimation sample
► Even full-sample models tend to over-estimate losses in recent history (whether with or without other economic conditioning variables)
► Autocorrelation issues in full sample model could be resolved using more ‘sophisticated’ estimation procedures in the full data sample, but may indicate more basic problems with model specification.
► Alternative estimation procedures are generally not viable in samples available in practice.
► Over-estimation problem worse for models fitted on more recent history ► Changes in borrower population and/or lender behaviour probably play some role
● Not possible to control for these in aggregated models estimated over longer history
Similar results in other applications using macroeconomic data:
► Loss models fitted on aggregated or account-level lender data ► PD models fitted on bureau data ► Argues against explanations based on model specification ► Problem is (partially) mitigated in models using disaggregated economic data – e.g.
unemployment by local area or by age
Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment
20 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public CSCC XIII – August 2013 ©2013 Experian Ltd. All rights reserved.
Experian Public
Good forecasting models replicate key features of ‘data generation process’ with minimal information losses
Criteria for effective models are challenging even with abundant data
Aggregate loss forecasting modelling approaches emphasizing time series movements in economic data suffer from:
► Excess / invalid aggregation of economic (and possibly credit) data
► Excess / invalid marginalization:
● Difficult to effectively control for changes in borrower characteristics, lender & borrower behaviour
● Limited ‘degrees of freedom’ forces concentration on small set of economic metrics that dominate time series movements in credit losses
► Limited scope for testing against design criteria – weak tests for innovation errors (including stationarity), exogeneity, parameter constancy, etc.
Disaggregated approaches (for both credit and economic data) reduce information losses resulting in more stable, robust forecast models.
Prediction errors in macroeconomic credit loss forecasting models Conclusions
21 © 2012 Experian Information Solutions, Inc. All rights reserved.
Experian Public.
©2013 Experian Ltd. All rights reserved.
Experian Public