De-stressing credit risk

De-stressing credit risk: An analysis of model risk and stress testing

AuthorStephen Morriss

EditorMarsha Irving [email protected]

Disclaimer The information and opinions in this document were prepared by FC Business Intelligence

Inc. and its partners. FC Business Intelligence Inc. has no obligation to tell you when

opinions or information in this document change. FC Business Intelligence Inc. makes every

effort to use reliable, comprehensive information, but we make no representation that it

is accurate or complete. In no event shall FC Business Intelligence Inc. and its partners be

liable for any damages, losses, expenses, loss of data, loss of opportunity or profit caused by

the use of the material or contents of this document.

No part of this document may be distributed, resold, copied or adapted without FC Business Intelligence prior written permission.

© FC Business Intelligence Inc. ® 2014

www.credit-risk-analytics-summit.com

De-stressing credit risk: An analysis of model risk and stress testing

With contributions from:

Ken Baruth, Vice President, Global Chief Risk Officer, Toyota Financial Services

Jose Canals-Cerda, Special Advisor, Supervision, Regulation and Credit Department, Federal Reserve Bank of Philadelphia

James Costa, Chief Risk Officer, TCF Financial Corporation

Swaroop Yalla, Senior Vice President, Head of Portfolio Strategies, Credit Portfolio Management, KeyBank

De-stressing credit risk:An analysis of model risk and stress testing

3

Credit Risk Analytics in Retail and Wholesale Financial ServicesConference & Networking EventNew York, 15-16 September 2014 10+ C-level speakers confirmed, including: Kevin Moss, CRO, Wells FargoJoshua Bruton, CFO, United Texas BankDavid Gleason, CDO, BNY MellonHoward Bruck, CIO, Hudson Valley BankZahid Afzal, COO, Capital BankRobert Thompson, CMO, Old Florida National Bank


The years since the credit crisis of 2008 have seen a wholesale shift in attitudes towards credit portfolio risk analysis.

Regulation has tightened, with greater transparency demanded, and systems are being put in place across the industry to try to ensure that lenders have a greater level of immunity to any future crisis. Consequently, research into the financial ‘vaccines’ that will hopefully provide this greater immunity has been progressing at a fierce pace.

Banks and lenders have all approached the task of developing model risk scenarios and stress testing in different ways, however. As we shall see, although all have the same final goal, institutions have implemented a variety of different strategies to reorganize their structures and models to comply with the demands of the post-2008 world.

Here, we talk to four industry insiders to get an in-depth look at the evolu-tion of credit model risk management since the financial crisis hit in 2008. We examine how institutions are strengthening the models themselves, how those models are managed and scrutinized, and how their philoso-phies towards risk management have changed. We’ll also try to gauge the levels of trust that risk managers have in their models, and find out what our respondents see as the greatest sources of stress going forward:

Ken Baruth is Vice President & Global Chief Risk Officer at Toyota Financial Services. He worked for the company in various positions for 27 years being appointed to his current position in 2010. Previous to joining Toyota, his career has also seen him work at both Chrysler and Ford Credit. Ken holds a bachelor’s degree in Psychology from Kansas State University.

Jose Canals-Cerda is Special Advisor, Supervision, Regulation and Credit Department at the Federal Reserve Bank of Philadelphia, which serves eastern Pennsylvania, southern New Jersey and Delaware. One of Jose’s primary responsibilities has recently been to serve as lead developer of the Federal Reserve System methodology for stress testing of credit cards and charge card portfolios during CCAR and Dodd-Frank stress test exercises.

James Costa is Chief Risk Officer at TCF Financial Corporation and its subsidiary TCF Bank, a position he has held since August 2013. James has over 20 years of financial services experience, including 13 years in risk management. Prior to joining TCF Financial, he served as Executive Vice President of Risk and Head of Enterprise Portfolio Management at PNC Financial Services Group, Inc., a financial services institution, and before that led enterprise credit strategy for Wachovia Corporation, a financial services institution.

4


Swaroop Yalla is a Senior Vice President and Head of Portfolio Strategies in the Credit Portfolio Management group at KeyBank, a regional bank headquartered in Cleveland, Ohio. Prior to joining KeyBank, Swaroop was a Vice President with Morgan Stanley in the Securitized Products Group (SPG) and CRE Strategy group. He obtained his bachelor’s degree from the Indian Institute of Technology, New Delhi and also holds a Masters in Financial Engineering degree from the Haas School of Business at the University of California, Berkeley, along with a Ph.D. in Engineering from the University of Notre Dame.

How has monitoring of model risk and stress testing changed since the credit crisis in 2008?

For all of the industry figures we spoke to, the simple fact is that model risk and stress testing have become a much more important, complex and resource-hungry part of their business since 2008 – all have further invested in human resources and expertise. In the words of James Costa of TCF, ‘The investment in specialized staff and policy and board level review is unlike anything that has been done before in this space with respect to stress testing and model risk.’

According to Jose Canals-Cerda at the Federal Reserve Bank of Philadelphia, there is now a greater awareness of the possible underper-formance of models, which seemed to vary greatly across asset classes and business lines during the crisis. ‘Industry practitioners as well as academic researchers spent the last few years analyzing and learning about observed weaknesses in models and data. Many of the lessons learned are being applied to all relevant areas of model development and application, as well as model validation and governance.’

These remedies are being put into place across the industry, albeit in a variety of ways. At KeyBank, there were a number of organizational changes made, Swaroop Yalla explains, as the bank looked to raise the importance of model risk and stress testing in the wake of the crisis. ‘We actually now have a separate group for model risk. This year was the first year they actually carved it out into a new department. So it’s gained in importance. That’s the biggest difference.

‘We’re currently going through the model risk process – we have an internal system which tracks all the inherent risks in each business unit. Obviously you have the big ones – credit, liquidity, but now also model risk has become its own sort of inherent risk. We evaluate that through various controls and procedures.’

And even for institutions operating in less regulated environments, a similar scenario has been unfolding, according to Ken Baruth at Toyota


5

Credit Risk Analytics in Retail and Wholesale Financial ServicesConference & Networking EventNew York, 15-16 September 2014 Model risk, stress testing, and many more topics to be discussed by senior credit risk professionals onsite. www.credit-risk-analytics-summit.com

Financial Services. ‘I was put in this position because the CEO at the time wanted somebody with practical experience, who could take a look at our modelling and see if what the models were saying made sense.

‘When I came in here in 2010, we did not really have any sense of model risk management. So we took the supervisory guidance bulletin from the OCC, the Federal Reserve from April 2011, and established an enterprise-wide model governance function in this company.’

So in what way can models be described as weak? According to Jose Canals-Cerda, there could be a variety of sources of weakness, from intrin-sic weaknesses in data (where it lacks data from a significant downturn) to flawed model specifications, which fail to identify a certain latent risk.

‘In some cases, models simply outlive their shelf life and became unreli-able, for instance as a result of some structural change in the economy or a change in behavior of economic agents.

‘Models are now more closely monitored and more often re-evaluated; model specifications are more heavily scrutinized; and the stability of specific behavioral patterns by economic agents is more frequently monitored.’

6


What are the significant sources of stress that could create a prob-lem for your credit risk model?

In his research into model risk, as applied to credit card portfolios during the crisis, Jose Canals-Cerda has reached a number of conclusions which he believes are consistent with studies looking at other asset types: ‘What I observed is that models’ performance deteriorates as the level of stress in a certain stress scenario starts to diverge from the level of stress experienced in the period reflected in the developmental data.’

‘In the case of a stress scenario significantly more adverse than what is reflected in the historical data, we can expect significant changes in the behav-ior of economic agents that the models may have a difficult time explaining.’

Because of this, Jose advises being particularly cautious about putting too much weight on model projections in these scenarios. ‘Specifically, it is advisable to monitor model performance frequently and more generally to ensure the stability of the model’s structure. We should also consider imposing a reasonable degree of conservatism on our projections.’

This preference for a more conservative, cautious approach to credit models in order to mitigate sources of stress was also held by James Costa. ‘I would say probably the biggest is that models are almost by definition a product of what history has told us. It’s kind of like the disclosure for investments – past performance is not necessarily indicative of future performance. You just don’t know what the next economic cycle is going to look like.

‘When you look at how an economic scenario might give you a view on how a portfolio would perform, it really is only able to speak as well as that relationship was drawn based upon historical relationships. So you don’t know what you don’t know. You may kid yourself into thinking that you have remediated previously unidentified weaknesses. And that could be wrong.’

That element of surprise, despite all the recent strengthening of models, also still featured highly as a source of stress for Ken Baruth, with over-con-fidence seen as a potential trap. ‘It would be, in this business, surprises, things that we didn’t see, and that would be sudden jumps in gas prices, changes in interest rates, sudden drops in the used car market. We’ve been around since 1983, so we’re fairly confident in what we do, but with leasing now an ever-expanding part of our business, that throws in some risk. I think where we got in trouble in 2008 is that we thought we could price for risk, and that didn’t work out so well.’

For Swaroop Yalla, meanwhile, implementing a consistent approach to data integrity over a variety of different asset systems was seen as crucial in order to retain a greater degree of control.


7



‘One of the big things which everybody will tell you is the data challenge. What happens is that you have multiple loans systems in the bank, and every loan system is built a little bit differently. And you need a founda-tional data mart to bring those things together. That is the bedrock of most models. To the extent that you have any gaps in your data, that is by far the most significant source of stress. It’s the normalization process that it needs to go through before you start feeding it into any sort of credit risk model.’

8


Which stress scenarios are creating the most significant projected losses?

Both Swaroop Yalla and James Costa, perhaps unsurprisingly, immedi-ately cited the seriously adverse CCAR stress scenario as being the most significant for their companies. ‘The seriously adverse CCAR scenario I would characterize as worse than the last great recession,’ Swaroop noted, ‘so obviously that creates the most significant loss projections.’

In terms of KeyBank’s particular situation, Swaroop explained that the bank had been focusing on what the effects of the seriously adverse global situation seen under CCAR would be for a regional US bank. ‘We try to tailor the scenarios to our footprint. This is a national/international shock to the economy, which is given by the Fed and is common for all banks. But given our footprint, given our regional focus, it may have a more severe impact, so we try to take those things into account when we’re modelling that. If anything, we are trying to add on to that scenario.’

‘I think for many commercial banks, what we’re finding is the pronounced stress resulting from peak unemployment rates really gives a big, big shock to portfolios’, added James Costa. That plus the residential real estate property values shock. It’s almost an unavoidable dent in the portfolio.’

For Toyota Financial Services, meanwhile, Ken Baruth echoed this in noting that negative trends in unemployment, GDP growth and consumer confidence remained the factors that had the greatest impact on projected losses. Again, however, there were concerns with regards to model accuracy: ‘We’re entered into a time where we have an all-time low in loss ratios and delinquencies, so when you sample that kind of data and build your scorecards, you’ve got to wonder how predictive data samples are to changes. Consumers have de-leveraged, we’re at an all-time low, and I think people are wondering how sensitive our models are to change.’

Going into detail on stress testing in his own research, Jose Canals-Cerda made the point that economic conditions would affect loss in different ways across different types of financial products and asset classes: ‘My work suggests that not all determinants of portfolio loss are impacted in the same way by stress economic conditions,’ he warned. ‘In particular, for credit cards the main impact was on the probability of default primarily, along with a decrease in the recovery rate. For mortgage portfolios, the significant decrease in home prices clearly had a determinant impact in mortgage defaults and was the primary driver of the reduction in collateral value, while the significant increase in the time to disposition of collateral also contributed significantly to an overall increase in loss rates.’


9


‘It is important to look at the crisis from an historical perspective in order to realize that things could have been much worse. In a recent white paper, I analyzed the hypothetical impact of a crisis of the magnitude of the Great Recession in a portfolio of credit cards and found that loss rates would have more than doubled for near-prime accounts and more than tripled for prime accounts. Of course the results from this sort of extreme extrapo-lation should be interpreted with great caution and only from a qualitative perspective, but this type of extreme crisis analysis may still be informative for risk managers.’

10


How is your organization developing specifications on model risk for your credit portfolio?

As mentioned before, lenders have been expending a lot of resources reorganizing the way they manage credit model risk and carry out stress testing. So what exactly has been the result of this restructuring of risk management, and what has changed in the approaches of the companies towards the specifications of policy and model risk?

The ‘tiering’ of models into distinct groups was an approach highlighted by two of our respondents here. As Ken Baruth explained, ‘we’ve kind of tiered our models, in terms of the ones with the most impact, or the largest impact based on portfolio size, their contributions to our earnings estimates, and how complex the models are.’

A similar process was also underway at KeyBank, according to Swaroop Yalla. ‘We’re just going through that exercise. It’s something which we have still not completely done, but the main idea is that we’re trying to tier the models into tier one, tier two and tier three and so on. Tier one and tier two are to do with anything which is impacting capital and allocation of capital. If you have risk rating models or anything else, those will all come into tier one and tier two. Then you have others which are in the lower order of importance.

‘Each tier has a more stringent set of criteria for the evaluation of the model. So for example, if it’s tier one, you have to make sure that the controls are all tested semi-annually, you’re making sure that the data and the documen-tation is completed, and you’re making sure that there’s no misuse of the model, or if it has passed through the first and second lines of validation.’

For James Costa at TCF, the approach was a little different, with the emphasis on improving stability of forecasting under a variety of different scenarios: ‘What we do is we look at how the models perform on a forecast basis and we evaluate the stability of the forecast. Stability is conditioned upon the evolution of the economy and the evolution of the portfolio, so we know there’s going to be some volatility with respect to response to shock and then recovery.

‘But within that known curvature, just how stable is that forecast? Therefore we develop a confidence interval around that and ask ourselves, ‘do we think that our model is giving us a reasonable prediction’? There’s also benchmarking, to historical performance. If it is miles away from what we’ve experienced historically, but the stress level, the shock, is equivalent to what was experienced historically, then that also gives you a lens on how accurately the model is predicting loss.’


11



At the Federal Reserve Bank of Philadelphia, meanwhile, Jose Canals-Cerda and his colleagues have been heavily involved in developing Federal Reserve System methodology for Stress Testing of credit cards and charge cards portfolios during CCAR and Dodd-Frank stress test exercises, subject to an independent model review and validation process:

‘I am fortunate to work with a very skilled and experienced group of Ph.D. economists and analysts that combine significant industry experience with strong academic credentials. Our efforts to deal with model risk are in line with the recently published regulatory model risk guidance (FRB SR letter 11-7).’

12


What qualities does your model need to ensure against economic stress and to maintain regulatory compliance?

At TCF, James Costa referenced the various points made in the OCC’s bulletin on model risk management, which the bank saw as guide on how to develop its model: ‘You need to explore alternate specifications of models, you need to have an independent review team, and you need to have a well-documented process. There also needs to be sufficient back testing, a thorough vetting of potential predictive variables – all the things that are part and parcel of regulatory compliance in this space are the requirements that we demand of our models, whether they be third-party validations or internal validations.’

For his part, Jose Canals-Cerda noted that the recent crisis has raised the level of awareness of model risks among regulators, and highlighted the recent supervisory guidance on model risk management (FRB SR Letter 11-7) as representing the most up-to-date interagency regulatory guidance: ‘This guidance emphasizes the importance of model risk management at every step of model development, implementation, use and governance; in particular the guidance emphasizes the need for on-going monitoring of model performance.

‘The guidance represents an excellent resource for risk professionals inter-ested in building a strong model risk management framework within their organization, while at the same time complying with regulatory require-ments. It is not a one-size-fits-all; the guidance emphasizes the need for a validation framework consistent with the complexity and criticality of models, and the potential impact of model risk, for the organization.’

In addition to official guidance, the adaptability of models was something that Swaroop Yalla saw as important in keeping them effective and compliant. ‘The main thing is that it needs to be flexible, to adapt itself to various different types of scenarios. Most of these models are fed with data that goes back maybe one or two cycles. You’re basically using something which happened in the last recession, which was basically a real estate driven recession. Now, is that good enough for the next one where you may have a totally different kind of recession? I think the flexibility of the models has to be tested with longer data.’

However, there was often a problem with the usability of older data, he added. ‘In our analysis, we use the last ten or twelve years of data because, beyond that, it gets really tough to use. So you’re talking about maybe two recessions, and even 2001 was a specialized tech recession which may not have impacted us as much as the other banks.’


13


For Ken Baruth, on the other hand, the quality of the data inputted into the models remained paramount, as it has for many other industry profes-sionals who have done a huge amount of work in this area: ‘We have a lot of data, so we’re actually in the process of creating ‘data marts’, what I call a sandbox to put our data in, to test our models – those are non-production models. We’re working on data that we feel is relevant – because there’s lots of data – and what is best used by us.

‘We have also taken a look at different economic stress scenarios, so we would use Moody’s and incorporate those scenarios in our models for testing. For me, as a kind of user, I’m looking for output from the models that makes sense to me. Say, for example, my loss forecasting models keep overstating what our losses are. I want my modelers to dig in there and find out what’s created that difference. And what I’ve found is that there are some assumptions in there that need to be changed.’

14


What are the solutions that you are finding in order to deal with credit model risk?

So, given all of the changes that banks and financial institutions have made to model risk management, what seems to have worked?

Making sure that a strong validation framework is in place could be seen as being the ‘front line’ of defense, according to Jose Canals-Cerda. Constant monitoring was seen as key, with recourse to benchmarks wherever possible.

‘The model risk guidance emphasizes the “effective challenge” aspect of validation, which assures that the proper incentives and competences are in place to guarantee that sound decision making is implemented. Also, the extent and rigor of the validation framework should be in line with the potential material impact of model errors. In times of crisis, when structural changes in the underlying economic or financial conditions are more likely, it is recommended to conduct more frequent ongoing analyses of model fit as well as more frequent analyses of the stability of model parameters, assumptions and structure.’

For Swaroop Yalla, meanwhile, having an alternative model made a lot of sense. ‘You need to have a challenger model. You may have built a really good ‘Rolls-Royce’ of a model, but you also want a backup. A challenger model is very important for most of these credit risk models. I think that should become a regular feature where you have a main model and a challenger model, with output coming from both, and doing a ‘what if ’ scenario analysis on the challenger model to figure out what you expect from the main model. In our case, we’re looking at ways to use vendor models, for example models from Moody’s which are being used as challenger models right now.’

There was danger too in seeing models as a ready-made, ‘silver bullet’ tool, said Ken Baruth, who also again emphasized the human factor in managing risk. ‘There’s no such thing as a perfect model. It can contin-ually be worked on. But you need to understand what’s in your model, what changes you propose to make, and watch its output. I think models are great, as long as they’re used as a tool, and not as the end-all for your decisions. You’ve got to have common sense in there.’

James Costa agreed, and, like Ken, was adamant that models should never be seen as static, but rather as a constantly evolving aspect of the business: ‘I don’t think the development of models is ever going to stop. There is no final state here. But where we acknowledge that there is uncertainty, we size it, and we communicate it to our oversight committee, which includes the board.


15



‘This is one of those situations where you can’t live without the models. Every model will have some degree of error in it, and that needs to be appreciated from the get-go. If the expectation of the model is going to be ‘right’, then you wouldn’t need a model because you would have full knowl-edge of what is in front of you.’

For James, as advanced as the models are, the human element is overlooked at your peril. ‘There’s no substitute for common sense and experience – even though managing complex portfolios is well beyond an individual’s ability to size the risk just based upon intuition and experience. There’s a necessary compliment between experience and sound business judgment and what a model can tell you. I would not do without either of those.’

16


Conclusions

There seems to be a different way of thinking about credit risk model-ing in the post-2008 world. Its importance seems to have increased, but conversely, confidence in the efficacy of the models – at least in their previ-ous form - seems to have declined.

As Jose Canals-Cerda noted - with his views echoed by our other respon-dents - there is a greater awareness now of possible underperformance, and a greater awareness of the potential impact of model errors. And with this, there is consequently a greater knowledge that models need to continuously evolve, be fed with better, more relevant and more stringently checked data, and have robust systems in place to check the validity of their results.

Given all this, some strategies for developing more robust risk modelling were seen across different institutions. Both Swaroop Yalla at KeyBank and Ken Baruth at Toyota Financial Services explained that they had been ‘tiering’ their models based on impact on capitals, while both Swaroop and James Costa at TCF stressed the importance of putting alternate, challenger models in place to create a competition amongst models.

The need to maintain the utmost consistency of data, alongside a strong validation framework, was another point that was brought up again and again. The difficulty of this, as a number of our respondents explained, was that due to the divergent nature of various recessionary periods, it was not always easy to reconcile the differences in data from different periods in time.

And finally, the human element seems to have gained in significance since 2008. The investment in specialized staff, policy and board level review that almost all institutions have undertaken is perhaps the greatest change the sector has seen, and perhaps can be regarded as the best guard against the overconfidence of the last decade.

That different way of thinking about model risk has not – and will not - come cheap.

17


FC Business Intelligence’s Credit Risk Analytics in Retail and Wholesale Financial Services conference and exhibition takes place on the 15-16 September 2014 at the New Yorker, New York. The event will draw together thought leaders from banks, credit unions, credit card companies, credit bureaus, ratings agencies, data services, software services and government to ignite discussion on critical issues around big data, credit risk and analytics in the financial services sector.

With contributors such as the ones who helped form this report along with other key spokespersons from international financial institutions, the event will be number one for credit risk analytics in 2014.

To register, visit our website www.credit-risk-analytics-summit.com and quote CRAR100 to get an additional $100 off of our current listed prices.

Credit Risk Analytics in Retail & Wholesale Financial Services

Revolutionary credit risk management! Increase profitability and decrease default risk using cutting-edge analytics

September 15-16, 2014 • The New Yorker, New York

to save up to $400

BOOK EARLY

Master the latest credit risk management techniques and update your data analytics know how:

✓ Put analytics at the heart of your business: create a central hub that integrates enterprise-wide systems and expertise

✓ Think above and beyond compliance: get to grips with regulatory variables and take the opportunity to outshine your competition

✓ Break-down big data: understand the latest trends in sourcing, aggregating and warehousing vast amounts of data

✓ Game changing analytics solutions: overhaul your existing credit risk analytics platform with up to the minute products and solutions

✓ Achieve a customer-centric vision: create the most effective response to customer behavior patterns to better mitigate risk

EXPERT SPEAKERS INCLUDE:

CREDIT RISK ANALYTICS EXPERTS FROM:

Organised by:

FC BusinessIntelligence

“This event is shaping up to be the best of its kind in 2014, with a Who’s Who lineup of speakers and attendees, and an agenda that addresses the full spectrum of issues pertaining to Credit Risk Analytics.”

Randy Banner, Credit Risk Manager

Kevin Moss, EVP, Chief Risk Officer, Consumer Lending Group, Wells Fargo

James Costa, EVP, Chief Risk Officer, TCF Bank

Bruce Brenkus, Chief Risk Officer, Kabbage

Joshua Bruton, Chief Financial Officer, United Texas Bank

Chris Ballinger, Chief Financial Officer, Toyota Financial Services

Robert Thompson, Chief Marketing Officer, Old Florida National Bank

http://www.credit-risk-analytics-summit.com