Model Governance and Model Risk Management: Risk Manager’s Perspective

Nikolai KukharkinQuantitative Risk Control, UBSMeasuring and Controlling Model Risk, New York, October 2011

DISCLAIMER

The views and opinions expressed in this presentation are those of the author and may not reflect the views and opinions of UBS and should not be cited as being those of UBS.

2

What Can Go Wrong With Models?

More extensive policies, stricter regulations, and more comprehensive model risk management programs.

3

Kill All the Quants?... *

Risky Business on Wall Street: High-tech supernerds are playing dangerous games with your money

TIME magazine, April 11, 1994

Recipe for Disaster: The Formula That Killed Wall Street

Wired Magazine, Feb 23, 2009

The Minds Behind the Meltdown: How a swashbuckling breed of mathematicians and computer scientists nearly destroyed Wall Street

WSJ, Jan 22, 2010

Financial Crisis Can Be Traced to “the Quants”

The Kansas City Star, Feb 22, 2010

*) Andrew W. Lo, “Kill All the Quants?: Models vs. Mania in the Current Financial Crisis”, 2009.

4

…Before They Are Born…

Too large a proportion of recent "mathematical“ economics are mere concoctions, as imprecise as the initial assumptions they rest on, which allow the author to lose sight of the complexities and interdependencies of the real world in a maze of pretentious and unhelpful symbols.

John Maynard Keynes, The General Theory of Employment, Interest and Money, 1935

5

…Or May Be Not?

Myron Scholes, Risk Magazine, September 2011

It should be a golden age for risk modeling and management

“One thing about a crisis is that it shakes old opinions and you start learning new things. I hope we do – I am very bullish on the future for quants.”

He warns against overreliance on models, and concedes they had a role in the crisis. But the common-sense reaction – embracing intuition, and rejecting the use of modeling and quantitative techniques – is also flawed, he argues…

Presumably you used your intuition in picking the model, and intuition can fail, too.

6

New FED/OCC Supervisory Guidance on Model Risk Management

Expands on existing regulatory guidance by broadening the scope beyond model validation to include all aspects of model risk management at all stages: model development, implementation, and use

Revises and expands model and model risk definition

Establishes comprehensive model risk management program requirements

More formalized and expanded model governance and controls

Increased standing of model risk management function: needs to be influential; have explicit authority to challenge model developers and users

Model validation

Introduces “effective challenge” standard

Key elements of comprehensive validation : Evaluation of conceptual soundness On-going monitoring Outcomes analysis

7

Comprehensive Model Risk Management Program Requirements

The bar has been raised significantly with respect to the scope, formality, rigor, and prominence expected of banks’ model risk management programs.

“Model risk should be managed like other types of risk.”

Life-cycle view of model risk - model risk management framework is expected to include standards for model development, use, and maintenance to which all model owners, users, and other stakeholders will be held.

Broader roles and responsibilities – it is not just the responsibility of the model validation unit: model developers / owners, users, validators, senior management, internal audit

Model risk management is an on-going, continuous, process – not a periodic activity: Monitoring model risks and limitations identified during development and

validation Monitoring and on-going validation of changes (i.e., products, exposures,

activities, clients, or market conditions) that may impact model risks Regular model performance monitoring (i.e., back-testing, benchmarking,

sensitivity analysis, and stress testing) Model risk reporting to senior management and the board of directors.

8

How Should Banks Respond?

Examiners expect a bank to perform a self-assessment against new regulatory guidance, and have a clear action plan for closing identified gaps.

Potential action plan items may include the following: Revisions to policies and procedures Revisions to roles and responsibilities Organizational changes Development of new standards and guidelines for model

development, implementation, and use Revised model inventories (including an inventory of model-specific

risks and limitations) Mappings of model risk mitigation controls against existing inventory

of model risks and limitations Creation / enhancement of on-going model monitoring processes Creation / enhancement of model risk reporting Additional model validation testing (e.g., vendor models) Creation of annual model review process

9

Model Validation: What’s Next?

Financial industry obtains a significant share of revenue from products valued by mathematical models

Models are here to stay and reliance on them will only grow

Consequently, model risk is a topic of great, and growing, interest in the risk management arena How to define it How to measure it How to manage it

Qualitatively reasonably well defined

Much less successfully quantified and even less successfully managed

What is expected from model validators and how is the role changing?

What are the key priorities for model validators?

10

Model Risk: Define and Manage A model can be defined as:

A simplified description or representation of an entity or process, property, characteristic or behavior which cannot be represented or predicted with complete certainty.The output of a model is therefore an estimate or approximation.

1. Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.

2. Model risk is the risk of error due to a deficiency in design or implementation of a pricing model. In other words, model risk is the risk of occurrence of a significant difference between the mark-to-model value of a product and its fair value.

Or more flexible definition by R. Rebonato

3. Model Risk is the risk of occurrence of a significant difference between the mark-to-model value of a complex and/or illiquid instrument, and the price at which the same instrument is revealed to have traded in the market.

Note that neither “true” nor “fair” value is mentioned, i.e. ->>> Market is the king More sophisticated / realistic / correct model is not necessarily the best

11

Model Governance Process

Regulators on Model Risk: FSA

“Model Risk contributes to overall valuation risk. Model validation and model risk management processes are important elements of any valuation control framework. Whilst effective model validation is fundamental, model validation, however good, does not remove model risk. Few firms have sufficiently well developed frameworks for articulating model risk tolerance, and measuring and controlling model risks within that tolerance. We believe a better defined and implemented model risk management framework could therefore feed into a better defined and implemented valuation risk-management framework.”

- from FSA’s “Dear CEO” letter on Valuation and Product Control principles, August 2008

Regulators on Model Risk: FED and OCC Supervisory Guidance on Model Risk Management, April 2011

Model risk should be managed like other types of risk. Banks should identify the sources of risk and assess the magnitude. Model risk increases with greater model complexity, higher uncertainty about inputs and assumptions, broader use, and larger potential impact. …With an understanding of the source and magnitude of model risk in place, the next step is to manage it properly.

12

Model Risk Management Framework

Goal is to set up a framework to explicitly, fully, and dynamically account for model risk

Partially accounted for by:

Qualitative capture Models certification Periodic model risk review

Quantitative capture Model reserves Sensitivity analysis Portfolio reviews

13

Model Risk Management Framework

Governance: Independent Verification Unit (IVU) with the mandate for:

- independent review and certification of valuation models

- independent risk-based review of model-related risks, i.e., the risk that the model either through a deficiency in design or implementation produces faulty output.

The aim of the certification process is to obtain the required level of comfort that the model in question is functioning in an accurate and appropriate way.

Model performance monitoring process:

- Full review of potential model risks and existing control processes across systems and product areas taking into account factors including materiality, model choice, model applicability

- This is independent from running certification processes, it challenges existing model issues and includes actions that are or should be taken

Frequency and depth of reviews:

Regulators requested model verification updates of the “high-risk” to be done more frequently, annually, compared to previous 5-y cycle for all certifications.

14

Model Governance Process – High-Risk Models

Definition of High-Risk Models Define dimensions and criteria to categorize models as

“high risk” Definition is altered based on new information / past

actions

Capture of High-Risk Models Agree on review types Run / participate in the reviews throughout the year

Review Filter all models according to preliminary criteria Working Groups of IVU product specialists adjust filtered

product list and define necessary actions

Actions based on Reviews Consolidate results of reviews already undertaken Take further actions if necessary

Definition

Capture

Review

Actions

15

Definition - Model Risk Types

The following major model risks types were defined

1. Model inconsistencies or approximationsInconsistencies in mathematical assumptions of the model or its implementationModel assumed fit for purpose, although does not fully capture some featuresThe model may be used outside of its range of applicability

2. Model choiceModel choice uncertainty - several models are available (and one is being used)Many solutions could satisfy the same constraints

3. Calibration, model parameters, and input data issuesMultiple sets of parameters can satisfy market; multiple sets of calibration instruments available, sometimes model cannot fit all of them simultaneously; uncertainty in model input parameters

4. Controls (booking approximations + level of oversight)The control environment into which the model is being released.Level of oversight by other control groups and therefore the probability of an error being detected

5. Complexity Exotic features, number of inputs, and the importance of inter-relationships between themmodels assumptions/conditions

6. Model/Product maturity and level of standardizationMaturity, liquidity and rate of change of the market

16

Capture – Model Risk Scores

Model risk scoring process

Start with the certified product list and link and rank it according to the risk scores in several dimensions as well as materiality of positions:

Each product is described in terms of “High=3”, “Medium=2” or “Low=1” Risk Scores for each of the following six risk factors (combined Risk Score is between 6 and 18):

1. Model inconsistencies or approximations2. Model Choice3. Calibration, model parameters, and input data issues4. Controls (booking approximations + level of oversight)5. Complexity (exotic features, number of inputs, models assumptions/conditions)6. Model/product maturity and level of standardization

Capability to compare model risk between models / products across all areas

This list is used as a starting point to identify models/products which will be subject to the annual review

Additional win – not just a formal more frequent re-certification, but rather a review targeting specific features which make this product/model high-risk

17

Is Theoretical Value Fair?

Most of the issues listed in the previous section can also give rise to model fair value adjustments

Fair value is the price that would be received to sell an asset or paid to

transfer a liability in an orderly transaction between market participants

at the measurement date (FAS 157)

Accounting standards demand that if the value of asset or liability is not directly observable but rather obtained from a model (“marked-to-model"), it needs to be further adjusted to bring it into closer alignment with the market fair value.

Why is such adjustment needed?

Model can have a known bias/deficiency There can be an uncertainty around the model generated value due to:

existence of the alternative models (i.e., no industry standard) non-uniqueness of calibration uncertainty around (unobservable) model inputs

Terms “Model Fair Value adjustments” and “Model reserves” are often used interchangeably, but they mean different things and should not be mixed up

18

What Do Accounting Standards Say About Model Reserves?

They say… NOTHING

Model reserves may serve as a proxy, intuitive way to account for perceived model risk

Model FV adjustments aim at “fine-tuning” model generated number to bring it into a better alignment with the market price. FV adjustments make the “best educated guess” of where the market is

Model risk attempts to assess the tails of the theoretical price distribution, expresses how far off our “best guess” might be from the realized price

Model risk arises from the uncertainty in model specification, be it the model parameters and/or inputs (i.e., function arguments), or the model (function) itself. When not observable, FV of an asset is a variable characterized by some probability distribution. While model FV adjustments attempt to pinpoint the center of such distribution, its higher moments are the domain of model risk.

NOTE: Frequent practice of using “parameter uncertainty” and sometimes “alternative model” reserves to create a “conservative cushion” roughly the size of perceived model risk, contradicts the accounting standards which concentrate on fair value.

19

Inherent Model Risk

Valuation uncertainty beyond model FV adjustments is the domain of (inherent) model risk

Can be viewed as “residual” model risk. With proper model validation in place, inherent model risk can be minimized but never eliminated

Needs to be measured, monitored, and managed

Deserves a place in overall risk management framework on a par with market and credit risk

Should be considered alongside the market and credit risks in allocating capital, making business decisions, and managing the trading positions

20

Final Observations

There is always a risk that a model can be “wrong”

In part model risk is a variety of operational risk, i.e. the possibility of a human error

However, there is also an inherent uncertainty due to the very nature of financial modeling

The purpose of model governance is to set up policies and procedures that: 1) minimize operational risk

- Achieved through model validation, periodic reviews, model change management, back-testing, etc. 2) provide for measurement, monitoring, and management of

inherent model risk (model uncertainty)

- Requires recognition of model uncertainty’s role alongside market and credit risk, and devising and implementing methods, processes and systems for measuring capturing reporting and managing model risk

Therefore, model risk deserves a place alongside market and credit risk in making business decisions (e.g., in capital allocation) as well as in the risk management and reporting process.