Tsuyoshi Nagafuji

7/28/2019 Tsuyoshi Nagafuji

http://slidepdf.com/reader/full/tsuyoshi-nagafuji 1/35

Validating AMA frameworks

- A Regulator’s Experience in Japan

2nd International Conference on Operational RiskSao Paulo, Brazil, June 5, 2009

Tsuyoshi NagafujiFinancial Services Agency, Japan

This presentation does not necessarily expressestablished views or policies of the FSA.



2

Overview : Objective

Sharing my experience in validating and approving Japanese

banks’ AMA applications.

¾Presenting what we have done or what we are actually doing inJapan, rather than what we hope to do.

¾Focusing on the factors that remain until the final stage for

application, which banks find difficult and time consuming to

address.

9Model – sensitivity analysis / stress testing: Do you know all

the possibilities for strange behavior?

9Scenarios – rules and documentation: Have you done your best to exclude subjectivity?

9Use test: Are you actually using the framework? Is it really

working?



3

Overview : Method

I am using a realistic “model of AMA models" as an example, inconsideration of anonymity.¾ I am using “The model of AMA models” that I presented at the

“Operational Risk Scenario Analysis Workshop” held at Bank of Japan, the central bank, in 2006*.* The model presented here is the same as the one I presented in 2006, but the descriptionis simplified. Please see “Quantification of Operational Risk Using Scenario Data (Nagafuji,2006)” for the details.

¾ The model is extremely simplified but still retains someaspects of typical AMA models used by Japanese banks.9 The model is based on real internal data and real scenario

data from major Japanese banks.

9 The model has a similar structure to typical Japanesemodels.



4

Overview : Outline

Presentation Overview (5 minutes)

1. Context (5 minutes)

2. Sample Model (10 minutes)

3. Validation of the Sample Model (15 minutes)

Concluding Remarks (5 minutes)

Q & A (5 minutes)

(Total: 45 minutes)



5

Overview1. Context

2. Sample Model3. Validation

Concluding Remarks

Appendix



6

0

20

40

60

80

100

120

H S B C C i

t i R B

S J P M B O

A

C r e d i t A

g I C B C S C

H B O

C

P a r i b

a s

B a r c l

a y s

C C B C H B

O S

U n i C

r e d i t I N

G

W a c h o

v i a R a

b o

B r a d

e s c o I t a

u

U n i b a

n c o

B B

Context: Japanese Banking Industry

Consists of three "Mega” banks and many smaller banks.

Foreign banks play a very small role.

World banks by Tier I capital ($billion) (Source: The Banker, July 2008)

MUFG$82 bil

Mizuho$49 bil6

1518

SMFG$44

… … … … … … … …

2838

48 50 66 81 89 94

Top 20 Banks Top 21 – 100

<Japanese banks>98 banks in the Top 1,000 list. <Brazilian banks>14 banks in the Top 1,000 list.

(Now merged)



7

Context: Op Risk in Japanese Banks Operational risk losses are extremely small.

Total annual lossamount

(Average Dollar Amount by year,percentages of total assets)

Loss frequencies(# of losses greater or equal to

$20,000, per year, per total assets of $1 billion)

* Both figures are medians of the banks that participated in the exercise

(Source) 2004 U.S. LDCE, 2007 Japan LDCE (See “Appendix: References aboutJapanese AMA implementation" for detail).

0 0.5 1 1.5 2

Japanese

banks

US banks

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07

Japanese

banks

US banks

About 1/40 About 1/20



8

Context: Application Timetable

■ Typical validation process

■ Currently one banking group has been approved for the AMA.

1. PreparationBanks are encouraged to develop their framework

to a practical level and use it for their internalpurposes before going into the parallel run.

2. Parallel Run (At least one year)

Two capital calculations are verified through visitsand regular discussions.

3. ApprovalBanks that do not meet the requirement stay at

stage 2 or go back to stage 1.3. Approval

OK?

1. Preparation

Ready?

2. Parallel Run



9

Overview

1. Context

2. Sample Model

3. Validation

Concluding Remarks

Appendix



10

Sample Model: Overview

Several major banks are treated as if they were a single big bank. 9 Their internal loss data and scenario data are put into a very

simple loss distribution model (LDA).

LDA Model Results

Scenario Data

Internal LossData

Bank A

Bank B

Bank C

Bank X



11

Sample Model: Quantification Model

■ Monte Carlo simulation (100,000 simulations)

¾ Frequency: Poisson distribution

λ = Frequency of Scenarios

+ Frequency of Real Loss Data

¾ Severity: Empirical distribution

■ Single unit of measure (= top of the house calculations)







14

Sample Model: Scenarios (2/4)

■ Common Scenarios 1: EarthquakesLosses by historical earthquakes are estimated for each bank and then aggregated.

Description (year ,magnitude of earthquake)

Frequency(once in Xyears)

Severity(largest=100)

Details

Earthquake in Tokyo 1,200 100 Earthquake greater than any of those below is assumed.Keian (1649, 7.1) 49Genroku (1703, 8.2) 85

Ansei Edo (1855, 6.9) 55Meiji Tokyo (1894, 7.0) 47Great Kanto (1923, 7.9) 82

Hoei (1707, 8.4) 57 Ansei (1854, 8.4) 50Nobi (1881, 8.0)

400 each

55

(Frequency) 8 large-scale earthquakes between 1600 and1925 in Tokyo, Nagoya and Osaka are listed, assuming eachwill occur once every 400 years.(Severity)<Buildings> The damage to the building, furniture and the

opportunity cost due to interruption of business arecalculated based on the earthquake intensity and quakeresistance of the buildings.

<Systems> Extra work cost, damage to the machines andequipment and the opportunity cost due to businessinterruption are calculated. Damage to the computer center and paralysis of the head office functions are assumed.

<Other> Declines in the value of the loans (includingimpairment of the value of collateral) are not factored in.

Tokyo (1926) - Aichi(1997) (61 earthquakes) 77 each

Average0.4

(Frequency) 61 earthquakes occurred between 1926 and 97(of intensity 5- or higher) are listed, assuming each will occur once every 77 years.(Severity) as shown above.



15


■ Common Scenario 2: Failure in the settlement system A scenario was created where “a failure occurs in the computer systems commonly

used by the banks once every twenty years, causing total damage of JPY 20 billion.”

The f ollowing scenarios from a bank were referred to in creating this scenario.Frequency Severity Details

Once inseveraldecades

JPY several billions($US tens of millions)

A failure in the accounting system or in the domesticnetwork, which would take 12 hours for full recovery.

Once in

severaldecades

JPY several

hundred millions($US several millions)

1) A failure occurs in the communication infrastructure, or, 2)

there is a flaw in the emergency handling procedures,causing interruption of the settlement operation for half aday. The compensation for damage paid to securitiesexchanges as clearing agents in charge of settlement of thegovernment bonds is included.

Once in

severaldecades

JPY several billions

($US tens of millions)

Foreign exchange / settlement operations are not performed

for a full day due to a system failure

Once everyseveral years

JPY severalhundred millions

($US several millions)

A failure occurs in the Zengin System just after 9:00 am. Thesystem recovers at around noon. However, the settlementoperation is erratic during that day.



16


■ Independent Scenarios<a> Actual scenarios collected from banks are used as they were.

<Scenarios Used>

Major scenarios (scenarios for larger amounts of losses)BIS event

types # of scenarios ExamplesInternal Fraud 30 Fraud in the market trading functions, withdrawal of customer fundsExternal Fraud 3 Swindles, compromised online bankingEmployment 5 DiscriminationClients,

Products

30 Lender’s liability, inappropriate advice to customers, failure to

explain the risks, etcPhysicalassets

11 Terrorist attacks

Systems 12 Failure in the accounts transfer system, including interruption of theaccounting system

Process 38 Failure in bond settlement (overseas), improper identity verification,

error in cash transfer, etcTotal 129

<b> Made-up scenarios for banks that did not have scenarios are also used.

→ Some scenarios from <a> scaled by the total assets of each bank

.







19

Overview1. Context


Concluding Remarks Appendix



20

Validation of the Sample Model: Overview

Three major points that banks have trouble with in completingtheir AMA application.

(1) Model: Sensitivity analysis / stress testing(2) Scenarios: Rules and documentations

(3) Use test: Are you actually using the framework?

Model Risk Amount

Scenarios

Internal Losses

BEICFs

External LossData

(1)

(2)

(3) Use test

Sensitivity analysis/

stress testing: Do youknow all the possibilitiesfor strange behavior?

Rules and documentations:

Have you done your best toexclude subjectivity?

Are you actually using

the framework? Is itreally working?

Scenarios



21

Validation: Model (1/3)

Statistical integrity of the model is essential, but has not been adetermining factor at the last stage.

<The sample model>

9 LDA model

Independence between frequency and severity should be accounted for.

9 Choice of distributions and estimation methods

Those may not be great discussion points as the sample model uses empirical

distribution.9 Granularity

Independence among the data points should be accounted for.

Æ As long as assumptions in the model are clarified and accountedfor, this factor is not a decisive one.



22


Instead, sensitivity analysis / stress testing has been a greatchallenge at the last stage.9No model is free from “strange” (counter-intuitive) behavior.9Comprehending all the possibilities of “strange” behavior is

time consuming, especially when the model is complex.

<The sample model>

Massive losses have a large impact.

A small change in frequency for a massive loss may have a large impact.<Simplified example> (Frequency: Poisson, Severity: Empirical)

Data set 1) and data set 2) give completely different risk, although the onlydifference is the frequency of a single big loss!

Losses EL 99.9％

1) One JPY100 billion Loss (Once in 999 years)

+ 100 JPY 10,000 Losses (Each once in 10 years)

0.1

billion

100

billion

2) One JPY100 billion Loss (Once in 1000 years)

+ 100 JPY 10,000 Losses (Each once in 10 years)

0.1

billion

0.0003

billion

3 / 1,000,000



23


Comprehending unusual (counter-intuitive) behavior of a modelis essential for regulatory and internal purposes.

We request banks:

9 To comprehend possible “strange” behaviors of their model.

9To address those “strange” behaviors. Accept them (Management should fully understand the

consequences). Revise and reconstruct their models.

Æ May take a lot of time.



24

Validation: Scenarios (1/3)

Estimation of frequency and severity is essential, but not adetermining factor at the last stage.Æ Scenarios for the sample model must include “once in 1000

years” events, as the model uses empirical distribution for severity.

ÆWe verify this through9 Checking the logic.9 Checking facts that scenarios are based on.9 Benchmarking scenarios between banks.

<Examples> Earthquakes:

Is the use of past earthquakes appropriate?What is included as losses from earthquakes?Is the latest seismological knowledge utilized?

System failures:

Are the statistics on computer failures utilized?What is the accuracy of the statistics?

Are the statistics used appropriately?

Æ As it is impossible and inappropriate to press one specificview, this factor has not been a decisive one at the last stage.





26

Validation: Scenarios (3/3)

We request banks to:

9Do their best to set rules that ensure the same estimateregardless of who the estimator is.

9Fully document and account for subjective judgments thatremain.

This looks easy to accomplish, but turns out to be verychallenging, because:

9Rules can be set only after experience is accumulated.

9This is often neglected until the last stage.



27

Validation: Use test (1/3)

“Use test” is a strong tool to improve an AMA framework. Thus, itis very challenging and often becomes a determining factor at thelast stage.

Our “use test” in Japan is not special.¾ Banks should show that they use their AMA framework in their

day-to-day risk management (= The framework is notexclusively for regulatory purposes).

¾ Use test is based on the idea that supervisors can be moreconfident with an AMA framework that is “really used”.



28


Banks decide how to demonstrate their compliance with the usetest (= Banks decide how to use their AMA framework).

Many Japanese banks choose to base their “risk management

cycle” on their AMA model.

Evaluate riskreductionmeasures based

on the model.

Implementrisk reductionmeasures

based on themodel.

Verify the

results usingthe model.

Risk reduction measures・ Introducing double

checking・Computerize operations ・Restricting operations ・…

3. See

2. Do.Plan

Model/

Risk



29


“Use test” imposes improvement on their overall AMA framework.

¾It imposes improvement on the model. 9 The model needs to be practically free from “counter-intuitive” behavior 9

The model needs to be sensitive enough.

¾Understanding by management and business units is essential.

Thus, it often becomes a determining factor at the final stage.

¾When the AMA framework does not meet the use testrequirement, it needs modification.

¾When the modification is drastic, banks are required to take the“use test” again, which needs at least half a year to complete.



30

Overview1. Context


Concluding Remarks Appendix



31

Concluding Remarks: In Theory…

■ In theory, capturing 99.9% risk is the single most importantrequirement of AMA models.9 Do distribution assumptions capture 99.9% risk?9 Are scenarios representing once-in-1000-year events?

■ However, this requirement does not turn out to be the remainingfactor at the last stage.9Choice of distribution or estimation of scenarios boils down to

subjective judgments, which are argumentative, but notdecisive.9Banks that cannot address these issues cannot enter the

parallel run.



32

Concluding Remarks: In Practice…

■ The following practical factors have often played a decisive roleat the last stage.

1. Comprehending unusual (counter-intuitive) behavior of themodel and preparing for it.

2. Setting rules and perfecting documentation to minimize thesubjectivity of scenarios.

3. Meeting the “Use test” requirement

■ Those factors ensure workable framework both for internal and

regulatory purposes.



33

Concluding Remarks: Challenges for Banks

■ We do not press banks to have an ideal or very sophisticatedframework. Rather, we ask them to have a practical, reliableframework to meet requirements for regulatory purposes.

■ After all, it is up to banks to build an AMA framework that trulyenhances their risk management.



34

ご清聴ありがとうございました

Questions?

For further questions, feel free to contact:



35

Appendix: References about Japanese AMA implementation

Sample model used in this presentation9 Quantification of Operational Risk Using Scenario Data (July, 2006)

http://www.boj.or.jp/en/type/release/zuiji_new/data/fsc0608be3.pdf

Losses of Japanese banks (Page 7 of this presentation)9 Results of the 2007 Operational Risk Data Collection Exercise (August, 2007)

http://www.fsa.go.jp/en/news/2007/20070810-2.pdf 9 A research paper comparing the loss data between the U.S. and Japan (April, 2008)

http://www.boj.or.jp/en/type/release/adhoc/data/risk0804a.pdf (For the results of U.S. LDCE, see http://www.bos.frb.org/bankinfo/qau/papers/pd051205.pdf )

Others

9 Use of External Data for Operational Risk Management Workshop (April, 2008)http://www.boj.or.jp/en/type/release/adhoc/fsc0804a.htm

9 The Effect of the Choice of the Loss Severity Distribution and the Parameter Estimation Methodon Operational Risk Measurement (December, 2007)

http://www.boj.or.jp/en/type/ronbun/ron/research07/ron0712c.htm

9 Discussions on Further Advancing Operational Risk Management (Part1: June 2006, Part2:

August 2006)Part1: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608c.pdf Part2: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0612a.pdf

9 Operational Risk Scenario Analysis Workshop (July 2006)http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608a_add.htm

Documents

Tsuyoshi Nagafuji