Operational risk is one of the mostsignificant risks that businesses facein today’s complex global economy.

For most of the world’s leading institutions ithas become more than apparent thatimplementing an effective operational riskmanagement programme can help reducelosses, lower costs associated with fixingproblems and increase customer andemployee satisfaction, thereby improvingfinancial performance and enhancingshareholder value.

Basel II may have forced banks to reviewtheir approach to managing operational risk,but for most leading institutions the questionwas never whether to establish such aprogramme, it was how. But many institutionsare still unsure of the benefits. Some are stillstruggling to decide whether to comply withthe BIS basic indicator, standardised oradvanced measurement approach.Nevertheless, compliance issues aside, mostbanks have come to the conclusion that if theyare going to have to establish an operationalrisk management programme then they want itto be based on a sound framework. What isperhaps surprising though is that while we aremany years into this process, there is still noindustry consensus on what shape or form thisframework ought to take. And while there hasbeen much heated debate on this issue, muchof it has been based on personal opinion andnot fact. This is because, even today, a numberof fundamental misconceptions exist about thetrue meaning of operational risk managementin its modern conception. The purpose of thispaper is to shed light on one of the main issuesthat is driving this confusion.

Many people believe that managingoperational risk can be accomplished byfollowing the Committee for Sponsoring

Organizations of the Treadway Commission(COSO) approach.1 The recently releasedCOSO framework sets the standards forenterprise-wide risk management (ERM).COSO views ERM as a process aimed athelping organisations identify potentiallyadverse events and subsequently manage theassociated risks in furtherance of the entity’sbusiness objectives. When applied tooperational risk management this is oftentranslated to mean: begin with acomprehensive survey of the organisation toidentify, define and assess the full spectrum of‘risks’ in each business’ underlying processes.Then define a series of responses or controlsto mitigate the risks that threaten to preventthe entity from meeting its objective. This isoften accomplished by establishing a list ofissues and follow-up action plans to ensurecompliance with this programme can beverified over time through the audit process.

At a macro level, this approach appears bothcomprehensive and sound, but the devil is in thedetails and the specious logic underlying COSObecomes evident during implementation. WhileCOSO may help organisations identify andresolve some of their more obvious controlweaknesses, in our view, it is completelyinappropriate for use in operational riskmanagement. Fundamentally, COSO isinappropriate for use in operational riskmanagement because the definition of risk usedunder this approach is wholly inconsistent withthe definition of risk used in the riskmanagement industry and by the BIS (see nextsection for a full explanation of this point). Inaddition, the method COSO prescribes for anorganisation to assess its risks is highly subjective,overly simplistic and conceptually flawed.

COSO not only fails to help a firm assess itsrisks, it actually obfuscates the risk assessmentprocess. Because risk assessment is afoundational element in the risk managementprocess, and because COSO yields an entirelycounterfeit set of risks, the spurious andmisleading results of the flawed riskassessment stage contaminate everysubsequent stage of the process. As a result,the recommended risk mitigation strategy –the set of controls and action plans designedto mitigate the identified risks – is likely to benon-optimal at best. In the worst case, it maylead organisations to expand and intensifycontrol structures in areas where they arealready over-controlled, while completelyignoring areas of major control weakness,leaving the organisations both oblivious andvulnerable to huge operational losses thatcould hit them like a bolt from the blue.

Ali Samad-Khan

Why COSO isflawedCOSO not only fails to help a firm assess its

risks, it actually obfuscates the risk

assessment process. By Ali Samad-Khan

January 2005 www.operationalriskonline.com 1

In our view, COSO – as it is currentlyapplied – is a wholly inappropriate approachfor managing operational risk; it is a hugewaste of resources and is very likely to domore harm than good.

One obvious issue with COSO is that it ishugely resource-intensive. This is becauseCOSO requires that all processes be assessed,irrespective of their individual contributions tothe organisation’s total risk (because onecannot know the level of contribution to totalrisk without first conducting a risk-assessment). Identifying and documenting therisks in each and every process could takemany person-years. One mid-sized bankrecently estimated that it would require 192person-years to complete such an assessmentacross the entire organisation. Clearly, the costof such massive resource commitment is notsomething an organisation can easily absorb,particularly if the exercise needs to be repeatedon an annual basis, which is necessary becausefor operational risk management to beeffective it must be implemented through adynamic process with continuous monitoring.Still, this is not a problem as long as the costcan be justified.

A serious problem with COSO has to dowith the way the ‘risk’ information iscollected. The starting point under COSO ina typical implementation is the identification,definition and assessment of risks in abusiness process. In general, the personsinterviewed are business managers. Whilethese persons may be well qualified to runtheir own businesses, they do not necessarilyknow anything about risk. Yes, they canprobably come up with a long list of potentialrisk scenarios, but that’s only half the story.To know which risks are real risks themanager would also have to know the relativeprobability of each ‘risk event’ that couldaffect his or her business. After all, a tsunamiand a wire-transfer error are both risks, butwithout knowing whether a 99% level tidalwave or 99% level fat-finger error could domore damage – in the context of theirexisting control environment – they cannotknow which risk poses a greater threat. Andas it turns out there is often a majordiscrepancy between perception and reality.

The only way one can identify one’s real risksis by studying historical loss data. A riskmanager, whose job it is to know about

historical losses, is much more likely to beaware of the full range of potential risksaffecting a business (and their relativeprobabilities) than is a business manager.Having a qualified risk manager ask a businessmanager where his or her major risks are issimilar to having a doctor ask his or herpatient: to which major diseases do you thinkyou are most exposed? Some patients will knowthe answer, but most will not, which is whythey went to see their doctor in the first place.In a well-managed organisation, the riskprofessional should serve as the doctor and thebusiness manager as the patient.

For those who still believe the rightapproach is to ask business managers to self-assess the risks within their organisation’sunderlying processes, we ask what would theyhave recommended to the Governments ofIndia, Indonesia or Sri Lanka, prior to therecent tragedy, considering that tsunami riskprobably was not a recognised risk in any oftheir processes?

Another major problem with COSO is thata typical risk-assessment implementationgenerally produces a huge catalogue of risks –often in the thousands. Thus, when it comesto actually managing these risks across anorganisation, ie, determining which riskmitigation strategy is optimal, it is verydifficult to prioritise actions because withouta ‘normalised’ rank ordering of risks onecannot know which controls should be givenprecedence in implementation.

To address this problem, COSO developedthe likelihood-impact method of riskassessment. Under this approach, businessescalculate the magnitude of their risks based ona mathematical formula, where risk is equal tothe likelihood that a given event will occurmultiplied by its effect (impact), should itoccur, such that, Likelihood x Impact = Risk.

FOR those who understand theconcept of risk, as it is used in the

risk management industry, it is clear that thereis something fundamentally wrong with thisapproach. Using the COSO formula theworst-case outcome is characterised by highlikelihood and high impact; however, underthe risk management approach, the worst-caseoutcome is characterised by a low probability(low frequency) – high impact (high severity)event, such as a $1 billion dollar unauthorisedtrading loss. In fact, there is no such thing asa high likelihood (high frequency) – highimpact (high severity) event. This wouldcharacterise a risk (type of loss) that occurshundreds of times a year and each time causes

1 COSO’s objective was to help standardise procedures for enterprise risk

management by developing a conceptually sound framework providing

integrated principles, common terminology and practical

implementation guidance. For more information on COSO, visit

www.erm.coso.org

2 January 2005

billion-dollar losses. This is clearly a phantom risk. What’s evenworse is that COSO also completely understates the one areaof real risk. In summary, the COSO approach to riskassessment will tell you your risk is very high in areas whereyou have no risk, and will also tell you that you have moderaterisk in the very area your risk is of the highest order. Simplystated, COSO produces both false positives and false negatives.

The contrast is illustrated in figure 1 (right).Some advocates of COSO have suggested that this problem

only exists when the analysis is qualitative or high level. Theyargue that likelihood and impact analysis works well when theinputs are expressed in more quantitative terms, such aspercent probability and dollar magnitude. To examine thisargument, let us express it in the context of a simple businessproblem. Suppose you want to know the risk associated withyour having a car accident during the coming year. If youknow that you have a 10% chance of having an accident andyou expect that accident will cost $10,000, then you wouldcalculate your risk as follows:

Likelihood x impact = risk Risk 1: 10% x $10,000 = $1,000

But as you further consider this matter you realise that theproblem is more complex than originally perceived. After all,the 10% likelihood relates only to a $10,000 event; and thereis also a possibility, let’s say a 1% probability, that you couldhave a very bad accident, which could result in the totaldestruction of your $50,000 car. Therefore you have twopossible ways of estimating your risk, as shown below:

Likelihood x impact = risk Risk 1: 10% x $10,000 = $1,000

Risk 2: 1% x $50,000 = $500

What becomes immediately apparent is that two completelyvalid assessments can yield different risk results. In fact, uponfurther consideration, it becomes evident that the problem isstill more complex because there are, in fact, multiple‘solutions’, because there are potentially an infinite number oflikelihood and impact combinations, as shown below:

Likelihood x impact = risk Risk 1: 10% x $10,000 = $1,000

Risk 2: 1% x $50,000 = $500. . . . . . . . . . . .

Risk 999: 5% x $25,000 = $1,250 Risk 1,000: 20% x $6,000 = $1,200

From the outcome, one can clearly see that all the ‘risk-results’ are banded together (from $500–$1,250) with littledifferentiation. This is because the higher the impact the lowerthe likelihood (an incremental gain in likelihood offsets anycorresponding reduction in impact). The major differences inthe risk-results are due to the fact that the product of twofigures near their respective means is greater than the productof two figures at opposite extremes. But this is an idiosyncrasyof the arithmetic process and is not reflective of any legitimatedifference in the level of risk; in fact, the opposite is true. For

example, if we were to think of a ‘risky event’ as a 99% level(1% likelihood) event, then from the table shown above onecan see that this would correspond to a $50,000 loss. Butwhen one defines risk as the product of likelihood and impactone can see the $50,000 x 1% ($500) event would imply lessrisk than the $25,000 x 5% ($1,250) event. This wholly absurdrank-ordering clearly demonstrates that, far from improvingoperational risk management, COSO obfuscates the process ofdetermining an organisation’s true risk profile.

So what should we do now? Suppose now we were to take aweighted average of all the risk-results drawn from the tableabove? What would that answer represent? The mean of allthe risk results would equate to probability weighted severity(which seems to equate to mean severity). But if this is truethen we have a problem, because mean severity is somewhatsimilar to expected loss (mean aggregate loss), whereas therisk management industry and BIS definition of operationalrisk equates to the unexpected loss. Without knowinganything else about COSO it is clear that the meaning of riskunder COSO is altogether inconsistent with the true meaningof operational risk. By following the COSO definition of risk,one is shooting at the wrong target, one that is not even aclose approximation!

What is fundamentally wrong with the COSO-based riskassessment approach is that the question is flawed. Instead oflooking for the product of likelihood and impact we shouldbe taking as the results of this process the full set of likelihoodand impact combinations. And if we were to plot them on agraph we would get something similar to what we see infigure 2 (right).

When you connect the dots, the full set of combinationswould represent a set of points on a continuum. This isknown in actuarial science as a severity distribution.

And what does one do with this severity distribution? Will theseverity distribution give us the answer we are looking for? If welook at the 1% probability event on this distribution, will thatnot tell us our level of risk? No, not quite yet. As it turns out theseverity distribution is just one piece of this puzzle. Returning toour example, the severity distribution is a distribution of single-event losses, showing the full set of losses and correspondingprobabilities associated with a single car accident. But this is notwhat we want. We want to know our operational risk in terms ofthe total amount of money we could lose from all the caraccidents we could have in the next year. For this we also need

Risk management industry

COSO

9

6

3

6

4

2

3

2

1

Coso

Like

liho

od

Real risks

Phantom risks

COSO

n/a

n/a

n/a

COSO

9

6

3

6

4

2

3

2

1

Low (1) Med (2) High (3)

Impact

High (3)

Med (2)

Low (1)

Low (1) Med (2) High (3)

Impact

High (3)

Med (2)

Low (1) COSO

n/a

n/a

n/a

Like

liho

od

1. COSO vs reality: COSO produces false positivesand false negatives

© 2005 Incisive Media Investments Ltd. Unauthorised photocopying or facsimile distribution of this copyrighted newsletter is prohibited. All rights reserved. ISSN 1741-8291 3

Industry viewpoint COSO

www.operationalriskonline.com 4

to know how many accidents we could have inone year – or more precisely a probabilitydistribution for the number of accidents wecould have in a given year. This is known inactuarial science as a frequency distribution.

Under the risk management industry andBIS definition, operational risk – as shownabove – is described in the context of anaggregate or total loss distribution, which isa convolution (a mathematical combination)of both a frequency and a severitydistribution, where the relevant points arethe expected loss and, more importantly, theunexpected loss. The expected loss is thetotal amount of money one expects to losein a year, on average, and the unexpectedloss is the total amount of money one couldlose in a very bad year (at a specifiedconfidence level) in excess of the average.(For a technical explanation of the termsexpected and unexpected loss, please refer tothe BIS guidelines).

From the above discussion it should beevident that the risk result under thelikelihood-impact approach equates to meanseverity, which is completely unrelated to theterm risk as it is defined by the riskmanagement industry and the BIS. In fact,mean severity multiplied by mean frequencygives you the mean aggregate loss – theexpected loss. Whereas the real measure ofrisk is the unexpected aggregate loss.

The likelihood-impact analysis approach torisk assessment can be summarised as follows:it is based on a process whereby one asks thewrong people to answer the wrong question,which in any case is a flawed question,because it has an infinite number of different,but theoretically valid answers. And even ifyou were to ignore the answers and take onlythe potentially useful information from thisprocess – the full set of input pairs – youwould still only have one part of the solution.No matter how you sum it up, four wrongsdon’t make a right.

In summary, implementing COSO requiresa gargantuan effort, and, in the context ofoperational risk management, it producesspurious and misleading results. Acting onthis information may divert managers’attention from their real risks and insteadfocuses their attention on phantom risks,while at the same time providing them with afalse sense of security. Furthermore, any riskmitigation strategy based on this flawed riskinformation is likely to focus attention andresources on the wrong controls. It is highlyconceivable that this approach could lead toan intensification of controls where a business

is already over-controlled, while completelyignoring areas of control weakness. As we allknow, the consequences could be disastrous.

Operational risk management in itsmodern conceptionAn effective operational risk managementprogramme requires a sound framework. Thegoal of this framework should be to providereliable information to key decision-makers sothat they are aware of their most significant risks

as well as the quality of their correspondinginternal controls, information that will allowthem to make educated decisions whendeveloping risk management, risk mitigationand risk transfer strategies. Managingoperational risk fundamentally revolves aroundthe process of optimising the risk-controlrelationship in the context of cost-benefitanalysis. This, in turn, requires a process foraccurately monitoring (measuring) eachbusiness’ changing risk and control profile.

To accomplish this goal four things must bedone correctly. First, the risk managementdepartment must be able to provide managerswith objective information to help thembetter understand where their risks really are,not ask them to guess where their risks mightbe. Fundamentally, one cannot manage one’s

SOME MAY STILL ARGUE THAT COSO ISUSEFUL BECAUSE IT IMPROVES BUSINESS

PROCESS MANAGEMENT; EVEN IF THIS IS TRUE,IT SHOULD NEVERTHELESS BE CLEAR

THAT BUSINESS PROCESS MANAGEMENT ISNOT OPERATIONAL RISK MANAGEMENT

“ “

40-50

30-40

20-30

10-20

0-10

Probability

(10%, $10,000)

(1%, $50,000)

(5%, $25,000)

(20%, $6,000)

Impact

2. Severity distribution

operational risks without measuring one’soperational risks. It is very difficult to be ableto differentiate between major risks and minorrisks and real risks and phantom risks withoutbeing able to accurately measure these risks inthe first place. It is also impossible to developan effective risk management programmewithout knowing which risks must be dealtwith as a top priority.

Second, one must help managersunderstand how well their real risks are beingmanaged through their existing set ofcontrols, so they can know where they are

over-controlled and where they are under-controlled in the context of their overalloperational risk strategy and risk (loss)tolerance. One cannot have a zero-tolerancepolicy towards operational risk, just as onecannot institute perfect controls. Anorganisation has to be realistic in establishinga level of risk and loss tolerance.

Third, one needs to determine what level ofcontrols is appropriate after havingconducted a circumspect analysis of theassociated costs and benefits of each riskmitigation and transfer strategy.

Fourth, one needs to institute acomprehensive and fully transparentmonitoring and reporting process with built-in incentives to encourage desiredbehavioural change.

It is difficult to think of ways one couldeven begin to manage operational riskwithout having these foundational elementsin place. Best-practice calls for an integratedoperational risk measurement-managementprogramme, whereby objective, transformed(normalised) measures are used to identifylevels of risk and internal control quality. Butfor these measures to be meaningful theymust be based on reliable information,specifically: internal and external loss data,theoretically valid risk measurement andassessment, objective control self-assessment,validated risk indicators, appropriate follow-up action results, disciplined scenario analysisand well founded VAR calculation.

Can this really be done and is it practical?The answer is yes to both questions, but onlyif the underlying framework is based on soundreasoning, which must in turn be based on acomprehensive understanding of the issues.And these issues must be addressed logicallyand objectively, one issue at a time.

ConclusionsCOSO was initially conceived in the early1990s, and for a long time represented bestpractices in enterprise risk management. Thenbanks began collecting historical loss data,and we entered the dawn of a new age. Asthe process of collecting loss data becamemore widespread, thanks to the bold

5 January 2005

MANAGING OPERATIONAL RISKFUNDAMENTALLY REVOLVES AROUND THE

PROCESS OF OPTIMISING THE RISK-CONTROLRELATIONSHIP IN THE CONTEXT

OF COST-BENEFIT ANALYSIS

“ “individual

loss eventsrisk matrix for

loss datavar

calculationtotal loss

distribution

74,712,34574,603,70974,457,74574,345,95774,344,576

167,245142,456123,345113,34294,458

•

•

•

loss distributions

Frequencyof events

Severityof loss

40-50

30-40

20-30

10-20

0-10

Annual aggregate loss ($)

Mean 99th Percentile

VaR Calculator

e.g.,Monte Carlo

Simulation Engine

43210

P

P

Risk

•

•

•

Mean 99th percentileP

P

Internal FraudExternal FraudEmploymPractice

WorkplacPhysical Assets

Clients, PrBusiness P

Damage toAssets

BusinesDisruptioSystem F

Execution,& Proce

Managem

Corporate Finance

Number 36 3 25 18 36 33 150 2

Mean 35,459 52,056 3,456 45,678 56,890 56,734 1,246 89,678

Standard Deviation 5,694 8,975 3,845 4,567 7,890 3,456 245 23,543

Trading & Sales

Number 50 4 35 25 50 46 210 3

Mean 53,189 78,084 5,184 68,517 85,335 85,101 1,869 134,517

Standard Deviation 8,541 13,463 5,768 6,851 11,835 5,184 368 35,315

Retail Banking

Number 45 4 32 23 45 42 189 3

Mean 47,870 70,276 4,666 61,665 76,802 76,591 1,682 121,065

Standard Deviation 7,687 12,116 5,191 6,165 10,652 4,666 331 31,783

Commercial Banking

Number 41 3 28 20 41 37 170 2

Mean 43,083 63,248 4,199 55,499 69,121 68,932 1,514 108,959

Standard Deviation 6,918 10,905 4,672 5,549 9,586 4,199 298 28,605

Payment & Settlements

Number 37 3 26 18 37 34 153 2

Mean 38,774 56,923 3,779 49,949 62,209 62,039 1,363 98,063

Standard Deviation 6,226 9,814 4,205 4,994 8,628 3,779 268 25,744

Agency Services

Number 44 4 31 22 44 40 184 2

Mean 46,529 68,308 4,535 59,939 74,651 74,446 1,635 117,675

Standard Deviation 7,472 11,777 5,045 5,993 10,353 4,535 321 30,893

Asset Management

Number 40 3 28 20 40 36 165 2

Mean 41,876 61,477 4,081 53,945 67,186 67,002 1,472 105,908

Standard Deviation 6,725 10,599 4,541 5,394 9,318 4,081 289 27,804

Retail Brokerage

Number 48 4 33 24 48 44 198 3

Mean 50,252 73,773 4,898 64,734 80,623 80,402 1,766 127,090

Standard Deviation 8069 12719 5449 6472 11182 4898 347 33365

Insurance

Number 43 4 30 21 43 39 179 2

Mean 45,226 66,395 4,408 58,260 72,561 72,362 1,589 114,381

Standard Deviation 7,262 11,447 4,904 5,825 10,063 4,408 312 30,028

Total

Number 435 36 302 217 435 399 1,812 24

Mean 45,653 67,021 4,450 58,810 73,245 73,044 1,604 115,459

Standard Deviation 7,331 11,555 4,950 5,880 10,158 4,450 315 30,311

3.The actuarial approach

Under the BIS definition operational risk is defined as the unexpected loss

insistence of the BIS, loss data began fuellingan entirely new and more scientific way ofthinking about what came to be known asoperational risk management. It was theanalysis of this data and the issuessubsequently raised that eventually led to thedevelopment of modern operational riskmanagement as an objective discipline.

Some may still argue that COSO is usefulbecause it improves business processmanagement; even if this is true, it shouldnevertheless be clear that business processmanagement is not operational risk management.

There are also those who speak ofoperational risk management as independentfrom operational risk measurement. In ourview measurement is an integral part of themanagement process. After all, what is riskmanagement other than the mitigation ofmajor risk in the most cost-effective way. It’sdifficult to see how one can accomplish this ina large organisation without reliable measures.

Basel II was introduced to encourage banks toimprove their operational risk management. Butfollowing COSO does not improve operationalrisk management; instead it promotes phantomrisk management and does more harm thangood. Furthermore, any organisation thatapplies COSO-based risk-assessment to this endwill clearly be demonstrating to its regulators,to its investors and to the rating agencies that ithas not yet grasped even the most basicunderstanding of operational risk management– ie, operational risk management is aboutmanaging risk. In our view, far from meetingthe standards of the advanced measurementapproach or even the standardised approach, aCOSO-based operational risk managementframework may only just barely meet theminimum standards of the basic indicatorapproach – which has no standards at all!.

One of the biggest problems we face in theoperational risk management area is thatmany of those professing to be experts in thisfield actually know very little aboutoperational risk management in its modernconception. By continuing to espouse theiroutmoded and impractical views on thesubject these individuals are unknowinglydoing more harm than good, as their flawedguidance is steering the industry in thewrong direction. Based on the advice of theseindividuals many organisations have investedmillions of dollars implementing frameworksand software that they will soon discover haveneither improved their management ofoperational risk nor achieved any level of BIScompliance. While it is easy to see how manybanks could have fallen into this conceptualblack-hole, if immediate steps are not taken

to lead them out, they are likely to fall intoan even deeper abyss.

There are no shortcuts to developing acomprehensive framework for managingoperational risk. And one cannot get on theright track without confronting the difficultissues head on. If an organisation’soperational risk management framework is

not founded on fundamentally soundreasoning the entire programme willeventually unravel at the seams. An ill-conceived operational risk managementprogramme is also likely to leave anorganisation vulnerable to major operationallosses. The damage from even one major losscould be far greater than the cost ofestablishing a state-of-the-art, integratedoperational risk measurement-managementprogramme. Just think how little a verysimple global-early warning system wouldhave cost to build and maintain relative tothe lives lost and property damage thatresulted from the recent Asian tsunami.

The operational risk management industryhas been plagued by disinformation andmethodology. The industry would be muchbetter served if instead of expressing thepersonal opinions of the ‘experts’ it made aneffort to understand the issues. Without doingso, it’s hard to see how anyone could becomequalified to address this challenging problem.We certainly don’t pretend to have all theanswers, but we do think we have hit uponmany of the right questions. It is important torecognise that one can never arrive at the rightanswers without probing the most importantissues. Only by analysing and re-analysing theseissues can one begin to shed light on what maybe the right questions. Finding the answers isthe easy part. Discovering what are the rightquestions is the major challenge. OpRisk

Ali Samad-Khan is president of OpRisk Advisory. He has

eight years’ experience in operational risk manage-

ment, having previously worked at Bankers Trust,

OpRisk Analytics (which was acquired by SAS) and

PricewaterhouseCoopers, where for over three years he

headed the operational risk group within the New York

FRM practice. He can be reached at

ali.samad-khan@opriskadvisory.com

www.operationalriskonline.com 6

Industry viewpoint COSO

ONE OF THE BIGGEST PROBLEMS WEFACE IN THE OPERATIONAL RISK MANAGEMENTAREA IS THAT MANY OF THOSE PROFESSING TO

BE EXPERTS IN THIS FIELD ACTUALLY KNOWVERY LITTLE ABOUT OPERATIONAL RISK

MANAGEMENT IN ITS MODERN CONCEPTION

“ “