Similarity between models

IAE-Logo

Similarity between Modelsand the

Measurement of Model Risk

P. Hnaff

IAE ParisUniversit Paris 1 Panthon-Sorbonne

26 Nov 2012

IAE-Logo

Model Risk

Model Risk Defined

Model Risk

I Risk related to the use of the wrong model (i.e. a modelthat does not account for important risk factors)

I Risk related to poor or unstable calibration of the model

Both issues translate into :

I Inaccurate initial pricingI Inaccurate risk indicators, leading to ineffective hedging

strategy

All of this while the model correctly prices benchmarkinstruments.

IAE-Logo

Model Risk

Model Risk Defined

Some References

I An inventory of model risks : Derman (1996) Model RiskGoldman Sachs

I Dynamic hedging simulations and empirical assessment ofmodel risk : Rubinstein (1985), Bakshi, Cao and Chen(1997), Madan & Eberlein (2007).

Overall, incorporating stochastic volatility andjumps is important for pricing and internalconsistency. But for hedging, modeling stochas-tic volatility alone yields the best performance.

IAE-Logo

Model Risk

Model Risk Defined

Some References

I N. El Karoui, et al. Robustness of the Black and Scholesformula. Mathematical Finance (1998).

I T.J. Lyons. Uncertain volatility and the risk-free synthesisof derivatives. Applied Mathematical Finance (1995).

I R. Cont. Model uncertainty and its impact on the pricing ofderivative instruments, Mathematical Finance (2006).

I PH and C. Martini. Model validation : theory, practice andperspectives, The Journal of Risk Model Validation (2011).

IAE-Logo

Model Risk

Model Risk Defined

From Observation to Measurement

Basel Committee directive of July 2009 on model risk :

I Quantify model riskI Provision the risk with Tiers I capital

Needed : A measure of model riskI expressed in currency termsI standardized across asset classes and models, to allow for

meaningful comparisons.

IAE-Logo

Model Risk

Problem Statement

A definition of model risk (R. Cont [2006])I Set of benchmark instruments,

HiI payoffs,CiI the mid-market prices, C

i [Cbidi ,Caski ].

Q Set of n models, consistent with the market pricesof benchmark instruments :

Q Q EQ[Hi ] [Cbidi ,Caski ], i I (1)Upper and lower price bounds over the family of models :

pi(X ) = supj=1,...,n

EQj [X ] pi(X ) = infj=1,...,n

EQj [X ]

Risk measure :

Q = pi(X ) pi(X )

IAE-Logo

Model Risk

Problem Statement

Focus of talk

The set of models Q, consistent with the market prices ofbenchmark instruments :

Q Q EQ[Hi ] [Cbidi ,Caski ], i I (2)

How to normalize Q in order to make meaningful comparisonsbetween various models and asset classes ?

IAE-Logo

Model Risk

Problem Statement

Focus of talk

Q Q EQ[Hi ] [Cbidi ,Caski ], i I (3)Define a notion of size for Q, in order to normalize the riskmeasure.

Q ={Q|EQ[Hi ] [Cbidi ,Caski ], i I, | D(Q,P) <

}(4)

Where P is the reference model, and D(Q,P) a measure ofsimilarity between Q and P.

IAE-Logo

Similarity between models

Similarity between modelsObserve n samples Xi , i = 1, . . . ,n from a multivariatedistribution. ci : number of occurrences of Xi .

Definition (Multivariate Likelihood)Probability of observing an empirical distribution Q : qi =

cin if

model P is true.L(Q|P) = n!

ci !

pcii

0 L(Q|P) 1Average log-likelihood :

L(Q|P) = log(L(Q|P)1/n

)

IAE-Logo


Kullback-Leibler Divergence from Q to P

limn L(Q|P) =

i

qi log(qipi

)= DKL(Q|P)

eDKL(Q|P) : average likelihood of observing distribution Q whenP is the actual process.

IAE-Logo


Illustration : Bivariate Normal Density

4 0 2 4

4

02

4

Bivariate normal densities

X

Y

P

Q

0.0

0.4

0.8

Avg. likelihood of P if Q is true

A

vg.

Lik

elih

ood

0 pi 8 pi 43pi 8pi 2

FIGURE: KL Divergence of 2 bivariate normal densities, as a functionof angle between PC axis of covariance matrix.

IAE-Logo


Illustration : Bivariate Normal Density

4 0 2 4

4

02

4

Bivariate normal densities

X

Y

= 0

= 0.8

1.0 0.0 1.0

0.0

0.4

0.8

Avg likelihood of P if Q is true

A

vg.

Lik

elih

ood

FIGURE: KL Divergence of 2 bivariate normal densities, as a functionof correlation.

IAE-Logo


Similarity of model features

Model Features

Definition (Model Features)The model features X are the set of observations on the riskfactors that are needed to determine the payoff of a financialinstrument.Example :

I European option on a futures at expiry (T ) :

X = {F (T ,T )}I Option expiring at T1, on a spread between two futures

contract expiring at T1 and T2 :

X = {F (T1,T1),F (T1,T2)}

IAE-Logo



Why Focus on Model Features ?For some payoffs, different models have similar features.

dF (t ,T )F (t ,T )

= dWt (5)

Var(ln(F (T ,T ))) = 2T (6)

dF (t ,T )F (t ,T )

= e(Tt)dWt (7)

Var(ln(F (T ,T ))) =2

2

(1 e2T

)(8)

Feature F (T ,T ) is log-normal in both cases.

IAE-Logo



Why Focus on Model Features ?... but the feature F (t ,T ), t < T does not have the samevariance.

0.0 0.5 1.0 1.5 2.0

0.00

0.05

0.10

0.15

Time

Inte

gra

ted

varia

nce

BSOnefactor

FIGURE: Integrated variance of log(F (t ,T ))

IAE-Logo


Computation of KL Divergence

Calculation of KL-Divergence

Kullback-Leibler divergence between two equivalent probabilitydistributions, with observed features in Rd .

I Discrete density

D(Q,P) =i

qi log(qipi

)I Continuous density

D(Q|P) =Rd

fQ(x) log(fQ(x)fP(x)

)dx

IAE-Logo



Kernel-based density estimationLet Xi , i = 1, . . . ,n be a sample of random vectors in U Rd ,drawn from a distribution with density function f . The kerneldensity estimate of f , noted f is defined by :

f (X ) =1n

ni=1

KH(X Xi)

where :X is a feature vector in Rd

H is a symmetric, positive definite matrix of rank dK is the kernel function : a symmetric multivariate

density :

KH(X ) = H12K (H

12X )

IAE-Logo



Kernel functionUse a uniform density over a domain which is function of thelocal density of sample points.

FIGURE: Uniform density over domains of varying size

IAE-Logo



Uniform kernel density

K (x) =1xxiRxRdx cd

with :Rx radius of uniform densityd dimension of features spacecd volume of unit sphere in Rd

Define Rx as the distance to the k -th nearest neighbor of x .

IAE-Logo



KL DivergenceUsing the expression for density in terms of k-nn statistics onegets :

D(Q,P) = log( |U||V |)

+d|V |

XV

log(k (U,X )k (V ,X )

)(9)

With :d dimension of feature space

U,V sample sdrawn from Q and Pk (U,X ) distance to k -th nearest neighbor of X in sample

U.See : Boltz, Debreuve and Barlaud. High-dimensional statisticalmeasure for region-of-interest tracking. IEEE Transactions onImage Processing (2009).

IAE-Logo


Determining the optimal k


I Total Calculation time T is proportional to nk

T = C n (1 + k)I Accuracy, measured by the variance of the D(Q|P)

estimator, increases with k .

IAE-Logo



Variance of kNN density estimator

Assume a sample of size n, and let p(x) be a kNN densityestimator with k neighbors. Let V (k , x) be the volume of thesphere containing k neighors of x .

p(x) =k 1

nV (k , x)

Proposition

VAR (p(x)) =(

(k 1)(n 1)(k 2)n) 1

)p(x)2

See : D.G. Luenberger and P. Woehrmann. On kNN DensityEstimation. Working Paper Series, FINRISK.

IAE-Logo



Variance of D(Q,P)

D(Q,P) = C +d|V |

XV

[log (k (U,X )) log (k (V ,X ))] (10)

We need to estimate (ignoring correlation between k (U,X )and k (V ,X )) :

VAR log (k (U,X ))

IAE-Logo



Variance of D(Q,P)

pU(x) =k 1

nk (U, x)cd

Use :k (U, x) =

k 1npU(x)cd

andVar (f (X )) f (E(X ))2Var(X )

IAE-Logo



Variance of D(Q,P)

pU(x) =k 1

nk (U, x)cdUse :

k (U, x) =k 1

npU(x)cdand

Var (f (X )) f (E(X ))2Var(X )

IAE-Logo



Variance of D(Q,P)

I Use Luenbergers expression for Var(pU(x))I The samples points are iid (pU(x) = 1/n)I Approximate also pV (x) = 1/n

to get :

Var(k (U, x)) =(

(k 1)(n 1)(k 2)n 1

)

IAE-Logo



Variance of D(Q,P)

Finally :

Var(DL(Q,P)) =2n|U|

((k 1)(n 1)

(k 2)n 1)

IAE-Logo



Optimization of k for a given computation time

Following Luenbergers method : Compute D(Q|P) byrepeating the calculation N times for fixed n and k , with n = k .

D(Q,P) =1N

i

D(Q,P)i

Total calculation time is :

T = Nn(1 + k)

and

Var(D(Q,P)

)=

1N(k 2)

IAE-Logo



Optimization of k for a given computation time

mink

1N(k 2)

such that :

Nn(1 + k) = Tk = n

which has the solution :

k = 2 +

4 +

2

IAE-Logo



Estimation of T = C(1 + k). = 0.091

2 4 6 8 10 12 14 160.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

FIGURE: D(Q,P) computation time as a function of k

IAE-Logo



Empirical validation

2 4 6 8 10 12 140.009

0.010

0.011

0.012

0.013

0.014

0.015

0.016

0.017

FIGURE: D(Q,P) computation time Variance as a function of k .Empirical minimum is k = 8 vs. predicted k = 7

IAE-Logo



Summary : Construction of Q

1. Simulate a sample V of feature vectors X for referencemodel P.

2. Define a list {Qj} of candidate models. This list may bebuilt by perturbation of the parameters of P, or bypostulating alternate stochastic processes. There are noconstraints on the method used.

3. Simulate samples Uj of feature vectors X for each modelQj .

4. Retain models Qj such that :

EQj (Hi) [Cbidi ,Caski

] i I

IAE-Logo


A Toxicity Index

Toxicity Index at level

When using model P to price payoff X with features F . Definemodel risk as :

Q = pi(X ) pi(X )Over normalized set

Q :{Q|eD(Q|P) > ,EQ[Hi ] [Cbidi ,Caski ]}0 < 1

IAE-Logo


A Toxicity Index

Calculation Steps for Toxicity Index

Calculation of model risk associated with the use of model P forpricing derivative X .

1. Calibrate model P on a set of benchmark instruments.2. Compute the minimum and maximum value of the exotic

derivative according to each model in set Q.3. Model risk is finally computed by

Q = pi(X ) pi(X )

IAE-Logo

Numerical Illustration

data

Illustration

I Benchmark :SPX optionsI Three families of models :

1. Double Exponential Jump Diffusion (Kou & Wang 2001)2. Stochastic Volatility (Heston 1993)3. Stochastic Volatility with Jumps (Bates 1996)

I Model risk of exotic options on SPX

IAE-Logo


data

Data

FIGURE: SPX implied volatility by log moneyness and time to expiry

IAE-Logo


Model Universe

Double Exponential Jump Diffusion [Kou & Wang,2001]

dStSt

=

( 1

22)dt + Wt +

Nti=1

Yi

fY (y) = p1e1y1y0 + (1 p)2e2y1y

IAE-Logo


Model Universe

Calibration of Double Exponential Jump DiffusionModel

FIGURE: Bid/Ask implied volatility and fitted volatility.

IAE-Logo


Model Universe


FIGURE: Fit by quartile. red : worst, green : best, yellow : middle

IAE-Logo


Model Universe


FIGURE: Best quartile. Blue dot : best fit

IAE-Logo


Model Universe

Calibration of Hestons Model

dStSt

= dt +tdW st

dt = ( t)dt + tdW t

t.1104 .5045 .3591 -.74 .0272

IAE-Logo


Model Universe


FIGURE: Bid/Ask implied volatility and fitted volatility.

IAE-Logo


Model Universe


FIGURE: Fit by quartile. red : worst, green : best, yellow : middle

IAE-Logo


Model Universe


FIGURE: Best quartile. Blue dot : best fit

IAE-Logo


Model Universe

Model Risk : Asian option in Hestons model

0.550.600.650.700.750.800.850.900.951.00Average Likelyhood0

5

10

15

20

25

Price range (P=103.40)

Asian option 2Yr ATM - Heston

FIGURE: Model risk as a function of average likelihood of observingmarket data when the alternate model is true

IAE-Logo


Model Universe

Model Risk : Asian option in Bates model

0.20.30.40.50.60.70.80.9Average Likelyhood0

5

10

15

20

25

Price range (P=103.40)

Asian option 2Yr ATM - Bates

FIGURE: Model risk as a function of average likelihood of observingmarket data when the alternate model is true

IAE-Logo


Model Universe

Model Risk (Heston+Bates)

ll l l l l l l l l

l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l

0.0 0.2 0.4 0.6 0.8 1.0

2040

6080

KL Divergence

Mod

el R

isk

l

l

l

l

ll

ll

ll l

l ll l l l

l ll l l l l l l l l l l l l

l l l l l l l l

l

l

l

ll

ll l

l l l l l

l l l l l l l l l l l l l l l l l l l l l l l l l l l

l

l

l

AsianLookbackRange

FIGURE: Model risk as a function of average likelihood of observingmarket data when the alternate model is true (inverted x axis)

IAE-Logo

Conclusion

Conclusion

I We have described a procedure, inspired from the field ofimage processing, for computing the degree of similaritybetween models, independently of the definition of theseprocesses.

I We then use this measure to compute a normalizedmeasure of model risk.

I The definition of the set of alternate models Q is crucial tothe measure.

I We have presented an implementations that iscomputationally intensive but does not put any restrictionson the composition of Q.

IAE-Logo

Technical Details

Implementation details

I model calibration and asset pricing : QuantLib/pythonI QuantLib : www.quantlib.orgI Python wrapper : pyql www.github.com

I size of Q : 2000 parameter perturbations per model.I calculation of D(Q|P) :

I 20,000 scenarios per QjI KL-Divergence by k -nn algorithm : FNN R package

(www.r-project.org)

Model RiskModel Risk DefinedProblem Statement

Similarity between modelsSimilarity of model featuresComputation of KL DivergenceDetermining the optimal kA Toxicity Index

Numerical IllustrationdataModel Universe

ConclusionTechnical Details

Documents

Similarity between models