--I

DIRECT ASSESSMENT OF LOCAL ACCURACYAND PRECISION

CV. DEUTSCHDepartment of Petroieum EngineeringStanford University, Stanford, CA 95305-2220

Abstract Geostatistical techniques are used to build probabilistic modelsof uncertainty about unknown true values. The goodness of these proba-bilistic models can be assessed by meas1n'es related to the two concepts ofaccuracy and precision.

A probability distribution is said to be accurate if the 1U% symmetricprobability interval [PI) contains the true value 10% {or more] of the time,the 20% Pl contains the true value 213% [or more] of the time, and so onfor increasingly wide probability intervals. To directly assess this measureof accuracy we can use the leave-one-out" cross validation approach or thekeep-sonic-back" jackknife approach. For a given probabilistic model, theidea is to build distributions of uncertainty at multiple locations where thetrue values are known. Accuracy may then be judged by counting the num-ber of times the true values actually fall within xed probability intervals.Accuracy could be quantied for different probabilistic models (Gaussian,Indicator, Object-based, and Iterative/Annealing) and different in1plemen-tation optio11s.

Precision is a measure of the narrowness of the distribution. Precisionis only dened for accurate probability distributions; without accuracy, aconstant vaiue or Dirac distribution would have the ultimate precision. Aprobability distribution where the 90% PI contains the true value 99% ofthe time is accurate but not precise. Optimal precision is when the BUCK: PIcontains the true value exactly 99% of the time.

1. Introduction

The basic paradigm of geostatistics is to model the uncertainty about anyunknown value e as a random variable [RV] Z characterised by a specicprobability distribution. The unknown value z could be a global parametersuch as the economic ultimate recovery of an oil eld or a local attribute

115

@i"1 --1""-1Benji and NA. Schojieid {eds}, Georronlnics Woiiongong '96, Voiione I, 115-125.

9'97 Kinwer Academic Pubiishers. Printed in the N:-rirerionds.

alessandroHighlight

alessandroHighlight

alessandroHighlight

alessandroHighlight

1 16 Ch. DEUTSCH

such as the porosity at a. specic location u. Often, global parameters arethe result of a complex 11on-linear process that may be simulated with rela-tively costly flow simulation of a high resolution numerical model. There aremany ways of building these numerical models: (1) parametric approachessuch as the multiGaussian model, (2] parameter-rich or distribution-freemodels such as provided by the indicator formalism, [3] object-based al-gorithms where objects are stochastically positioned in space, or (4) viasimulated annealing. Moreover, each approach calls for many subjectiveimplementation decisions related to specic computer coding, variograrnmeasures of continuity, search strategies, sise distributions, and conver-gence parameters.

This paper is concerned with checking the goodness of a probabilisticmodel, comparing it to alternative models, and perhaps ne-tuning theparameters of a chosen model. Before proceeding further, we must recognisethat probabilistic models may only be checked by: [1] the data used formodeling, [2] some data held back from the beginning, or [3] additionalknowledge of the physics of the phenomena, e.g., information that wouldallow classifying some realizations as implausible. In this paper, the leave-one-out cross validation and the keep-some-back jackknife approachsare considered. As for point [3], the goal is to incorporate such knowledgeinto the probabilistic model as soft or secondary data.

The goodness of a probabilistic model may be checked by its accu-racy and precision. In general, accuracy refers to the ultimate excellence ofthe data or computed results, e.g., conformity to truth or to a standard.Precision refers to the repeatability or renement (signicant gures] of ameasurement or computed result.

For clarity and in the context of evaluating the goodness of a prob-abilistic model, we propose specic denitions of accuracy and precision.For a probability distribution, accuracy and precision are based on the ac-tual fraction of true values falling within symmetric probability intervals ofvarying width p:

~ A probability distribution is accurate if the fraction of true valuesfalling in the p interval exceeds p for all p in [0,1].

- The precision of an accurate probability distribution is measured bythe closeness of the fraction of true values to p for all p in [[1,1].

A procedure for the direct assessment of local accuracy and precision isnow described.

i

in

alessandroHighlight

alessandroHighlight

alessandroHighlight

alessandroHighlight

alessandroHighlight

alessandroHighlight

alessandroHighlight

alessandroRectangle

DIRECT ASSESSMENT UF LCICAL ACCURACY AND PRECISION 11'?

2. Assessing Local Accuracy2.1. DEFINITIONS

Consider the leave-oneout cross validation approach. Values at n datalocations u,;,i = 1,... ,n, are simulated one at a time using the remainingn 1 data values, i.e., leaving out the data value e[u;]. Stochastic sim-ulation leads to L [L large] stochastic realizations {alli'[u,],l = 1,. . . ,L}at each left out data location. These L realizations provide a model of theconditional cumulative distribution function {ccdfjz

F[u,;;:-:ln[u,]] = Prob{Z[u.,-] a|n[u,-]} [1]

where n[u,-] is the set of n data minus the data at location u,-. These localccdf models may be [1] derived from a set of L realizations, [2] calculateddirectly from indicator-kriging, or [3] dened by a Gaussian mean, variance,and transformation.

The probabilities associated to the true values .z[u,-],i = 1,... ,n arecalculated from the previous ccdf as:

F[u";; a[u,]|n[u,]], -i = I, . . .,n

For example, if the true value at location u.,- is at the median of the simulatedvalues then .F'[u,-; :[u,]|n[u;]] would be 0.5.

Consider a range of syrmnetric p-probability intervals [Pls], say, thecentiles 0.01 to 0.99 in increments of 0.01. The symmetric p-PI is denedby corresponding lower and upper probability values:

__ l1Pl C, _ u+p1plow " 3-I1 Pupj!;r T

For example, for p 0.9, pg.,,_., = 0.05 and p,,,,,, = 0.95.Next, dene an indicator function .[ui;p] at each l cation u, as-D .

H __ ]__{ 1: ii'F(11i~=(111'}|'-'1(11="l]E(aia..,P..,a] (.2)ullp _ 0 otherwise

The average of {[11,-;p] over the n locations u,:

Z:G= Ilsa)

l 13 C.'*v'. DEUTSCH41.?-4-1.

way to check this assessment of accuracy is to cross plot E [p] versus p andsee that all of the points fall above or on the 45" line. This plot is referredto as an occurncy plot, see Figure 1 and example hereafter.

An equivalent way to calculate m is to dene the indicator u; p] =1 ifs-(11) E [F'1(v;aia..|n(l).F"(;Pa-pliullli thsrwis air) = itThat is, dene the indicator on z values instead of p values. The advantageof dening the indicator on probability values as in relation [2] is thatsignicantly fewer quantiles must be calculated because ppm, and pupy, areglobal parameters independent of the location 11,-. For example, with n =1000 and np = 90 centiles we calculate either 1000 probabilities or 99 - 2 -1000 = 198000 quantile values for the same result.

2.2. A FIRST EXAMPLE

Consider n = 1000 independent uniformly distributed random numberss.-,*i = 1, . . . ,n drawn from the ACORN generator [Wikramaratna 1989].For each outcome, the correct corresponding ccdf of type [1] is a uniformdistribution in [0,1]:

0.0, for z 0F[i,-,s|[n[i]]] ={ z, for 0 < z 1. for s __

The probability of each true value s,;,i = 1,. . . , n is that value itself, i.e.,

F[i;s;|{n[i]]]=.e,-, i=1,...,n [4]

The average indicator function [p] was calculated for each centile p =I-,%,j = 1, . . . ,90 and the resulting accuracy plot is shown o11 the top ofFigure 1. Note that the plot is very close to a 45" line indicating that thisccdf model is both accurate and precise.

The middle row graphs of Figure 1 illustrate the accuracy plot if theccdfs were expected to be uniform between -0.5 and 1.5. The simulateddistributions are still accurate, because the probability intervals are suffi-ciently wide, but not precise. The 0.5 PI of this ccdf model contains all ofthe true values, therefore, -f[0.5] = 1.0.

The lower graphs on Figure 1 show the results when the ccdfs are mod-eled to be triangular distributions between 0 and 1 with mode at 0.5. Thesimulated distributions are no longer accurate. In fact the 0.5 PI only con-tains 25% of the true values.

2.3. QUANTITATIVE MEASURES OF ACCURACY AND PRECISION-,.,_i.,--,-

A distribution is accurate when [p] Z p. To develop a measure of accuracy,an indicator fimction n[p] is dened for each probability interval p, p E

-_

alessandroHighlight

alessandroHighlight

-1-.

DIRECT ASSESSMENT DF LOCAL ACCURACY AND PRECISIDN 1 151

1.0_

0.0..

0.5..tivi I f

0.-ii

;1 0 1 2-

i||ii||i|||iL||||i|0.0 0-2 0. 0.5 0.0 1.0

probability interval i- p

1.1:- _' .-

1IiI1LLu.|J vi,-ii

E

55*.

1-H

0.3

0.E__

anvs 1 .-" i i- yr . ,

* I | i i I

0' 1 I | I 1 I | 1 I I l I I I I I I I I

0.0 0.2 0.4 0.0 0-0 1.0probability interval - p

L.-_I

'5,"

ii _n Q I'\J0.2

1-0

0.0

__p-Etivi f .- - - - - . . ..

0.4 +I I0 II I

as -1 1 1 2an , H ,

n Cl GT,,,,l nh Q F-Cmi PIm _ *1aprobability interval - p

Figure 1. Ai:cura.cy plots for n = 1000 inrlependeiit iuiiforrnly rlistriliiiteil in [11, 1]ralidom mnuhers a,-,i = 1.. . . ,n. The plots curresponrl tn [1] a correct uni ornidistribution between 0 and 1, [2] a uniform ilistrihution between -0.5 and 1.5. and13} a trii-uigular ilisi.rilmtious iietween 0 and 1 with mode at 0.5.

'1

A

F1-.

12:] c.v. DEUTSCH

[0, 1]:-I1l'I'liI-Ias-{ 5; is

where o.[p] is an indicator variable set to 1 if the distribution is accurateat p and 0 if not. A summary measure of accuracy is dened as:

A = [1 voids

DIRECT ASSESSMENT OF LOCAL ACCURACY AND FRECISIDN 121

CGDF Model Accuracy A I"re-cisiou P Gooilness G

u1.LifIJ1'I.L'i E [0, 1] 0.40-*1 0.001 0.005uniform E [0.5,1.5] 1.000 0.000 0.'i'00triangular E [(1.1] {uioile at 0.5] 0.000 0.000 0.640

TABLE 1. Summary of results shown on Figure 1.

2.4. ANOTHER WAY OF LOOKING AT ACCURACY AND PRECISION

The distribution of the random variable 1"[u,-] = F[u,-;a[u,][n[u,-]], i =1,. . . ,n, denition 4, will now be considered. For the probabilistic modelto be accurate and precise, the 1*-marginal distribution f[p'],p" E [0,1]must be uniform in Indeed, from the denition of an accurate andprecise distribution = p, tip E [0,1]]:

if(a)da = FEE = PH-is E 10. ll => Ha) = 1-0. Va E 10.1]

Thus, when the interval [0,1] is divided into N subsets one should getapproximately the same number of y-values in each subset for an accurateand precise distribution. A chisquare test could be used to check how closethat y-distribution actually is to uniform. Figure 2 shows the histogramsfor the example illustrated on Figure 1.

The histogram of y[u,;],i = 1,. . . ,n should be considered together withthe accuracy plot and summary statistics. There are pathological caseswhere incorrect [aysmmetric] distributions of uncertainty lead to a cor-rect accuracy plot. This histogram identies those situations of systematicoverestimation or underestimation.

2.5. UNCERTAINTY

Good probabilistic models must be both accurate and precise. There isanother aspect of probabilistic modeling, however, that must be considered:the spread or uncertainty represented by the distributions. For example, twodifferent probabilistic models may be equally accurate and precise [G 2 1.0]and yet we would prefer the model that has the least uncertainty. Subject tothe constraints of accuracy and precision we want to account for all relevantdata to reduce uncertainty. For example, as new data becomes available ourprobabilistic model may remain equally good [G = 1.0] and yet there isless uncertainty.

alessandroHighlight

I" "'1

112 CH. DEUTSCH

Gnu D1

_ I.0.1125 p . J. -

[_'||_ I II It = __ -3 II

Frequency PD E

_.,_| __ 1| .ms ' _L F 5 _

| I I; H

.~'; _ 1|:| |__ | -l_an-us " 1,- ' , :@..'I.' "'i H "i'--.|ir__l_r|__JI|':| :

0.0 u.2 n.4 as n.a 1.0 ;quanti e 1|

Cni?U115 "'1

_ Fl ' .1".EH34 - "LI .."-*

"L-Fl "-r' -I| .-an':--

Frequency

P8.'!EH12 - = '-'

_.|__I101

DIE} | | 1 F | :l E I 11-:-| I | I | | I r

EI.l.'] D2 U.-1 '15 {LB 1 .0quanti a

a-w-I:1:-.10.}-I'.'|' |-1-!.|EI.lJB_'

Frequency

n.|:=ej 1:

l.'J.U-4..* !

|j_[}2 _ |I.|I I 1

ml ||||m|l||||||||||.|||l|||||||||ln||||||||||0.0 0.2 0.4 as ma 1.u

quantila

Figum 2. Histug1'eu11:-1 uf ;r;[1.1;].i = 1, . . . .,n f-:.u' the |;h1'eP. H1-Ealulries illllstratmi nuFigure 1.

_ - - -1

DIRECT ASSESSMENT UF LUCAL ACCURACY AND PRECISION 123

If we systematically tool-: the global histogram as the model of uncer-tainty at each location we would nd that this conservative probabilisticmodel is both accurate and precise. The uncertainty, however, is large. Un-certainty increases as the spread of the probability distribution increasesand could be quantied by measures such as entropy, interquartile range,D1 "u"Et1'I EIIICE.

The measure of entropy has attractive features from the perspectiveof information theory or statistical mechanics. The interquartile range hasattractive features from the perspective of robust statistics. The variance ispreferred here because of its simplicity and wide acceptance as a measureof spread. The uncertainty of a probabilistic model may be dened as theaverage conditional variance of all locations in the area of interest:

U1N 2 -} 9_Efg'5"luI

where there are N locations of interest u,-,i = 1,...,N with each varianceozln,-) calculated from the local ccdf F(u,; z|n(n,)).

The uncertainty statistics for the simple example presented in Figure 1are (H1333, 0.3333, 0.0424, respectively. Although the triangular distribu-tion has the least uncertainty, it is not recommended since it is neitheraccurate nor precise, see Figure 1 and Table 1. To be legitimate, uncer-tainty can not be articially reduced at the expense of accuracy.

3. Reservoir Case Study

To further illustrate the direct assessment of accuracy and precision, con-sider the F-1.moco" data consisting of T4 well data related to a West Texascarbonate reservoir. Figure 3 shows a location map of the T4 well data anda histogram of the vertically averaged porosity for the main reservoir layerof interest.

Sequential Gaussian simulation (Isaaks 199i], Deutsch Ea Journel 1992]was performed at each of the ?4 well locations using the leave-one-out"cross-validation approach. Two alternative semivariogram models for thenormal score transform of porosity were considered: [1] that built from thecomplete 3-D set of porosity data, and [2] that built from the T4 verticallyaveraged values. These two different variogram models result in two differ-ent sets of ccdfs. Figures 4 and 5 show the two semivariogram models andthe corresponding accuracy plots, A, P, and G scores. The semivariogrambuilt from the vertically averaged values leads to better ccdf models. Also,that semivariogram model leads to a lesser uncertainty measure U = El.-433down from U = ll.-"5'? for the rst semivariogram model.

alessandroHighlight

alessandroHighlight

ima-

124 C."v". DEUTSCH

1--'.U"-I.-I CI "-

oeb col-coo. * .IIBIIIIIII.Ii |.'..:

c *

|c4cc|su|ny1c1-unisexitF IB . |II'lIlfiils! in

.53,-|. I. II - ...':,. ittitii itE',_. illii-2:

-_ .. |

' B 'e o I c I .,,__ , = -_ ' i

I. E . . ' . . .. i '|e ,.' I

Q El C: If I I ' ' 'ill Li ll ill TL!

I- E I I I I v-nuns-;:.um,||urus|ny.ai

FiI1IT 3. Location map and histogram of T4 well data.LI

nLII _ _ _ _ _ . _ . _ . . - . . . - - _ _.- ll I

II

In ' '_ . u '

ml . ' llll 'Y HIll-1| Q I '|.Dl]'iil

F =: U15?11.: H i G = 'l].BTH

In I f T " F1 I I l . I I I I I | l | | | | l l | I. In I |-F-I'I-f -|- 1 + -IiI |- 1- -Ii I- -i -Ii

I1 III}. IND. Ii. -Ii. FIN. U.-U Ill-I Ill I-LI H-.I LEI

anus: Dmhablly interval - p

Figure 4'. Horisontal normal scores seluivariogrutn from the 3-D porosity data[calculated using stratigraphic coordinates} and the corresponding accuracy plotconsidering cross validation with the normal scores of the T4 well data on Figure 3.

The page limitation prevents full presentation of the case study. Unthe basis of work not shown, considering seismic data as soft informationfor simulation of porosity with a carefully t linear model of coregional-isation allows further reduction of uncertainty while maintaining accurateand precise distributions. Also, indicator simulation and annealing-basedsimulation were shown to perform slightly better than Gaussian-based sim-ulation because more spatial information is considered through multipleindicator variograms.

4. Conclusions

We have developed the idea of directly checking local accuracy and precisionthrough cross validation. A probabilistic model of uncertainty is good ifit is both accurate and precise. In addition, the uncertainty should be as

DIRECT ASSESSMENT OF LOCAL ACCURACY AND PRECISION I25

1.0-

13% 0.1:]

.' 0.0- ''* tin) _

Y I 04; ',1 _ - A = aces'4'-I _ P = U15?1 *3 e = 0.952

m

-4- + 1- |-- --|-- --i-+-| l | l | I l l | I | | |

1 L lmu. N. Ii. MO. GI DJ I.-I ll Ill Ll

Dianne: Dfibilily iI"li'IIl"li'El - II

Figure 5. Hoi'ii:ontal iioriiial scores seinivariograni from the 74 vertically averagedporosity and the corresponding accuracy plot considering cross validation with thenormal scores of the 74 well data on Figure 3.

small as possible while preserving accuracy and precision. The accuracyplot combined with measures of accuracy (A), precision (P), goodness (G),and uncertainty (U) are useful summaries to quantify the goodness of aprobabilistic model.

The main uses for the diagnostic tools presented here are: [1] detectingimplementation errors, (2) quantifying uncertainty, (3) comparing differentsimulation algorithms (e.g., Gaussian-based algorithms versus indicator-based algorithms versus simulated annealing-based algorithms), and (4)ne-tuning the parameters of any particular probabilistic model (e.g., thevariogram model used).

These tools provide basic checks, i.e., necessary but not sufficient teststhat any reasonable stochastic simulation algorithm should pass. They donot assess the multivariate properties of the simulation. Care is neededto ensure that features that impact the ultimate prediction and decisionmaking, such as continuity of extreme values, are adequately representedin the model.

ReferencesDcutscli. C. iii. Joiirncl. A. (1992). GSLIB: Ger.r.rlriti.vtir:ril Sfliilflfrf Lil-r'riry rind .1".w:r".'i

Gnirie, Uxfiiril University Press, New York.lsaaks, E. [199UlI. The Ajijilicritiori of Mriritr: C'rii"lo Mrithrirls to Hie r-lririiysis of Sgiiifiriil-y

C'or'rr.'l0ir:ri Brita. PhD thesis, Stanford University, Stiinforil. CA.Wikraiiniratiia, R. (1989). ACURN - a new method for gciicratiiig seqiii:iicc.-s of uiiiformly

ilistrilnitcd pseudo-raiidoiii iiuinlicrs, Jriuiviril of C'rir:ipiitriti'ririril Pfiy.-:ir.~.-i 83: 16---31.

alessandroHighlight

1. Introduction2. Assessing Local Accuracy3. Reservoir Case Study4. Conclusions