Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software

IntroductionMathematical Background & Concepts

Implementation of a MC SimulationPost-processing and Analysis of a Simulation

Discussion

Review of Methodology and Rationale ofMonte Carlo Simulation

Application to Metrology with Open Source Software

Vishal [email protected]

Mechanical Metrology GroupNational Metrology Institute of South Africa

November 6, 2008

Vishal Ramnath [email protected] NMISA-08-0121: Review of Methodology & Rationale of MC Simulation



Discussion

Overview of Presentation

IntroductionReview of GUM MethodologyReview of Monte Carlo Methodology

Mathematical Background & Concepts

Implementation of a MC SimulationDeveloping a Mathematical ModelAssigning Uncertainties and PDF’s to the ModelIllustrative Mass Unc Example

Post-processing and Analysis of a SimulationAnalysing and Understanding the DataReporting Results in GUM Terms

Discussion




Discussion

Review of GUM MethodologyReview of Monte Carlo Methodology

Introduction

This is an introductory presentation to convey the basic ideas behindthe mechanics of the Monte Carlo technique as applied to metrologymeasurement uncertainty problems.The rationale for the need to understand and implement Monte Carlo(MC) techniques in the context of metrology is that with the advanceof science and technology more accurate measurements are forvarious reasons increasingly necessary in many economies and MCsimulations present the most accurate and readily available numericaltechnology to solve such challenges taking into account certainlimitations in existing approaches such as the well known ISO Guideto Uncertainty in Measurement (GUM).




Discussion


GUM Review 1 - Essential Information Needed

For an input quantity xi in the GUM framework three quantities are neededviz.

◮ the expectation of xi which is just the estimate of this input

◮ the standard deviation of xi which is the standard deviation of this inputσ(xi)

◮ the corresponding degrees of freedom νi associated with xi

If there are dependencies with other input quantities xj , j 6= i then covariancesare also required:in the case of two inputs xi and xj the covariance u(xi , xj) and correlationcoefficient r(xi , xj) are related by

(1) u(xi , xj) = r(xi , xj)u(xi)u(xj),−1 6 r(xi , xj) 6 1




Discussion


GUM Review 1 cont. - Essential Information Needed

(2) u(xi , xj) =1

q(q − 1)

q∑

k=1

(xi,k − xi)(xj,k − xj)

◮ If r(xi , xj) = 0 then there is no correlation and if r(xi , xj) ≈ 1 thenthere is strong correlation

◮ Most uncertainty calculations assume r(xi , xj) ≈ 0 for simplicityi.e. no correlation between input quantities but if necessarycorrelation can be explicitly incorporated into calculations

◮ In the case of correlation between more than two variables e.g.xi , xj , xk with i 6= j 6= k then a covariance matrix and not a scalarcorrelation coefficient is required




Discussion


GUM Review 1 cont. - Essential Information Needed

The GUM approach is the propagation of uncertainties associatedwith input quantities in a measurement model to provide estimates ofthe model output quantity (univariate) or quantities (multivariate) Itshould be noted that:

◮ Within the framework of the GUM a mathematical model ofthe measurand is a prerequisite in order to implement anuncertainty calculation




Discussion


GUM Review 2 - Standard Calculation Technique

y = f (x1, . . . , xn) ⇐ math model(3a)

ci =∂f∂xi

⇐ sens coeff(3b)

u2(y) =

n∑

i=1

[

∂f∂xi

]2

u2(xi) ⇐ std unc(3c)

u4(y)

νeff=

n∑

i=1

c4i u4(xi)

νi⇐ calc eff deg freedom(3d)

k ⇔∫ t

−t

Γ[νeff +12 ]

√πνeff Γ[νeff

2 ]

(

1 +u2

νeff

)−νeff +1

2

du = p

⇑coverage factor(3e)




Discussion


GUM Review 2 - Brief Comment on Sensitivity coeff’s

Various possibilities will arise in practise with real inputs x ∈ Rn:

◮ univariate, explicit, real model or multivariate, explicit, real model◮ univariate, implicit, real model or multivariate, implicit, real model

In the case of an implicit model i.e. where an explicit functionalrelationship between the input and output(s) is not known thenadditional matrix algebraic manipulations are necessary and suchmanipulations require the solution of linear systems of equationsIn addition as per the above but with complex models i.e. with x ∈ Cn

require analogous sensitivity conterparts where now partialderivatives for both the real and imaginary components of an inputare necessary




Discussion


GUM Review 3 - Assumptions & Limitations of theGUM

There are three chief requirements that limit the applicability of theGUM:

◮ the non-linearity for the measurand as modelled by a functionf (x) must be insignificant [GUM Clause 5.1.2]

◮ the Central Limit Theorem must be assumed to apply for themodel of the measurand i.e. the PDF for the output must beGaussian (alternately in terms of a t-distribution) [GUM ClausesG.2.1 and G.6.6] and

◮ the necessary conditions for a Welch-Satterthwaite formula tocalculate the effective degrees of freedom must apply [GUMClause G.4.2]




Discussion


GUM Review 4 - When & how will the GUM not work

As per the three requirements for the GUM to adequately apply it will then byimplication not function adequately when:

◮ non-linearities in the model are significant - when the model can notaccurately be represented by a first order Taylor series expansion thenthe probability distribution of the measurand can similarly not beaccurately represented in terms of the convolution integral of thedistributions of the input quantities;

◮ the conditions for the validity of the Central Limit Theorem as applicableto the measurement model are not sufficiently strong - theoretically theCLT predicts a Gaussian distribution for the measurand only in the limitas the number of input quantities increases i.e. it is not necessarily atrue or accurate representation of the measurand PDF for a small finitenumber of input parameters




Discussion


GUM Review 4 cont. - When & how will the GUM notwork

As per the three requirements for the GUM to adequately apply it will then byimplication not function adequately when:

◮ the conditions for the validity of the Welch-Satterthwaite formula are notpresent i.e. in the case for a univariate, real output y where the inputquantities x are not mutually independent - the GUM does not state howνeff is to be calculated when the input quantities are correlated† i.e. eventhough correlation coefficients (alternately covariance matrix) may bemodelled / calculated from experimental data there is no methodology toestimate νeff and hence a corresponding coverage factor k unless oneassumes u(xi , xj) ≈ 0∀i 6= j†Correlation coefficients r(xi , xj) are used for calculating the combinedstandard uncertainty uc, cf. U = k(p, νeff ) · uc




Discussion


Review of GUM Methodology - Background to why MCis being utilized

With the three requirements for the GUM to adequately apply andwith limitations and lack of applicability that arises when theseconditions are not met for many practical measurement models ofreal measurement systems and standards we then see that:

◮ Due to the sometimes restrictive conditions on the limitations andapplicability of the GUM that many NMI’s and possibly evenindustrial metrology laboratories are starting to investigate andimplement Monte Carlo simulations for their own laboratorystandards and in inter-comparisons for e.g. CMC justifications




Discussion


Review of MC Methodology

In a measurement uncertainty analysis one is concerned withpropagating uncertainties from inputs to outputs and the GUMpropogates uncertainties from a first order approximation from amodel of a measurement system with the assumption that themeasurand has a Gaussian distribution whilst a MC method directlypropogates PDF information without any prior assumptions.A MC method can be accurately described as a statistical samplingtechnique.




Discussion


Outline of MC Process Used for a Univariate Model

1. select M Monte Carlo trials

2. generate M vectors by sampling from the PDF’s for the set of Ninput quantities

3. for each vector evaluate the model to give the correspondingvalue of the output quantity

4. calculate the estimate of the output quantity i.e. the measurandand its associated standard uncertainty

5. use the simulation data to build a discrete representation of thedistribution function

6. use the distribution function to calculate the coverage interval forthe measurand




Discussion


For continous random variables recall: f (x) is a PDF for a randomvariable x if (i) f (x) > 0∀x ∈ R, (ii)

∫∞

−∞f (x)dx = 1, (iii)

P(a < X < b) =∫ b

a f (x)dxThe corresponding cumulative distribution function isF (x) =

∫ x−∞ f (t)dt

For MC work we will use the following nomenclature: Let the PDF forinput Xi be gi(ξi ), the PDF for the measurand Y be g(η), andG(η) =

∫ η

−∞g(z)dz denote the distribution function corresponding to

g(η)




Discussion


There are additional mathematical definitions and terminology thatare necessary to more fully understand how a Monte Carlo simulationworks in practice but for our purposes we will not delve too deeply intothe finer details in this presentation and rather concentrate on someof the more practical considerations that are needed if one wishes toundertake and implement a MC measurement uncertainty analysis




Discussion

Developing a Mathematical ModelAssigning Uncertainties and PDF’s to the ModelIllustrative Mass Unc Example

Implementation of a MC Simulation

A few preliminary points should be noted:◮ a good random number generator is essential for reliable work -

the MS Excel RNG is not satisfactory and will introduce problems◮ the software code used should allow definition of a model and

the parameters defining the PDF’s for the input quantities◮ symmetry in the output PDF is not assumed◮ no derivatives are required◮ there is an avoidance of the concept of effective degrees of

freedom◮ sensitivity coefficients are not calculated or needed: possible to

modify post-processing to calc a sensitivity coeff




Discussion


Developing a Mathematical Model

When developing a mathematical model for a MC simulation it shouldbe noted that there is no distinction between Type A and Type Buncertainty contributors and that the measurand is simply defined interms of a function e.g.

(4) y = f (x1, x2, . . . , xn)

where the inputs x1, . . . , xn directly model and describe the influenceif an input is changed - it is this variation/change in the inputparameters that is propogated through the model and henceinfluences the output expressible as an uncertainty.




Discussion


Developing a Mathematical Model

A MC simulation is therefore different from the GUM in the sense thatone has to have a full and complete understanding of the entiremeasurement system under investigation and one can not simplyassign an input uncertainty and a unity sensitivity coefficient withoutadequate justification




Discussion


Assigning Uncertainties and PDF’s to the Model

In a MC simulation the PDF’s of the input quantities g1(ξ1), . . . , gN(ξN)are required and the following options are possible:

◮ if the input xi is a quantity that has been measured/calibratedthen it will have a measurement/calibration certificate that wasdone with the GUM so the quoted value is the mean µi and itsstandard uncertainty is obtained by dividing the expandeduncertainty by the applicable coverage factor - this is enoughinformation to infer its PDF since the GUM result is alwaysexpressed in terms of a Gaussian PDF which is completelydefined in terms of µ and σ

◮ similarly as above for rectangular (particularly if estimated frome.g. literature), triangular, U shaped distributions etc.




Discussion


Assigning Uncertainties and PDF’s to the Model cont.

◮ a statistical analysis based on relevant theory may indicate aninputs PDF e.g. dimensional cosine terms and one then just hasto estimate some parameters to fully define the PDF

◮ there may be discrete numerical data for an input parameterwhich means that input parameter’s PDF can be built up with itsfrequency data (like a histogram)




Discussion


Mass Meas Unc Example to Contrast the GUM vs. MC

Example (Math Model Explanation & Details)

Consider a mass calibration toillustrate the principles of a MCsimulation and some of thedifferences with the standard GUMapproach with a schematicillustration of the measurementsystem




Discussion





mW

δmR

mR

pivot




Discussion





mW

δmR

mR

pivot

The principle of a mass measurement with an equal arm force balance is of abalance of moments generated which by applying a force balance withArchimedes’ principle for buoyancy effects we then get




Discussion





mW

δmR

mR

pivot

The principle of a mass measurement with an equal arm force balance is of abalance of moments generated which by applying a force balance withArchimedes’ principle for buoyancy effects we then get

(5) mW g − ρairmW

ρWg = (mR + δmR)g − ρair

mR + δmR

ρRg




Discussion


Developing the Meas Model

Example (Mass example cont.)

Rearranging




Discussion


Developing the Meas Model

Example (Mass example cont.)

Rearranging

mW

(

1 − ρair

ρW

)

= (mR + δmR)

(

1 − ρair

ρR

)

Symbol DescriptionmW mass of weight piece WmR mass of reference weight piece RδmR small test mass to add onto mR to achieve force balanceρi mass density with i respectively that of the weight W

air medium air or that of the reference’s density R




Discussion


Developing the Meas Model cont.

Example (Expressing model with lab specific stds)Mass is an invariant quantity as any physical parameter in metrology but thevalues quoted will differ depending on the lab system in use e.g. massis“heavier” in air than in water which is why astronauts train under water tosimulate weightlessness:




Discussion


Developing the Meas Model cont.

Example (Expressing model with lab specific stds)Mass is an invariant quantity as any physical parameter in metrology but thevalues quoted will differ depending on the lab system in use e.g. massis“heavier” in air than in water which is why astronauts train under water tosimulate weightlessness:In mass metrology laboratories use the concept of “conventional mass”

Definition (Conventional Mass)The conventional mass mW ,c of a weight W is the apparent mass of ahypothetical weight of density ρW 0 = 8000 kg.m−3 that balances W in airwhen the air density is ρair0 = 1.2 kg.m−3 i.e.mW (1 − ρair0/ρW ) = mW ,c (1 − ρair0/ρW 0)

Fact (Usage of Conventional Mass)Conventional mass is simply a measurement tool used to incorporate theinvariance of inertial mass i.e. “compare apples with apples”




Discussion


Final Measurement Model

We utilize the Open Source computer algebra system Maxima tosimplify our calculations due to the substitions that the conventionalmass introduces.The reason for why one may prefer to use a CAS is in the cases whenfairly complicated expressions and hand calculations carry the risk oflengthy time and error generation.




Discussion


Review of calculas theory - cf. GUM assumptionsRecall that from multi-variable calculas that Taylor series expansionsfor the case of a single variable

(6)

f (x) = f (a)+f ′(a)

1!(x−a)+

f ′′(a)

2!(x−a)2+· · ·+ f (n−1)(a)

(n − 1)!(x−a)n−1+Rn(x)

can be generalized to the case for multiple variables. An example fortwo variables would be

f (x , y) = f (a, b) +∂f∂x

(a, b)(x − a) +∂f∂y

(a, b)(y − b)

+12!

[

∂2f∂x2 (a, b)(x − a)2 + 2

∂2f∂x∂y

(a, b)(x − a)(y − b)

+∂2f∂y2 (a, b)(y − b)2

]

+ · · ·




Discussion


Approx. of functions - linearized in 1D

Let f (x) = exp(x2) and expand about a = 1

1

2

3

4

5

6

7

8

9

-1.5 -1 -0.5 0 0.5 1 1.5x-axis

y-a

xis

f = exp(x2)

function ex2

1st e + 2e(x − 1)2nd e + 2e(x − 1) + 3e(x − 1)2




Discussion


Approx. of functions - linearized in 2DLet f (x) = exp(x2 + y2) and expand about a = [1, 1]T

Linear approx: L(f ) = 2e2(y − 1) + 2e2(x − 1) + e2

3d plot

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 0.96

0.97 0.98

0.99 1 1.01

1.02 1.03

1.04 1.05

5.5 6

6.5 7

7.5 8

8.5 9

z1

x1

y1

z1

5.5 6

6.5

7

7.5

8

8.5

9

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05

x1

0.95

0.96

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

y1

-3

-2.5

-2

-1.5

-1

-0.5

0

% error




Discussion




3d plot

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 0.96

0.97 0.98

0.99 1 1.01

1.02 1.03

1.04 1.05

5.5 6

6.5 7

7.5 8

8.5 9

z1

x1

y1

z1

5.5 6

6.5

7

7.5

8

8.5

9

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05

x1

0.95

0.96

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

y1

-3

-2.5

-2

-1.5

-1

-0.5

0

% error

Quadratic approx:

f = L(f )+ 12 [6e2(y −1)2 +8e2(x −1)(y −1)+6e2(x −1)2 ]

3d plot

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 0.96

0.97 0.98

0.99 1 1.01

1.02 1.03

1.04 1.05

6 6.5

7 7.5

8 8.5

9 9.5

z1

x1

y1

z1

6 6.5

7

7.5

8

8.5

9

9.5

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05

x1

0.95

0.96

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

y1

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

% error




Discussion




3d plot

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 0.96

0.97 0.98

0.99 1 1.01

1.02 1.03

1.04 1.05

5.5 6

6.5 7

7.5 8

8.5 9

z1

x1

y1

z1

5.5 6

6.5

7

7.5

8

8.5

9

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05

x1

0.95

0.96

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

y1

-3

-2.5

-2

-1.5

-1

-0.5

0

% error

Quadratic approx:

f = L(f )+ 12 [6e2(y −1)2 +8e2(x −1)(y −1)+6e2(x −1)2 ]

3d plot

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 0.96

0.97 0.98

0.99 1 1.01

1.02 1.03

1.04 1.05

6 6.5

7 7.5

8 8.5

9 9.5

z1

x1

y1

z1

6 6.5

7

7.5

8

8.5

9

9.5

0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05

x1

0.95

0.96

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

y1

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

% errorThe error for a linear approximation of the very non-linear model function i.e. using the GUM method is ∼ 3% and for a quadratic

approximation ∼ 0.3%




Discussion


Taylor series for f (x1, . . . , xd)

For n variables we then have the Taylor series T (x1, . . . , xd) forf (x1, . . . , xd) as

T (x1, . . . , xd) =

∞∑

n1=0

· · ·∞∑

nd=0

∂n1

∂xn11

· · · ∂nd

∂xndd

f (a1, . . . , ad)

n1! · · ·nd !

×(x1 − a1)n1 · · · (xd − ad)nd(7)

(8) T (x1, . . . , xd) =∑

i∈N0

Dαf (a)

α!(x − a)α




Discussion


Special cases for Taylor series expansions

For a model f (x) of the measurement system given with inputsx = [x1, . . . , xn]

T (x is a column vector with dimensions n × 1) andwith nominal value a which is the state that the measurement ystemis in then making use of the general Taylor series expansion formultiple variables T (x) =

∑

|α|60Dαf (a)

α! (x − a)α we note that:◮ First order approximation:

(9) f (x) ≈ f (a1, . . . , an) +∂f∂x1

∣

∣

∣

∣

a(x1 − a1)+ · · ·+ ∂f

∂xn

∣

∣

∣

∣

a(xn − an)




Discussion


Special cases for Taylor series expansions cont.In most cases it is seldom beneficial to construct a multiple variableTaylor series expansion for 3rd or higher order

◮ Second order approximation:

f (x) ≈ f (a1, . . . , an) +

[

∂f∂xa

∣

∣

∣

∣

a· · · ∂f

∂xn

∣

∣

∣

∣

a

]

x1 − a1...

xn − an

+12!

∂2f∂x2

1

∂2 f∂x1∂x2

· · · ∂2 f∂x1∂xn

∂2f∂x2∂x1

∂2 f∂x2

2· · · ∂2 f

∂x2∂xn

......

. . ....

∂2f∂xn∂x1

∂2 f∂xn∂x2

· · · ∂2 f∂x2

n

x1 − a1

· · ·xn − an

(10)

Comment: the n × n square matrix above is the Hessian matrixfor f and all the entries are evaluated at a




Discussion


Final Measurement Model - Calc detailsRecall that we have a model for the mass to be measured

(11) mW

(

1 − ρair

ρW

)

= (mR + δmR)

(

1 − ρair

ρR

)

and we wish to write the model in terms of the ‘conventional’ mass bysubsituting the formulae

mW = mW ,c

(

1 − ρair0

ρW 0

)(

1 − ρair0

ρW

)−1

(12a)

mR = mR,c

(

1 − ρair0

ρW 0

)(

1 − ρair0

ρR

)−1

(12b)

δmR = δmR,c

(

1 − ρair0

ρW 0

)(

1 − ρair0

ρR

)−1

(12c)

Once the equations are substited we then want to solve for mW ,c

which is the mass of the weight that we wish to calibrateVishal Ramnath [email protected] NMISA-08-0121: Review of Methodology & Rationale of MC Simulation



Discussion


Measurement Model - Formulating and solving inMaxima

In Maxima we use the following computer code:

LHS: mW*(1 - rhoair/rhoW);RHS: (mR + deltamR)*(1 - rhoair/rhoR);LHS1: ev(LHS, mW = mWc*(1 - rhoair0/rhoW0)/(1 - rhoair0/rhoW));RHS1: ev(RHS, mR = mRc*(1 - rhoair0/rhoW0)/(1 - rhoair0/rhoR),

deltamR =deltamRc*(1 - rhoair0/rhoW0)/(1 - rhoair0/rhoR));

soln: solve(LHS1 = RHS1, mWc);mWc: rhs(soln[1]);

whence

mW ,c =1

(ρR − ρair0)ρW − ρairρR + ρairρair0×

[((mR,c + δmR,c)ρR − ρairmR,c − ρairδmR,c)ρW

+(−ρair0mR,c − ρair0δmR,c)ρR

+ρairρair0mR,c + ρairρair0δmR,c](13)




Discussion


Measurement Model - GUM Formulation

The measurement model consists of five input parameters which we list below:symbol description PDF commentsmR,c reference mass Gaussian from meas certδmR,c balance mass Gaussian from meas certρair density of air rectangular estimated from CIPM formulaρW density of weight rectangular estimate that is equally likelyρR density of reference rectangular from literature of physical

propertiesComment on parameters that are not included:

◮ the density ρW 0 does not explicitly appear in the model equation as itcancels out when the model equation is algebraically solved for mW ,c

which would not be obvious in a spreadsheet

◮ the air density ρair0 is not included in the model as a variable but as aconstant since this is known exactly




Discussion


Practical Implementation of MC Model InputsDefinition (Model constants & parameters)In a measurement mathematical model working in SI units one shoulddistinguish between how to incorporate constants and parameters. Aparameter is a variable that one is uncertain of and which has a statisticaluncertainty (however small) and PDF whilst a constant is exactly known.1

Fact (Theories which use exact constants i.e. zero unc)An example would be the speed of light which was historically measuredusing various techniques with associated experimental uncertainties and withEinstein’s Special Theory of Relativity fixed and then later defined as

c0 = 299792458 m.s−1 where σ(c0)def= 0

Fact (Theories which use approx constants i.e. finite unc)An example would be the Avogadro number NA = 6.02214179× 1023 mol−1

which as a constant of nature is fixed but which is currently experimentallyknown to an accuracy of σ(NA) = 0.00000030× 1023 mol−1

1For details see the CODATA website for physical and chemical reference values at

http://physics.nist.gov/cuu/Constants/international.html




Discussion


Math Model - GUM Calcs 1We will consider both first and second order calc’s using the GUM forcomparison with a Monte Carlo simulation.

◮ 1st order: apply the GUM as usual with sums of products ofgradients etc.

◮ 2nd order: must build up f with additional terms using Hessianmatrix etc.

Fact (Practical observation of GUM calc’s)The GUM makes use of the assumption that there is a linearized model of thesystem to propogate the uncertainties and to be strictly consistent one shouldapply a linearized model when calculating the standard uncertainty in ordernot to mix of terms from different assumptions and approximations, howeverwe note that in practice most metrologists would most likely take the originalexpression to evaluate the model and not its linearization - this is only valid ifthe model is approximately linear in a neighbourhood of a where a is thestate space that the system is in.




Discussion


Math Model - GUM Calcs 2

Since there are five input variables in our mathematical model thedistinction and implications of 1st and 2nd order approximations arenot immediately obvious to appreciate. Noting the variables

mR,c , δmR,c , ρair , ρW , ρR

we can reasonably conclude that the two variables that are mostlikely to be uncertain and vary are

◮ δmR,c because this must be adequately controlled to achieve abalance and equlibrium on the force beam and the equilibriumcan be a bit subjective if there isn’t an exact balance and thebeam is moving very slowly

◮ ρair the actual air density which will depend and vary with thelaboratory’s ambient temperature and pressure




Discussion


Math Model - GUM Calcs 2 cont

Setting the nominal conditions for argument as

a = (mR,c , δmR,c , ρair , ρW , ρR)

= (0.099 kg, 0.001 kg,

1.17 kg.m−3, 7800 kg.m−3, 8000 kg.m−3)

for mW ,c ≈ 0.100 kg we can then see in a limited sense theimplications of the GUM requirement for a linearized model.




Discussion


Mass Unc Math Model - Approx

Linear approx of f error [ppm]O(4 × 10−3)

0 0.0005 0.001 0.0015 0.002 0.0025 1.1

1.15

1.2

1.25

1.3

1.35

-0.004

-0.003

-0.002

-0.001

0

0.001

0.002

0.003

0.004

δmR,c / kg

ρai

r/

kg.m

−3

Quadratic approx of f error [ppm]O(5 × 10−8)

0 0.0005 0.001 0.0015 0.002 0.0025 1.1

1.15

1.2

1.25

1.3

1.35

-5e-008

-4e-008

-3e-008

-2e-008

-1e-008

0

1e-008

2e-008

3e-008

4e-008

5e-008

δmR,c / kgρ

air/

kg.m

−3




Discussion


Uncertainty results using the GUM 1Assuming for argument that all the inputs formW ,c = f (mR,c , δmR,c, ρair , ρW , ρR) have uncertainties of 0.1% and in additionare not correlated we then have that the uncertainty estimate for the massbeing weighed mW ,c reported in standard uncertainty is:

◮ 1st order:

(14) u(mW ,c) = 1.0000000740094588× 10−6 kg

◮ 2nd order:

u2(f ) =N∑

i=1

(

∂f∂xi

)2

u2(xi)

+

N∑

i=1

N∑

j=1

[

12

(

∂2f∂xi∂xj

)2

+∂f∂xi

∂3f∂xi∂x2

j

]

u2(xi)u2(xj)

= L + H

u(f ) = 1.0001135041200297× 10−6 kg(15)

◮ difference in uncertainty is underestimated by approx. 113 ppm




Discussion


Uncertainty results using the GUM 2

Full uncertainty can be misleading if linearization not accurate if f isvery non-linear – in the above it is not too significant since f is not toonon-linearIt should be noted that H is formed out of double sum over thenumber of variables N and that such a calculation can onlyrealistically be performed in a computer algebra system due to theexcessive number of partial derivatives that must be computed e.g.with N = 5 then 100 partial derivatives must be calculated.The computer code to perform this computation within a CAS e.g.Maxima is relatively straightforward to implement but it should benoted that the full expression can become algebraically large andunwieldy – the non-linear terms correctly evaluate to zero when themodel is indeed linear.




Discussion


Including non-linear terms using the GUMExample (A nonlinear functional)Work out the uncertainty for f (x1, x2) = exp[x2

1 + x22 ] assuming

u(x1) = u(x2) = 0.1% at the point a = [x1 = 1, x2 = 1]T and comparethe linear and nonlinear answers using the GUM.The linearization of f is 2e2(x2 − 1) + 2e2(x1 − 1) + e2 and now thenonlinear term H where u2(f ) = L + H is

H = u21

(

u22

(

8 x21 x2

2 e2(

x22 +x2

1

)

+ 2 x1 ex22 +x2

1

(

8 x1 x22 e

x22 +x2

1 + 4 x1 ex22 +x2

1

))

+u21

2 x1 ex22 +x2

1

(

8 x31 e

x22 +x2

1 + 12 x1 ex22 +x2

1

)

+

(

4 x21 e

x22 +x2

1 + 2 ex22 +x2

1

)2

2

+u22

(

u21

(

8 x21 x2

2 e2(

x22 +x2

1

)

+ 2 x2 ex22 +x2

1

(

8 x21 x2 e

x22 +x2

1 + 4 x2 ex22 +x2

1

))

+u22

2 x2 ex22 +x2

1

(

8 x32 e

x22 +x2

1 + 12 x2 ex22 +x2

1

)

+

(

4 x22 e

x22 +x2

1 + 2 ex22 +x2

1

)2

2




Discussion


Non-linear calc - how significant are the GUM approx?

For f = ex21 +x2

2 the non-linear contribution H is non-zero as indicatedabove, and the difference between the linear and non-linearuncertainty estimates is

◮ u(flinear) = 0.020899406696487◮ u(fnon−linear) = 0.02089964181349◮ the difference in uncertainty is therefore underestimated by

approx. 11 ppm

Fact (GUM linear model underestimates uncertainty)It is seen that a linearized model may underestimate the actualuncertainty by 10 – 100 ppm




Discussion


Comparison of PDF’s for Math Model’s Calc UncWhen calculating uncertainties there are three PDF’s that one mustconsider when interpreting the measurand’s uncertainty:

◮ the actual PDF for the measurand as computed in terms of aconvolution integral (Markov formula2)

(16) g(η) =

∫ ∞

−∞

∫ ∞

−∞

· · ·∫ ∞

−∞

g(ξ)δ(y − f (ξ))dξNdξN−1 · · · dξ1

◮ a Gaussian3 like i.e. a t-distribution with νeff degrees-of-freedomvia. the Welch-Satterwaithe formula for the calculation of themeasurand y ’s PDF as per the GUM approach

◮ a discrete PDF in a Monte Carlo simulation that is built up withsampled data from the input PDF’s g1(ξ1), . . . , gN(ξN) that willconverge to the measurand’s actual PDF (as calculated with acovolution integral) as the number of MC events M → ∞

2adequate mathematical statistics working knowledge is necessary to fullyunderstand the conditions/derivation of the Markov formula wrt. GUM

3A Gaussian PDF i.e. N(µ = 0; σ2) is entirely defined in terms of the variance σ2

whilst a t-distribution needs ν to define its shapeVishal Ramnath [email protected] NMISA-08-0121: Review of Methodology & Rationale of MC Simulation



Discussion


Comment on Application of Markov formula◮ The GUM is based on the application of the Markov formula to

linearized models and all of the results and formulae in the GUMcan be derived (with certain assumptions) via. application of theMarkov formula

◮ Practical examples:1. Higher order terms are necessary in the GUM for non-linear

models where the GUM will not work yielding incorrect results e.g.Y = X 2 where u(y) = 2x · u(x)∀x if just linear terms of the formu2(f ) =

∑Ni=1[∂xi f · u(xi)]

2 are used2. The Markov formula will yield the correct result with

u(y) = u(x)√

4x2 + 45 u2(x) which is true even for x = 0

◮ In general a direct evaluation is only analytically possible forcertain simple cases whilst symbolic evaluation is only feasiblewith a low order of variables requiring transformations andevaluation/calculation of Jacobians with a numerical approachpreferred




Discussion


Comment on Application of Markov formula cont.

◮ The joint PDF gX1,X2,...(ξ1, ξ2, . . .) built up in terms of matrixmultiplications requires the use of a Dirac delta function δ asdefined in terms of a sum with derivative terms and in additionmanipulation of the inputs ξi wrt. the output η

◮ Such calculations in the GUM require the application of furthermatrix algebra and will not be considered in this presentation

◮ The direct application of the Markov formula is in practiceawkward and difficult to implement particularly in the case ofnon-linear models and the use of a Monte Carlo approach isentirely consistent with the Markov formula and is in fact a morepractical calculation method that does not rely on any of theassumptions inherent as in the GUM




Discussion


Implementation of the MC Algorithm

We proceed as per the six steps as previously discussed as follows:

1. set the number of simulations to a minimum number of events toyield adequate statistical data to analyse, say M = 104

2. the model for our measurand is

mW ,c = f (mR,c , δmR,c , ρair , ρW , ρR)(17)

y = f (x1, x2, x3, x4, x5)(18)

so there are 5 inputs for the model and all of these inputs aresimple scalar quantities. The first two inputs have a GaussianPDF whilst the remaining inputs all have rectangular PDF’s.Our numerical calculations will be performed in GNU Octave forconvenience




Discussion


Implementation of the MC Algorithm cont.

2. implementation for sample draws in Octave:◮ for Gaussian PDF with g ∼ N(µ, σ): set z = µ + σ · randn()◮ for a rectangular PDF with mean µ and half-width a: set

z = µ + a ∗ 2 ∗ (rand() − 0.5)

3. the model is just evaluated by specifying the functional f as am-file of form




Discussion



3. sample m-file:

function value = mWc(mRc,deltamRc,rhoair,rhoW,rhoR)rhoair0 = 1.2;value = ( ((mRc + deltamRc)*rhoR - rhoair*mRc

- rhoair*deltamRc)*rhoW+ (-rhoair0*mRc - rhoair0*deltamRc)*rhoR+ rhoair*rhoair0*mRc+ rhoair*rhoair0*deltamRc )/( (rhoR - rhoair0)*rhoW

- rhoair*rhoR + rhoair*rhoair0 );

sample function call:

test = mWc(0.9999, 0.00099, 1.19, 7950, 8003.5)test = 1.00088999158172




Discussion


Implementation of the MC Algorithm cont.3. Short overview of a MC program (actual program ∼ 300 lines of code)

clear allclcM = input(’Enter the number of MC events: ’);% nominal values in modelmRc_ref = 0.099; deltamRc_ref = 0.001;rhoair_ref = 1.17;rhoW_ref = 7800; rhoR_ref = 8000;% uncertainties in modelux1 = 0.1E-2*mRc_ref; ux2 = 0.1E-2*deltamRc_ref;ux3 = 0.1E-2*rhoair_ref;ux4 = 0.1E-2*rhoW_ref; ux5 = 0.1E-2*rhoR_ref;data = zeros(M,1);ticfor i=1:M

x1 = mRc_ref + ux1*randn();x2 = deltamRc_ref + ux2*randn();x3 = rhoair_ref + ux3*2*(rand() - 0.5);x4 = rhoW_ref + ux4*2*(rand() - 0.5);x5 = rhoR_ref + ux5*2*(rand() - 0.5);f = mWc(x1,x2,x3,x4,x5);data(i) = f;

endtochist(data)print(’C:/Draft/test.eps’,’-color’)

0

500

1000

1500

2000

2500

0.0997 0.0998 0.0999 0.1 0.1001 0.1002 0.1003 0.1004

Running the above program on a single 2.4 GHz CPU takes approx. 1.8seconds for 104 MC events i.e. simple models can realistically be simulatedwith run times of a few minutes.




Discussion



4. the output quantity is estimated by applying a weighted sum of allthe MC events – this is generally indistinguishable from thearithmetic mean for large M e.g. M = O(106)

5. the simulation is used to build up a discrete representation of theunderlying model’s distribution function – this is equivalent tocalc’s via. a PDF but is more accurate: a rough analogy would bethe use of a histogram of measured data as a indicator of theunderlying PDF as illustrated in the previous slide

6. once all the data is generated a coverage interval is estimated byinverse linear interpolation using G(η) – a rough explanationwould be a minima search of a parameter α for a function built upin terms of the inverse function of the distribution function whereα would be used to calculate lower and upper bounds for y for agiven confidence limit e.g. 95.45%




Discussion

Analysing and Understanding the DataReporting Results in GUM Terms

Post-processing and Analysis of a SimulationA measurement uncertainty calculation using the Monte Carlomethod requires an underlying model of the measurand (univariate ormultivariate) and could be in the form of an explicit or implicitformulation. The steps involved in order to perform post-processingand analysis are simply:

◮ formulate a model of the measurand◮ apply multiple simulations of the model using sampled draws

from the input PDF’s to generate a data set of the measurand toanalyse

The data set to perform further operations on is simply a collection ofthe model evaluated M times e.g. if the model is univariate andexplicit i.e. y = f (x1, . . . , xN) then the data set is just an array ofnumbers

(19) data = [y1, . . . , yM ]




Discussion


Analysing and Understanding the DataAn statistical analysis of the data using formulae as defined in thetechniques/algorithms inherent in a Monte Carlo measurementuncertainty calculation can be thought of as a numerical/statisticalexperiment that approximates a real physical experimental situationwith simulated experimental conditions. The data that results from aMC simulation are:

◮ y = the expected value of the measurand◮ u(y) = the measurand’s standard deviation◮ [ylow, yhigh] = the confidence interval of the measurand for a given

probability e.g. p = 95.45% using the actual PDF g(η) for themeasurand i.e. the “expanded uncertainty”

The measurand’s PDF g(η) is not necessarily symmetric or evenGaussian which means a simple ± for expanded uncertainty may bemisleading with results more accurately reported of form y

+(yhigh−y)

−(y−ylow )




Discussion


Reporting Results in GUM Terms◮ In essence a MC simulation will yield the same outputs that are required

in a measurement uncertainty calculation viz. standard and expandeduncertainties for the expected value of the measurand (y ≡ µ):

u(Y ) = u(y) ≡ σ(20)

U(Y ) ≈12

[(yhigh − y) + (y − ylow )](21)

◮ the difference is that since the method does not involve the concept ofeffective degrees of freedom νeff there is no corresponding “coveragefactor” e.g. k = 2 for a p = 95.45% confidence level so an analogousquantity would be

(22) k =U(y)

u(y)

◮ Recall that in the GUM one must convert to standard uncertaintieswhere the PDF is by definition Gaussian i.e. N(µ, σ2)




Discussion

Discussion

◮ The GUM is based on certain assumptions/approximations andonly be accurately applied in certain limiting cases

◮ Modifications to the standard GUM technique are available incertain limiting cases but are generally limiting and difficult toapply with only modest improved uncertainty accuracyimprovement

◮ A Monte Carlo measurement uncertainty simulation will work forany problem of arbitary complexity and will always yield accurateand reliable results with sufficient MC simulation events (anadaptive MC algorithm to test for convergence is possible toavoid unnessary computational overhead)

◮ Questions


Documents

Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software