Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission. 1 UNCERTAINTY ANALYSIS: A BASIC OVERVIEW

Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission. 1

UNCERTAINTY ANALYSIS:A BASIC OVERVIEW

presented at

CAVS

by

GLENN STEELE

www.uncertainty-analysis.com

August 31, 2011


EXPERIMENTAL UNCERTAINTY REFERENCES

The ISO GUM:The de facto international standard


EXPERIMENTAL UNCERTAINTY REFERENCEShttp://www.oiml.org/publications/?publi=3&publi_langue=en


VALIDATION REFERENCES


VALIDATION REFERENCES

Copyright 2010 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.6

“Degree of Goodness”

• When we use experimental results (such as property values) in an analytical solution, we should consider “how good” the data are and what influence that degree of goodness has on the interpretation and usefulness of the solution

• When we compare model predictions with experimental data, as in a validation process, we should consider the degree of goodness of the model results and the degree of goodness of the data.


Typical comparison of predictions and data, considering no uncertainties:

Res

ult,

C

D

Set point, Re


Comparison of predictions and data considering only the likely uncertainty in the experimental result:

Res

ult,

C

D

Set point, Re

Uncertainties set the resolution at whichmeaningful comparisons can be made.


Validation comparison considering all uncertainties:

E

U D

U x

S + U

r

X

D

S

SIM

S value from the simulation

D data value from experiment

E comparison error

E = S - D = S- D

where (S= model+ input+ num)

URe

US

UCD

Res

ult,

C

D

Set point, Re


“Degree of Goodness” and Uncertainty Analysis

• When an experimental approach to solving a problem is to be

used, the question of “how good must the results be?” should be

answered at the very beginning of the effort. This required

degree of goodness can then be used as guidance in the

planning and design of the experiment.

• We use the concept of uncertainty to describe the “degree of

goodness” of a measurement or an experimental result.


ERRORS&

UNCERTAINTIES


An error is a quantity with a sign and magnitude. (We assume any error whose sign and magnitude is known has been corrected for, so the errors that remain are of unknown sign and magnitude.)

An uncertainty u is an estimate of an interval u that should contain .


Consider making a measurement of a steady variable X (whose

true value is designated as Xtrue) that is influenced by errors i

from 5 elemental error sources.

Postulate that errors 1 and 2 do not vary as measurements are

made, and 3, 4, and 5 do vary during the measurement period:


The total error () is the sum of

(= 1 + 2) the systematic, or fixed, error

(= 3 + 4 + 5) the random, or repeatability, error

= +

varies)

β (does not vary) 1 2


The kth measurement of X then appears as

The total error (k) is the sum of

k the systematic, or fixed, error

k the random, or repeatability, error


• Central Limit Theorem statistics ???


Histogram of temperatures read from a thermometer by 24 students


varies)

β (does not vary) 1 2

Now consider again making the measurements of X


We can calculate the standard deviation sX of the distribution of N measurements of X

and that will correspond to a standard uncertainty (u) estimate of the range of the i’s. We will call sX the random standard uncertainty.

i21truei )()β(βXX


i21truei )()β(βXX


We will estimate systematic standard uncertainties corresponding to

the elemental systematic errors i and use the symbol bi to denote

such an uncertainty. Thus ±b1 will be an uncertainty interval that

should contain 1, ±b2 will be an uncertainty interval that should

contain 2, and so on....

The systematic standard uncertainty bi is understood to be an estimate

of the standard deviation of the parent population from which the

systematic error i is a single realization.


i true 1 2 iX X (β β ) ( )

The standard uncertainty in X -- denoted uX -- is defined

such that the interval ± uX contains the (unknown)

combination

and, in accordance with the GUM, is given by

)()β(β 21

2 2 21 2X Xu b b s


Categorizing and Estimating Uncertainties in the Measurement of a Variable

• GUM categorization by method of evaluation:– Type A “method of evaluation of uncertainty by the

statistical analysis of series of observations”– Type B “method of evaluation of uncertainty by means other

than the statistical analysis of series of observations”

• Traditional U.S. categorization by effect on measurement:– Random (component of) uncertainty estimate of the effect of

the random errors on the measured value– Systematic (component of) uncertainty estimate of the effect

of the systematic errors on the measured value

Both are useful, and they are not inconsistent. Use of both will be illustrated in the examples in this course.


An Additional Uncertainty Categorization

• In the fields of Risk Analysis, Reliability Engineering, Systems Safety Assessment, and others, uncertainties are often categorized as

• Aleatory – Variability

– Due to a random process

• Epistemic– Incertitude

– Due to lack of knowledge


Uncertainty Categorization

100 %

The key is to identify the significant errors and estimate the corresponding uncertainties – whether one divides them into categories for convenience of

Random – Systematic

Type A – Type B

Aleatory – Epistemic

Lemons – Chipmunks

should make no difference in the overall estimate u if one proceeds properly.


OVERALL UNCERTAINTY OF A MEASUREMENT

At the standard deviation level

Systematic Standard Uncertainty =

(for 2 elemental systematic errors)

Random Standard Uncertainty = sX (or )

Combined Standard Uncertainty = uX

Overall or Expanded Uncertainty at C % confidence

2 2X 1 2b b b

Xs

2 2 2XX Xu b s

% % XU =k u


For large samples, assuming the total errors in the measurements have a roughly Gaussian distribution, and using a 95% confidence level, k95 = 2 and

The true value of the variable will then be within the limits

about 95 times out of 100.

1/ 22 295 2 X XU b s

95 95- trueX U X X U

To obtain a value of the coverage factor k, an assumption about the form of the distribution of the

total errors (the ’s) in X is necessary.


RESULT DETERMINED FROM

MULTIPLE MEASURED

VARIABLES


• We usually combine several variables using a

Data Reduction Equation (DRE)

to determine an experimental result.

• These have the general DRE form

• There are two approaches used for propagating uncertainties through the DREs:– the Taylor Series Method (TSM)

– the Monte Carlo Method (MCM)

p

RT

212

D

DC

AV

1 2( , ,..., )Jr r X X X

' 'r u v


Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

For the case where the result r is a function of two variables x and y

r = f(x,y) the combined standard uncertainty of the result, ur, is given by

222 2 2 2r x y r

systematic errorr ru b b s

x y correlation effects

where sr is calculated from multiple result determinations and the bx and by

systematic standard uncertainties are determined from the combination of elemental systematic uncertainties that affect x and y as

and

TAYLOR SERIES METHOD OF UNCERTAINTY PROPAGATION

x

k

M

1k

2x

2x bb

y

k

M

1k

2y

2y bb


Monte Carlo Method of

Uncertainty Propagation

32Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Applying General Uncertainty Analysis – Experimental Planning Phase


GENERAL UNCERTAINTY ANALYSIS

• For a result given by a data reduction equation (DRE)

• the uncertainty is given by

• Example DRE

• Note that (assuming the large sample approximation) the U in the propagation equation can be interpreted as the 95% confidence U95 = 2 u or as the standard uncertainty u as long as each term in the equation is treated consistently.

1 2( , ,..., )Jr r X X X

1 2

22 2

2 2 2 2

1 2

...Jr X X X

J

r r rU U U U

X X X

2

2 DD

FC

V A


Example

It is proposed that the shear modulus, MS, be determined for an alloy

by measuring the angular deformation produced when a torque T is

applied to a cylindrical rod of the alloy with radius R and length L. The

expression relating these variables is

We wish to examine the sensitivity of the experimental result to the

uncertainties in the variables that must be measured before we

proceed with a detailed experimental design. The physical situation

shown below (where torque T is given by aF) is described by the data

reduction equation for the shear modulus

4

2

S

LT

R M

4

2

S

LaFM

R


S

S

S

2 2 22 2 22 2 2 2 2M aL F

S 4

2 2 22 22 2M S S a SL F2 2 2 2

S S SS

2 2 22S SR

2

R2 2 2 2 2 2S

22 2 2M

2

2

S

S S

2LaFM =

πR

U M M U MU UL a F

M L M a M FM L a F

M M UUR

M R MR

U U UU U U1 1 1 4 1

M L a F R

U0.01 0.01 0.01 16 0.

M

S S

2 2

22M M

2S S

01 0.01

U U20 0.01 0.045 4.5%

M M


ESTIMATING RANDOM

UNCERTAINTIES


Data sets for determining estimates of standard deviations and

random uncertainties should be acquired over a time period that is

large relative to the time scales of the factors that have a significant

influence on the data and that contribute to the random errors.


Direct Calculation Approach for Random Uncertainty

For a result that is determined M times

the mean value of the result is

and

M21 r,...,r,r

M

1kkrM

1r

1/2M 2

r kk 1

1s r r

M 1

1/2M 2

r kk 1

1 1s r r

M 1M


ESTIMATING SYSTEMATIC

UNCERTAINTIES


Propagation of systematic errors into an experimental result:


The systematic standard uncertainties for the elemental error sources are estimated in a variety of ways that were discussed in some detail in the course. Among the ways used to obtain estimates are:

use of previous experience,

manufacturer’s specifications,

calibration data,

results from specially designed “side” experiments,

results from analytical models,

and others.


Recall the definition of a systematic standard uncertainty, b. It is not the most likely value of , nor the maximum value. It is the standard deviation of the

assumed parent population of possible values of .


SYSTEMATIC STANDARD UNCERTAINTY

b A / 3

b A / 6



Correlated Systematic Errors

• Typically occur when different measured variables share one or more

elemental error sources

– multiple variables measured with same transducer• probe traversed across flow field

• multiple pressures ported sequentially to the same transducer (scanivalve)

– multiple transducers calibrated against same standard • electronically scanned pressure (ESP) systems in use in aerospace ground test

facilities

• Examples

– q = m Cp (To – Ti)

–

– u’v’

1 2 N

1P P P ... P

N


Using the TSM, there is a

term in the br2 equation for each pair of variables in the DRE that might

share an error source:

• For q = m Cp (To – Ti)

• For

• For u’v’ ....

b b b o o i

22 2q T T T

o o i

q q q... ... 2

T T T

N21 P...PPN1

P

b b b 1 2 1 3

2PP PPP

1 2 1 3

P P P P... 2 2 ...

P P P P

1 2

1 2

2 x x

r rb

x x



Some Final Practical Points on Estimating Systematic Uncertainties

• When estimating b, we are not trying to estimate the most probable value nor

the maximum possible value of

• Always remember to view and use estimates with common sense. For example,

a “% of full scale” b should not apply near zero if the instrument is nulled.

• Resources should not be wasted on obtaining good uncertainty estimates for

insignificant sources – a practice we have observed too many times….


“V&V” – Verification & Validation: The Process

• Preparation

– Specification of validation variables, validation set points, etc. (This specification determines the resource commitment that is necessary.)

– It is critical for modelers and experimentalists to work together in this phase. The experimental and simulation results to be compared must be conceptually identical.

• Verification

– Are the equations solved correctly? (MMS for code verification. Grid convergence

studies, etc, for solution verification to estimate unum .)

• Validation

– Are the correct equations being solved? (Compare with experimental data and

attempt to assess model )

• Documentation


A Validation Comparison


Reality of Interest (Truth): Experiment “as run”

Simulation

Simulation Inputs (Properties, etc.)

Modeling Assumptions

Numerical Solution of Equations

Simulation Result, S

Experimental Data, D

Experimental Errors

model

input

num

D

Comparison Error, E = S - D

Validation Uncertainty,

uval

E = (model) + (input+ num - D)

V&V Overview – Sources of Error Shown in Ovals


• Isolate the modeling error, having a value or uncertainty for everything else

E=S-D = model + (input +num - D)

model = E - (input +num - D)

• If ± uval is an interval that includes (input +num - D)

then model lies within the interval

E ± uval

Strategy of the Approach

E

± uval


Uncertainty Estimates Necessary to Obtain the Validation Uncertainty uval

• Uncertainty in simulation result due to numerical solution of the

equations, unum (code and solution verification)

• Uncertainty in experimental result, uD

• Uncertainty in simulation result due to

uncertainties in code inputs, uinput

1/ 22 2 2val D num inputu u u u

Propagation by(A) Taylor Series(B) Monte Carlo

MethodologySimulation Uncertainty

Modeling error for uncalibrated model used to make calculations between validation points

where usp = uncertainty contribution from the uncertainty of input

parameters at the simulation calculation point

and uE = uncertainty in E at the calculation point from the

interpolation process

2X

2J

1i i

2sp

i

uXs

u


2sp

2val

2Eelmod uuuEervalintthewithinlies


Uncertainty of Calibrated Models

MethodologyInstrument Calibration Analogy

• Uncalibrated instrumentation system

where ut = uncertainty of the transducer

and um = uncertainty of the meter

• Calibrated instrumentation system

where uc is the calibration uncertainty

2m

2tI uuu

1

cI uu2



• If a curve-fit is used to develop a relationship between the meter reading and the calibrated output value, then

where ucf = the curve-fit uncertainty

• If the meter used in testing (m2) is different from the meter used in calibration (m1), then

2cf

2cI uuu

3

2m

2m

2cf

2cI

214uuuuu



• The uncertainties, u, in the previous expressions are standard uncertainties, at the standard deviation level. To express the uncertainty at a given confidence level, such as 95%, the standard uncertainty is multiplied by an expansion factor. For most engineering applications, the expansion factor is 2 for 95% confidence.

u2U95


MethodologyCalibrated Model

• To calibrate a model, the simulation results are compared with a set of data and corrections are applied to the model to make it match the data. The simulation uncertainty is then

• As in the curve-fit uncertainty in the calibration of a transducer, there will be additional uncertainty in the calibrated model based on the error between the corrected simulation results and the data.

2E

2ds uuu

2

ds uu1


MethodologyCalibrated Model

• would apply for simulation results over the range of the input parameter values used in the calibration of the model with the assumption that the input parameters in the simulation have the same uncertainties that they had in the calibration process.

• If the input parameter sources or transducers change for a simulation result, then

2su

2sp

2sp

2E

2ds 213

uuuuu


Documents

Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission. 1 UNCERTAINTY ANALYSIS: A BASIC OVERVIEW