Copyright 2004 David J. Lilja1 Errors in Experimental Measurements Sources of errors Accuracy, precision, resolution A mathematical model of errors Confidence

Copyright 2004 David J. Lilja 1

Errors in Experimental Measurements

Sources of errors Accuracy, precision, resolution A mathematical model of errors Confidence intervals

For means For proportions

How many measurements are needed for desired error?


Why do we need statistics?

1. Noise, noise, noise, noise, noise!

OK – not really this type of noise


Why do we need statistics?

2. Aggregate data into meaningful information.445 446 397 226

388 3445 188 100247762 432 54 1298 345 2245 883977492 472 565 9991 34 882 545 4022827 572 597 364

...x


What is a statistic?

“A quantity that is computed from a sample [of data].”

Merriam-Webster

→ A single number used to summarize a larger collection of values.


What are statistics?

“A branch of mathematics dealing with the collection, analysis, interpretation, and presentation of masses of numerical data.”

Merriam-Webster

→ We are most interested in analysis and interpretation here.

“Lies, damn lies, and statistics!”


Goals

Provide intuitive conceptual background for some standard statistical tools.

• Draw meaningful conclusions in presence of noisy measurements.

• Allow you to correctly and intelligently apply techniques in new situations.

→ Don’t simply plug and crank from a formula.


Goals

Present techniques for aggregating large quantities of data.

• Obtain a big-picture view of your results.• Obtain new insights from complex

measurement and simulation results.

→ E.g. How does a new feature impact the overall system?


Sources of Experimental Errors Accuracy, precision, resolution


Experimental errors

Errors → noise in measured values Systematic errors

Result of an experimental “mistake” Typically produce constant or slowly varying bias

Controlled through skill of experimenter Examples

Temperature change causes clock drift Forget to clear cache before timing run


Experimental errors Random errors

Unpredictable, non-deterministic Unbiased → equal probability of increasing or decreasing

measured value Result of

Limitations of measuring tool Observer reading output of tool Random processes within system

Typically cannot be controlled Use statistical tools to characterize and quantify


Example: Quantization → Random error


Quantization error

Timer resolution

→ quantization error Repeated measurements

X ± Δ

Completely unpredictable


A Model of Errors

Error Measured value

Probability

-E x – E ½

+E x + E ½


A Model of Errors

Error 1 Error 2 Measured value

Probability

-E -E x – 2E ¼

-E+E x ¼

+E -E x ¼

+E +E x + 2E ¼


A Model of Errors

Probability

0

0.1

0.2

0.3

0.4

0.5

0.6

x-E x x+E

Measured value


Probability of Obtaining a Specific Measured Value


A Model of Errors

Pr(X=xi) = Pr(measure xi)

= number of paths from real value to xi

Pr(X=xi) ~ binomial distribution As number of error sources becomes large

n → ∞, Binomial → Gaussian (Normal)

Thus, the bell curve


Frequency of Measuring Specific Values

Mean of measured values

True valueResolution

Precision

Accuracy


Accuracy, Precision, Resolution

Systematic errors → accuracy How close mean of measured values is to true

value Random errors → precision

Repeatability of measurements Characteristics of tools → resolution

Smallest increment between measured values


Quantifying Accuracy, Precision, Resolution

Accuracy Hard to determine true accuracy Relative to a predefined standard

E.g. definition of a “second”

Resolution Dependent on tools

Precision Quantify amount of imprecision using statistical

tools


Confidence Interval for the Mean

c1 c2

1-α

α/2 α/2


Normalize x

1

)(deviation standard

mean

tsmeasuremen ofnumber /

n

1i

2

1

n

xxs

x x

nns

xxz

i

n

ii



Normalized z follows a Student’s t distribution (n-1) degrees of freedom Area left of c2 = 1 – α/2 Tabulated values for t

c1 c2

1-α

α/2 α/2



As n → ∞, normalized distribution becomes Gaussian (normal)

c1 c2

1-α

α/2 α/2



1)Pr(

Then,

21

1;2/12

1;2/11

cxc

n

stxc

n

stxc

n

n


An Example

Experiment Measured value

1 8.0 s

2 7.0 s

3 5.0 s

4 9.0 s

5 9.5 s

6 11.3 s

7 5.2 s

8 8.5 s


An Example (cont.)

14.2deviation standard sample

94.71

sn

xx

n

i i


An Example (cont.)

90% CI → 90% chance actual value in interval 90% CI → α = 0.10

1 - α /2 = 0.95 n = 8 → 7 degrees of freedom

c1 c2

1-α

α/2 α/2


90% Confidence Interval

a

n 0.90 0.95 0.975

… … … …

5 1.476 2.015 2.571

6 1.440 1.943 2.447

7 1.415 1.895 2.365

… … … …

∞ 1.282 1.645 1.960

4.98

)14.2(895.194.7

5.68

)14.2(895.194.7

895.1

95.02/10.012/1

2

1

7;95.01;

c

c

tt

a

na


95% Confidence Interval

a

n 0.90 0.95 0.975

… … … …

5 1.476 2.015 2.571

6 1.440 1.943 2.447

7 1.415 1.895 2.365

… … … …

∞ 1.282 1.645 1.960

7.98

)14.2(365.294.7

1.68

)14.2(365.294.7

365.2

975.02/10.012/1

2

1

7;975.01;

c

c

tt

a

na


What does it mean?

90% CI = [6.5, 9.4] 90% chance real value is between 6.5, 9.4

95% CI = [6.1, 9.7] 95% chance real value is between 6.1, 9.7

Why is interval wider when we are more confident?


Higher Confidence → Wider Interval?

6.5 9.4

90%

6.1 9.7

95%


Key Assumption

Measurement errors are Normally distributed.

Is this true for most measurements on real computer systems?

c1 c2

1-α

α/2 α/2


Key Assumption

Saved by the Central Limit TheoremSum of a “large number” of values from

any distribution will be Normally (Gaussian) distributed.

What is a “large number?” Typically assumed to be >≈ 6 or 7.


How many measurements?

Width of interval inversely proportional to √n Want to minimize number of measurements Find confidence interval for mean, such that:

Pr(actual mean in interval) = (1 – α)

xexecc )1(,)1(),( 21



2

2/1

2/1

2/1

21 )1(),(

ex

szn

exn

sz

n

szx

xecc



But n depends on knowing mean and standard deviation!

Estimate s with small number of measurements

Use this s to find n needed for desired interval width



Mean = 7.94 s Standard deviation = 2.14 s Want 90% confidence mean is within 7% of

actual mean.



Mean = 7.94 s Standard deviation = 2.14 s Want 90% confidence mean is within 7% of

actual mean. α = 0.90 (1-α/2) = 0.95 Error = ± 3.5% e = 0.035



9.212)94.7(035.0

)14.2(895.12

2/1

ex

szn

213 measurements

→ 90% chance true mean is within ± 3.5% interval


Proportions

p = Pr(success) in n trials of binomial experiment

Estimate proportion: p = m/n m = number of successes n = total number of trials


Proportions

n

ppzpc

n

ppzpc

)1(

)1(

2/12

2/11


Proportions

How much time does processor spend in OS?

Interrupt every 10 ms Increment counters

n = number of interrupts m = number of interrupts when PC within OS


Proportions

How much time does processor spend in OS?

Interrupt every 10 ms Increment counters

n = number of interrupts m = number of interrupts when PC within OS

Run for 1 minute n = 6000 m = 658


Proportions

)1176.0,1018.0(6000

)1097.01(1097.096.11097.0

)1(),( 2/121

n

ppzpcc

95% confidence interval for proportion So 95% certain processor spends 10.2-11.8% of its

time in OS


Number of measurements for proportions

2

22/1

2/1

2/1

)(

)1(

)1(

)1()1(

pe

ppzn

n

ppzpe

n

ppzppe



How long to run OS experiment? Want 95% confidence ± 0.5%



How long to run OS experiment? Want 95% confidence ± 0.5% e = 0.005 p = 0.1097



102,247,1

)1097.0(005.0

)1097.01)(1097.0()960.1(

)(

)1(

2

2

2

22/1

pe

ppzn

10 ms interrupts

→ 3.46 hours


Important Points

Use statistics to Deal with noisy measurements Aggregate large amounts of data

Errors in measurements are due to: Accuracy, precision, resolution of tools Other sources of noise

→ Systematic, random errors


Important Points: Model errors with bell curve

True value

Precision

Mean of measured values

Resolution

Accuracy


Important Points

Use confidence intervals to quantify precision Confidence intervals for

Mean of n samples Proportions

Confidence level Pr(actual mean within computed interval)

Compute number of measurements needed for desired interval width

Documents

Copyright 2004 David J. Lilja1 Errors in Experimental Measurements Sources of errors Accuracy, precision, resolution A mathematical model of errors Confidence