Simplifying Measurement Uncertainties
Bill Hirt, Ph.D / February 2016
Start the Process
• Light the fuse (candle)
• Shed light on the way
2
Back to the beginning
• Accredited CALIBRATION labs report measurements on tools and devices needing regular service … typically including uncertainties for those measurements.
• Accredited TESTING labs, when requested or when needed to interpret Statements of Compliance, report uncertainties alongside their test measurements.
3
Framework for uncertainties
• Unless calculating MDL’s or LOD’s or LOQ’s …
• It is presumed that ISO 17025 accredited labs demonstrate the competence to calculate and report GUM uncertainties … estimated at roughly a 95 % confidence.
4
When do accredited labsreport MU’s ?
• Every accredited calibration certificate, unless requested otherwise
• Test reports :
• When a customer requests it
• When the uncertainty affects compliance with a specification limit
• When it is relevant to the validity or application of the result
5
What do we mean by MU?
• Measurement value ± uncertainty (MU)
• MU usually at 95 % confidence – Why?
• MU usually reported at k = 2
6
7
Where do we find most MUs?
• On calibration certificates
• On test reports
• In footnotes
8
Did you know that …
• A high percentage of ISO 17025 accredited calibration laboratories issue incorrect uncertainties on their certificates ?
9
ISO 17025 includes –Calibration and Testing
• A good percentage of calibration labs do not report MU’s on their certificates
• Most testing labs do not report MU’s on their test reports
• And WHY ???
10
First Group Discussion
• Scenario used in 3-day MU course outlined first
• Have groups discuss other real-life scenarios where uncertainty can be critical or life-saving
• Put examples on index cards on the tables
11
Uncertainty and Traceability• Metrological traceability (not sample
traceability)
• Two platforms of confidence
• Comparison with hi-quality stds thru chain to SI through an NMI or DI
• Confidence that GUM unc’s used … and actual measurement error clearly known
12
Uncertainty and Proficiency Testing (PT)
• Calibration PT programs typically request that one or more devices have measurements taken … and Both the measurements and their uncertainties be reported.
• The Cal PT study issues a report with standardized results including an En
13
En value Determination (Calibration Labs)
Interpretation of En values
Accredited Testing Labsand PT
• Very few ISO 17025 accredited PT programs for testing include a requirement for MU’s
• Some now request them
• Many in the future will require them
• In addition to z-scores, many future reports will include En values too
16
17
“The Big Three” in 17025
• Measurement Uncertainty
• Metrological Traceability
• Proficiency Testing
18
Uncertainty and Statements of Compliance• High percentage of measurements are made
to ensure manufactured or natural materials meet a narrow range of specifications
• Manufacturers and regulators define specs
• No measurement is perfect
• MU may or may not be critical to have CONFIDENCE that specs are being met
19
Group discussion - 2
• At your tables, discuss specific areas of compliance specs that you are aware of
• Make a list of at least 6 different groups or types of specs at your table to report later
• The group will share after your discussion
20
Uncertainty and Guard Banding
• Management of manufacturing or other monitoring to assure material is safely within specifications, including consideration of uncertainty
• Organization may adjust their spec to capture either more potentially defective samples or allow more acceptable product out the door
21
22
23
Definitions • Uncertainty – a property of a measurement result
that defines the range of probable values of the measurand
• Uncertainty budget – the systematic description of uncertainty determinations relevant to specific measurements including ranges plus all factors, assumptions and calculations included (must include both type A and type B factors)
24
Basic Steps in Uncertainty Budgets
1. List ALL potential factors affecting variability in measurements - make table
2. Determine the standard uncertainty for each factor (includes distribution)
3. Perform RSS for all factors to create the combined (standard) uncertainty
4. Multiply by distribution factor (k=2 … or ?)
25
Root Sum Squaring
26
Sample Standard Deviation(s)
Calibration MU example
Budget for 6 inch CALIPERS
i Component of UncertaintyUncertainty,
U(xi) Distribution Divisor Std Unc, u(xi)
1 Standard uncertainty 6 Normal, 2s 2.00 3 uin2 Resolution 500 Rectangular 1.73 289 uin3 Repeatability 297.98 Normal, 1s 1.00 298 uin4 Uncompensated error 25 Rectangular 1.73 14.4 uin5 Temperature difference between
instrument and gage blocks20.1 U-Shaped 1.41 14.2 uin
6 Temperature variance from 68º F 16.08 U-Shaped 1.41 11.4 uin7
combined standard uncertainty, uc 416 uincoverage factor, k 2
expanded uncertainty, Uc 832 uinExpanded uncertainty rounded UP to 2 significant figures 840 uin
27
Second Cal MU example
Uncertainty Budget for Bench Scale - 150 lb capacity
i Component of UncertaintyUncertainty,
U(xi) Distribution Divisor Std Unc, u(xi)
1 Standard uncertainty 0.000106 Normal, 2s 2.00 0.000 lb2 Resolution 0.05 Rectangular 1.73 0.029 lb3 Uncompensated error (Three 50 #
weights)0.0105 Rectangular 1.73 0.006 lb
4 Repeatability 0.0217 Normal, 1s 1.00 0.02 lb
combined standard uncertainty, uc 0.0366 lb
coverage factor, k 2
expanded uncertainty, Uc 0.0732 lb
Expanded uncertainty rounded UP to 2 significant figures 0.074 lb
28
Testing MU
• Very often much more complicated than calibration -- why ??
• Many stages in test processes
• Many error types with different units of measure
• Many errors not defined and require guess
29
30
31
Table A6.2: Summary of results from collaborative trial of the method and in-house repeatability check
32
Table A6.4: Combined standard uncertainties
33
Key current and future factor
• SAMPLING and sample error factors
34
Nitrogen in Forage Budget
35
Nitrogen in Dry Feed Budget
36
New proposed TABLE
• Handout shows old / traditional version of the Student’s T Table
• Back side shows our proposed new version
• Let’s review its features
37
Open quiz for the room
• Using the new Student’s T table, what is the k-factor … for an MU … at 95% confidence … when the number of repeatability measurements is :
• 3 ?
• 5 ?
• 10 ?
• 30 ?
38
Open quiz for the room - 2
• … and the test measurement mean is 100 mg and the combined uncertainty is 5 mg, what is each 95% MU :
• With 3 repeats
• With 5 repeats
• With 10 repeats
• With 30 repeats
39
• For an MU … at 95% confidence … when the number of repeatability measurements is :
• 3
• 5
• 10
• 30
Open quiz for the room - 3
# of RepeatMeasurements
k-factorMultiplier
Uexp (95%)To report
3
5
10
30
( test measurement mean is 100 mg and the combined uncertainty is 5 mg )
40
Open quiz for the room – 3+
# of RepeatMeasurements
k-factorMultiplier
Uexp (95%)To report
3 4.3 21.5
5 2.78 13.9
10 2.26 11.3
30 2.0 10
41
Example of questionable MU on
commercial calibration certificate
42
Standard Deviation of the Mean
• The equation on the left for repeatability / SD becomes
43
• Caution – it does NOT replace repeatability SD
44
Control Charts
• Plots of long-term measurements of a single parameter to note trends or variability
• Often the main basis for testing uncertainties
• Main testing lab equivalent of repeatability and/or reproducibility
45
46
47
AAFCO example of Control Chart of Matrix Spike
48
49
Basic Steps in Uncertainty Budgets
1. List ALL potential factors affecting variability in measurements - make table
2. Determine the standard uncertainty for each factor (includes distribution)
3. Perform RSS for all factors to create the combined (standard) uncertainty
4. Multiply by distribution factor (k=2 … or ?)
50
Group discussion - 3
• We want to simplify the process for MU determination
• We want the fundamentals of uncertainties to be understood
• We know that may tests may be confusing even as to whether an MU is needed
• Discuss at your tables and list confusing examples … for our larger discussion
51
52
Fundamental MU Nuggets
• Key definitions to be able to distinguish:
• Type A and Type B factors
• Repeatability vs reproducibility
• Combined uncertainty vs expanded uncertainty (crucial k-factor to use)
• In unc budgets, any factor contributing less than 10% to the total can be eliminated
53
Fundamental MU Nuggets - 2• Use at least 7 data points for any standard
deviation
• Try to use at least 30 data points (equivalent to infinite number) in Type A factors
• Metrological traceability and the traceability chain needs GUM uncertainties
• Calibration PT studies involve uncertainties and testing PT studies may do so soon (Enand z-scores)
54
Key Nuggets - 3• Repeatability/reproducibility studies may not
adequately cover the range of test tolerances. You may need a high and low range repeatability study.
• Be careful not to replace repeatability standard deviation factor with standard error of the mean in a MU budget. Add not replace.
Key Nuggets - 4• Control charts can often capture full testing
errors, but use caution to be sure ALL significant errors are included
• Repeatability studies or reproducibility studies can often represent the basis for MU determinations
• For any test, consider all potential errors but budgets and calculations may NOT be needed
55
When may MU budgets and calculations not be needed?• If alternative statistics used, eg MDLs, LODs
• If tests qualitative
• If test protocols already define testing variability, precision, repeatability
• This includes 95% confidence determinations for microbial MPN studies
56
Remember …
• No measurement is perfect
• Measurement errors historically combined as Uncertainties … BUT ….
• Uncertainties trigger too often – confusion
• 95% confidence is the basis for uncertainties
57
Final Mantra …
• (not) UNCERTAINTY …
• (but) … CONFIDENCE RANGE !!
58
Thank you
59