ISO/IEC 17025:2017Statement or Certificate of
Conformance&
Decision rulesBy
John Wilson
Accreditation and Metrology Services (Pty) LTD
John Wilson 2019 1
Introduction• This presentation is presented in conjunction
with the presentation by Steve Sidney who
has covered the new aspects in the 2017
version of ISO/IEC 17025 related to RISK.
• It incorporates some of the material covered
in a previous NLA Test & Measurement
conference but expands on the Decision Rule
and Statements of Conformity now
incorporated in the ISO/IEC 17025:2017 as
well as a summary of what is in the New ILAC
G8:2019 guidance document.
John Wilson 2019 2
What is a decision rule• Let’s start with what is a decision?
John Wilson 2019 3
What is a decision rule• Let’s start with what is a decision?
John Wilson 2019 4
What is a decision rule• Let’s start with what is a decision?
John Wilson 2019 5
What is a decision rule• Let’s start with what is a decision?
John Wilson 2019 6
What is a decision rule• Let’s look at some metrology (measurements)
Years
Where will the next measurement probably be?
John Wilson 2019 7
What is a decision rule• Let’s look at some metrology
Years
Yes Likely
John Wilson 2019 8
What is a decision rule• Let’s look at some metrology
Years
Not likely but possible
John Wilson 2019 9
What is a decision rule• Let’s look at some metrology
Years
Are these measurements “in Specification” ?
John Wilson 2019 10
What is a decision rule• Let’s look at some metrology
Nominal
Are these measurements“in Specification” ?
Before we decide, we need to know the limits (specification)
John Wilson 2019 11
What is a decision rule• Let’s look at some metrology
Nominal
So are the measurements “in Specification” ?
Upper Limit
Lower Limit
John Wilson 2019 12
What is a decision rule• In the 2005 version of ISO/IEC 17025 it states
in clause 5.10.4.2 “When statements of conformance are made, the uncertainty of measurement shall be taken into account”
• A statement of conformance is something
like “The calibration of this instrument shows
that all the readings are within the
manufacturer’s specifications” or “This
instrument complies with the requirements of
the XYZ standard”.
John Wilson 2019 13
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
Lower Limit
On this graph, the uncertainty of measurement has not been taken into account.
John Wilson 2019 14
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
Lower Limit
The Uncertainty bars are now included
John Wilson 2019 15
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
Lower Limit
So are these measurements “in Specification”?
John Wilson 2019 16
What is a decision rule• Let’s look at some metrology
Nominal
Are these measurements “in Specification” ?
Upper Limit
Lower Limit
John Wilson 2019 17
What is a decision rule• Let’s look at some metrology
Nominal
Three different groups of measurements.
Upper Limit
Lower Limit
John Wilson 2019 18
What is a decision rule• Let’s look at some metrology
Nominal
So are the measurements“in Specification” ?
Upper Limit
Lower LimitPass
Fail
Undecided
John Wilson 2019 19
What is a decision rule• Let’s look at some metrology
Nominal
So does this meet the requirements of ISO/IEC 17025?
Upper Limit
Lower LimitPass
Fail
Undecided
John Wilson 2019 20
What is a decision rule• Let’s look at some metrology
Nominal
The undecided could however lead to lots of arguments.
Upper Limit
Lower LimitPass
Fail
Undecided
John Wilson 2019 21
What is a decision rule• Let’s look at some metrology
Nominal
The undecided could however lead to lots of arguments.
Upper Limit
Lower LimitPass
Fail
Undecided
Readings 1 to 4 are clearly “pass” and within Specification.
Reading 7 is clearly “fail” and outside of Specification.
It is readings 5 and 6 that cause the arguments.
John Wilson 2019 22
What is a decision rule• Let’s look at some metrology
Nominal
The undecided could however lead to lots of arguments.
Upper Limit
Lower LimitPass
Fail
Undecided
If you are the user, then the undecided are not acceptable
but if you are the supplier you could argue that that they
are acceptable because of the uncertainty associated with
the readings.
John Wilson 2019 23
What is a decision rule• Let’s look at some metrology
Nominal
The undecided could however lead to lots of arguments.
Upper Limit
Lower LimitPass
Fail
Undecided
In this case we have a series of measurements over a period
of time so we have an understanding of how the artefact
behaves and can reasonably predict what will happen in the
future. This reduces the risk involved in a decision.
John Wilson 2019 24
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
Lower LimitSimilarly – if we had several of the same instruments, such
as a production run, and get the above results, we know a
lot more about the behaviour of the products
John Wilson 2019 25
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
If we had only one (or one set) of measurements,
would we view it any differently?
Is this a pass or fail?
There may be a lot of money or risk resting on the decision.
John Wilson 2019 26
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
In the past many cal labs would not make a decision based
on this information but would rather pass the information to
the customer to decide for themselves.
So this would not be a statement of conformance but merely
a calibration report.
John Wilson 2019 27
What is a decision rule• Let’s look at some metrology
Nominal
Upper Limit
If however the lab was tasked to give a statement
of compliance, what would the decision be?
Clearly it depends who’s side you are on.
Different labs could give different decisions.
So the customer needs to tell you or be told what
“decision rule” you applied to the statement of
compliance.
John Wilson 2019 28
What is a decision rule• The wording of ISO/IEC 17025:2005 leaves it
very vague and open to a wide range of
interpretation.
• The old ILAC-G8:2009 tried to provide some
clarity by writing the guideline on how to look
at pass/fail conformity assessment.
• The JCGM 106:2012 “Evaluation of
Measurement Data- the Role of measurement
uncertainty in conformance assessment is a
further guideline on how to deal with the
problem.
John Wilson 2019 29
What is a decision rule• In the 2005 version of ISO/IEC 17025 it states
in clause 5.10.4.2 “When statements of
conformance are made, the uncertainty of
measurement shall be taken into account”
John Wilson 2019 30
What is a decision rule• The ILAC G8:2009 tried to provide some
clarity by writing this guideline on how to look at pass/fail conformity assessment.
• Compliance: If the specification limit is not breached by the measurement result plus the expanded uncertainty with a 95% coverage probability, then compliance with the specification can be stated (See Case 1 of Fig.1). This can be reported as “Compliance” or “Compliance – The measurement result is within (or below) the specification limit when the measurement uncertainty is taken into account”. In calibration this is often reported as “Pass”;
John Wilson 2019 31
What is a decision rule• The ILAC G8:2009 tried to provide some
clarity by writing this guideline on how to look at pass/fail conformity assessment.
• Non-compliance: If the specification limit is exceeded by the measurement result minus the expanded uncertainty with a 95% coverage probability, then noncompliance with the specification can be stated. (See Case 4 of Fig.1) This can be reported as “Non-compliance” or “Non-compliance – The measurement result is outside (or above) the specification limit when the measurement uncertainty is taken into account”. In calibration this is often reported as “Fail”;
John Wilson 2019 32
What is a decision rule• The ILAC G8:2009 tried to provide some clarity by
writing this guideline on how to look at pass/fail conformity assessment.
• If the measurement result plus/minus the expanded uncertainty with a 95 % coverage probability overlaps the limit, it is not possible to state compliance or non-compliance. The measurement result and the expanded uncertainty with a 95 % coverage probability should then be reported together with a statement indicating that neither compliance nor non-compliance was demonstrated.
• This is what most SANAS labs used in the past if called on for a certificate of conformance or a statement of compliance.
John Wilson 2019 33
What is a decision rule• The ILAC G8:2009 approach can thus still be
used by the laboratories as a guide to their default decision rule.
• Compliance: If the specification limit is not breached by the measurement result plus the expanded uncertainty with a 95% coverage probability, then compliance with the specification can be stated. This can be reported as “Compliance” or “Compliance – The measurement result is within (or below) the specification limit when the measurement uncertainty is taken into account”.
• In calibration this is often just reported as “Pass”;
John Wilson 2019 34
What is a decision rule• The JCGM 106:2012 “Evaluation of
Measurement Data - the Role of measurement
uncertainty in conformance assessment” is a
very comprehensive document.
• The document address a lot more detail and
gives different ways of interpreting results
• As this is a fairly complicated document ILAC
has revised the ILAC-G8:09/2019 document to
assist laboratories (and assessors) on
understanding the requirements of ISO/IEC
17025:2017
John Wilson 2019 35
What is a decision rule• So the bottom line is that much more
knowledge on the subject is required to enter
into an educated discussion with the
customer about the probability of “False
accepts” when a product should fail and
“False rejects” when a product should pass.
• It is suggested that a good study of JCGM
106:2012 is made.
• It is available free off the BIPM web site
• There is also a lot more information available
on the Web related to this subject.
John Wilson 2019 36
What is a decision rule• So why all the fuss? Why did ISO/IEC
17025:2017 address this matter differently?
• It became clear that there were significant
RISK issues at stake and more clarity was
required to protect both the Laboratory User
and the Laboratory itself.
• Again, it depends if you are the equipment
user or the supplier and it boils down to the
issue of the Risk related to the probability of
“False Accept” or “False Reject”. What is the
Risk to all parties?
John Wilson 2019 37
What is a decision rule
• So what is in ISO/IEC 17025:2017 ?…. Definitions …
• Clause 3.7: a decision rule is defined as “a rule that
describes how measurement uncertainty will be
accounted for when stating conformity with a specified
requirement”.
…. Resource requirements – Personnel …
• Clause 6.2.6 requires that the laboratory shall authorize
personnel to perform “analysis of results, including
statements of conformity or opinions and interpretations”.
John Wilson 2019 38
What is a decision rule
• So what is in ISO/IEC 17025:2017 ?…. Review of requests, tenders and contracts …
• Clause 7.1.3 requires that “When the customer requests
a statement of conformity to a specification or standard
for the test or calibration (e.g. pass/fail, in-tolerance/out-
of-tolerance), the specification or standard and the
decision rule shall be clearly defined. Unless inherent in
the requested specification or standard, the decision rule
selected shall be communicated to, and agreed with, the
customer.”
John Wilson 2019 39
What is a decision rule• So what is in ISO/IEC 17025:2017 ?….Specific requirements for test reports ….
….test reports shall, …. for the interpretation of test results, include ..
• Clause 7.8.3.1b) states “where relevant, a statement of conformity with requirements or specifications” and
• Clause 7.8.3.1c) states “where applicable, the measurement uncertainty presented in the same unit as that of the measurand or in a term relative to the measurand (e.g. percent), when it is relevant to the validity or application of the test results, when a customer's instruction so requires, or when the measurement uncertainty affects conformity to a specification limit”.
John Wilson 2019 40
What is a decision rule
• So what is in ISO/IEC 17025:2017 ?….Specific requirements for calibration certificates ….
…..calibration certificates shall include …..
• Clause 7.8.4.1a) states “the measurement uncertainty of
the measurement result presented in the same unit as
that of the measurand or in a term relative to the
measurand (e.g. percent).”
• Clause 7.8.4.1e) also states “where relevant, a
statement of conformity with requirements or
specifications”.
John Wilson 2019 41
What is a decision rule
• So what is in ISO/IEC 17025:2017 ?……Reporting statements of conformity …
• Clause 7.8.6.1 states “When a statement of conformity
to a specification or standard for test or calibration is
provided, the laboratory shall document the decision rule
employed, taking into account the level of risk (such as
false accept and false reject and statistical assumptions)
associated with the decision rule employed and apply
the decision rule.”
John Wilson 2019 42
What is a decision rule
• So what is in ISO/IEC 17025:2017 ?……Reporting statements of conformity …
• Clause 7.8.6.2 requires that “the laboratory shall report
on the statement of conformity, such that the statement
clearly identifies:
• a) to which results the statement of conformity applies;
• b) which specifications, standard or parts thereof are met
or not met;
• c) the decision rule applied (unless it is inherent in the
requested specification or standard).”
John Wilson 2019 43
What is a decision rule• So breaking this down further
When statement of conformity to a specification or
standard for test or calibration is requested, the
laboratory shall:
• The key here is “when it is requested” which
implies it is requested by the customer.
• This means that “contract review” must take
place and a clear definition agreed on
BEFORE the job is started.
John Wilson 2019 44
What is a decision rule• During Contract review
The laboratory needs to be prepared to discuss
what the decision rule options are regarding the
compliance statement.
They also need to understand what the customer
may require and where the risk of “false accept” or
“false reject” lies so that they do not get drawn into
legal battles or compensation claims.
As with all statistical analysis, it is open to different
interpretations and care must be taken to ensure a
correct agreement between parties.
John Wilson 2019 45
What is a decision rule• During Contract review
There are many models which may be applied
and they will vary greatly depending on the
application.
You may be testing a new prototype where no
previous or group history is available or you
may be checking a widely manufactured and
accepted item against a manufacturer’s
specification. The approach will be very
different.
John Wilson 2019 46
What is a decision rule• Different approaches to each condition
John Wilson 2019 47
One instrument
over a long time One batch of the same model instrument
One reading (or set of readings on a prototype
What is a decision rule• Different approaches to each condition
• A knowledge of “Guard Bands” and the way they are
implemented is advisable.
John Wilson 2019 48
Nominal
Upper Limit
Lower Limit
New Limit
New Limit
Guard band
Guard band
What is a decision rule• Different approaches to each condition
• Guard bands and Uncertainties
John Wilson 2019 49
Nominal
Upper Limit
Lower Limit
Guard band Limit - upper
Guard band Limit - lower
What is a decision rule• ILAC G-8:09/2019 covers decision rules and
statements of conformity and tries to capture and explain in one document the new requirements of ISO/IEC 17025:2017.
• Topics covered in ILAC-G8 include:-o ISO/IEC 17025: 2017 references to decision rules and statements
of conformity
o Measurement Uncertainty and Decision Rule overview
o Guard Bands and Decision Rules
o Taking measurement Uncertainty into account
o Decision Rule Selection Flow Chart
o Decision Rule Documentation and Application
o Simple Checklist (for Labs and Assessors)
o Examples of Decision Rules
John Wilson 2019 50
What is a decision rule• Decision Rule Flow Chart (from ILAC-G8)
John Wilson 2019 51
John Wilson 2019 52
What is a decision rule• During Contract review
Once agreed and documented with the customer –
the work can continue.
John Wilson 2019 53
What is a decision rule• In the Lab… the laboratory shall:
a) document the decision rules employed taking into account the level of risk
associated with the decision rule employed (false accept and false reject
and statistical assumptions associated with the decision rule employed);
• Take the time to clearly document what has
been agreed and ensure that lab procedures
allow for the correct treatment of the
readings and uncertainties.
John Wilson 2019 54
What is a decision rule• In the Lab
… the laboratory shall:
a) document the decision rules employed taking into account the level
of risk associated with the decision rule employed (false accept and
false reject and statistical assumptions associated with the decision
rule employed);
• Might need to change test procedures to
cater for the decision rule that will be
applied. It has to be such that anybody in the
lab will know what is required and what the
limits are.
John Wilson 2019 55
What is a decision rule• In the report
… the laboratory shall:
a) document the decision rules employed taking into account the level
of risk associated with the decision rule employed (false accept and
false reject and statistical assumptions associated with the decision
rule employed);
b) apply the decision rule .
• The report must contain either the decision
rule used or a reference to a standard or
agreed decision rule
John Wilson 2019 56
What is a decision rule• In the report
… the laboratory shall:
a) document the decision rules employed taking into account the level
of risk associated with the decision rule employed (false accept and
false reject and statistical assumptions associated with the decision
rule employed);
b) apply the decision rule .
• The appropriate statement of conformity
which has been requested or required by a
standard.
John Wilson 2019 57
What is a decision rule• In the report
• There must be enough detail so that if
another lab is tasked with taking the same
measurements and using the same decision
rule, that they will come to the same
conclusion about a “pass” or “fail”
John Wilson 2019 58
What is a decision rule• The measurement decision RISK
… the laboratory shall:
a) document the decision rules employed taking into account the level
of risk associated with the decision rule employed (false accept and
false reject and statistical assumptions associated with the decision
rule employed);
• There are two classifications of
measurement decision risk.
• They are Specific and Global risk.
• These are different to the Consumer and
Producer risk already discussed.
John Wilson 2019 59
What is a decision rule• The measurement decision RISK
• If a laboratory only measures a single instrument and has no history of calibration for that serial number or has no information on the behaviour of that model number as a population, then it can be considered as having “meagre prior information” it is classified as Specific Risk.
• When calibration records are actively managed by the customer or laboratory for an instrument or population, they can be directed as Global Risk.
John Wilson 2019 60
What is a decision rule• The measurement decision RISK
• Similarly, as discussed earlier, there is the
consumer risk approach and the producer
risk.
• It follows that when using binary decision
rules to reduce consumers risk, it will always
increase the producers risk.
• JCGM 106 has a lot of information on these
various risk based approaches and shows
how to calculate the appropriate risk involved
with a specific decision rule.
John Wilson 2019 61
What is a decision rule• In Summary – it is the balance of risk
John Wilson 2019 62
What is a decision rule• The measurement decision RISK• Taken from ILAC-G8 (in this table U = U95)
John Wilson 2019 63
Appendix B of ILAC-G8 09/2019 give more examples.
What is a decision rule• In general
• This means that if a certificate or statement of conformance is required, a lot more detail is required to be addressed before the measurements are taken and then the Lab will need to be vey clear about the criteria used and this must be defined on the test report, certificate of conformance or statement of compliance.
• The following Checklist from ILAC-G8 09/2019 (Appendix A) gives a simple guideline to follow to ensure that all the key issues are covered.
John Wilson 2019 64
What is a decision ruleThe checklist from ILAC-G8 09/2019
APPENDIX A - Sample Checklist for meeting the requirements of ISO/IEC
17025:2017
a)Documentation and records reflecting customer
agreement requesting statement of conformity to a
specification or standard. (clause 7.1.3 [1])
b)Records of the selection of test limits and associated
tolerances and compatibility with customer
requirements (clause 7.1.3 [1])
John Wilson 2019 65
What is a decision ruleThe checklist from ILAC-G8 09/2019
APPENDIX A - Sample Checklist for meeting the requirements of ISO/IEC
17025:2017
c)A documented decision rule for calculating,
controlling, and reporting levels of risk associated
with statement of conformity. (clause 7.1.3 [1])
d)Documentation of laboratory personnel that include
the knowledge, skill, and authorization to apply the
decision rule and make statements of conformity.
(clause 6.2.6 c [1])
John Wilson 2019 66
What is a decision ruleThe checklist from ILAC-G8 09/2019
e)Documentation of the calculation or estimation of
level of risk and measurement uncertainty. (clause
7.8.6.1 [1])
f) Documentation of other supporting factors such as
statistical assumptions including type of risk (i.e.
specific or global) and measurement uncertainty.
(clause 7.8.6.1 [1]) NOTE: for more information on
specific and global risk see section 5.3 of this
document.
John Wilson 2019 67
What is a decision ruleThe checklist from ILAC-G8 09/2019
g)Documentation of the conformity assessment type
and statements of conformance. (clause 7.8.6.2 [1])
NOTE: for more info see section 4.2 of this document
h) Decision rule documentation to be included with
test and calibration records. (clause 7.8.6.2 [1])
John Wilson 2019 68
What is a decision rule• In general
• This is not so difficult if it is broken down in
simple logical steps.
• However it will require laboratories to
seriously consider how they will approach the
whole job when they are requested to supply
a certificate or statement of conformance
• Suggestion:- The laboratory documents a few
decision rules that they are able to offer their
customers.
John Wilson 2019 69
Practical ExampleAssume a 3½ digit DMM has a specification of
±(0.1% of Reading + 2 Digits)
on the 6 Vdc Range.
The tolerance limits for a nominal of 5.900 Vdc
would be:-
Lower Limit 5.892 Vdc & Upper Limit 5.908 Vdc
If zero guard band was applied, the risk of PFA
is < 50%
If a guard band of U95 is applied the risk of PFA
drops to < 2.5%
John Wilson 2019 70
Practical ExampleMass Verification
OIML R76-1:2006 (NAWIs) cl. 3.7.1
where it is required that
“…the standard masses used for the type
examination or verification of an instrument ….
shall not have an error greater than 1/3 of the
MPE. If they belong to class E2 or better, their
uncertainty is allowed to be not greater than
1/3 of the MPE of the instrument (the
tolerance)”
John Wilson 2019 71
Practical ExampleLiquids (other than water)
OIML R117-1:2007 Dynamic measuring systems
for liquids other than water Part 1: Metrological
and technical requirements A.2
Uncertainties of measurement:
When a test is conducted, the expanded
uncertainty of the determination of errors on
indications of volume or mass shall be less than
one-fifth of the maximum permissible error
(MPE) (the tolerance)
John Wilson 2019 72
Practical ExampleCivils
John Wilson 2019 73
• As we have a few Material Testing labs in
the audience lets consider a Mat Civils
problem. (forgive my lack of knowledge wrt Civils tests)
• What is involved with Specific Density
Tests ? Mass measurements
• Similarly with Marshall Control Tests.
• If no uncertainty is applied to any one
measurement, your risk of False Accept
approaches 50% as you get closer to the
tolerance limit.
• So when you then include multiple
measurements, the situation gets worse.
Practical Example• It should be clear from the previous examples
that in some cases there are clear
instructions on what to do.
• In other cases, the laboratory and the
customer will need to decided what decision
rule must be applied.
• In all cases, this relates to the Risk that
needs to be evaluated in an educated
discussion with the customer about the
probability of “False accepts” when a product
should fail and “False rejects” when a
product should pass.
John Wilson 2019 74
What is a decision rule
John Wilson 2019 75
It is all about communication!
ISO/IEC 17025:2017Statement or Certificate of Conformance
&Decision rules
By
John Wilson
Accreditation and Metrology Services (Pty) LTD
John Wilson 2019 76