Interlaboratory Tests

In: Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

Koch, M.: Interlaboratory Tests © Springer-Verlag Berlin Heidelberg 2003

Interlaboratory Tests

Michael Koch



What are interlaboratory tests?

Randomly selected sub-samples from a source of material are distributed simultaneously to participating laboratories for concurrent testing



Types of interlaboratory tests

method validation

reference material characterization

proficiency testing



Interlaboratory tests for the validation of a method

objective: best possible characterization of the method

laboratories have to use exactly the same method

assistance should be given to assure this



Interlaboratory tests for characte-rization of a reference material

concentration of the analyte in the material must be analysed by experienced laboratories

less experienced laboratories should not be allowed to participate

objective: best possible estimation of the “true value” of the concentration



Interlaboratory tests for proficiency testing of laboratories

objective: to get an indication of the performance of an individual laboratory or a group of laboratories as a whole

laboratories should work under routine conditions

help for the laboratory to improve its quality can be used by customers or regulatory

bodies for the selection of qualified laboratories



Objectives of proficiency tests

basic concern is accuracy inaccuracy contains systematic and

random effects laboratory can determine, whether

imprecision or bias is the reason for its inaccuracy



Motivation for the laboratories

to uncover errors that couldn’t be found with internal quality control

use as certificate for competence in this testing field for clients, authorities and accreditation bodies



Limitations

Interlaboratory tests are always retrospective organisation, distribution of samples, analyses,

evaluation take time it is dangerous to rely only on interlaboratory tests

Proficiency tests cover only a small fraction of the often wide variety of analyses

Proficiency tests do not reflect routine analyses



Standards and guidelines for proficiency testing - I

ISO Guide 43: Proficiency testing by interlaboratory comparisons Part 1: Development and operation of proficiency

testing schemes. Part 2: Selection and use of proficiency testing

schemes by laboratory accreditation bodies.

IUPAC, ISO, AOAC (1991): The International Harmonized Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories.



Standards and guidelines for proficiency testing - II

International laboratory accreditation cooperation (ILAC): Guidelines for the requirements for the competence of providers of proficiency testing schemes.

Draft ISO 13528: Statistical Methods for the Use in Proficiency Testing by Interlaboratory comparisons.



Demands on the provider -personnel

special organizational capabilities technical experts for the analysis statisticians all staff have to be competent for the

work it is responsible for



Demands on the provider –Planning - I

The interlaboratory test should be carefully prepared.

The planning must be documented before the start of the test



Demands on the provider –Planning - II

The plan should typically include: name and address of the PT provider name and address of the coordinator and other

personnel nature and purpose of the PT scheme procedure for the manner in which the participants

are selected or criteria which have to met before participation is allowed

name and address of the laboratory performing the scheme (e.g. sampling, sampling processing, homogeneity testing and assigning values) and the number of expected participants.



Demands on the provider –Planning - III

Planning content (contd.): nature of the test items and of the tests selected description of the manner in which the test items

are obtained, processed, checked and transported.

description of the information that is supplied to participants and the time schedule for the various phases.

information on methods or procedures which participants may need to use to perform the tests or measurements.



Demands on the provider –Planning - IV

Planning content (contd.): outline of the statistical analysis to be used. description of how the assigned value is

determined. description of the data or information to be

returned to participants. basis techniques and methods used for evaluation description of the extent to which the test results

are to made public.



Demands on the provider –Data-processing equipment

Equipment should be adequate for data processing statistical analysis to provide timely and valid results

Software must be verified and backed up



Demands on the provider – Test item preparation and management - I

For the selection of the test item all characteristics that could affect the integrity of the interlaboratory comparison should be considered homogeneity stability possible changes during transport effects of ambient conditions (e.g.

temperature)



Demands on the provider – Test item preparation and management - II

samples used in the proficiency test should be similar to the samples that are routinely analysed in the laboratories

sample amount surplus of sample can be used as

reference material surplus can be used to make excessive

effort on the analyses



Demands on the provider -Homogeneity - I

The PT provider has to ensure that every laboratory will receive samples that do not differ significantly in the parameters to be measured documented procedure for establishing this

homogeneity degree of homogeneity evaluation of the

laboratories results must not be significantly affected

any variation between the portions must be negligible in relation to the expected variations between the participants



Demands on the provider -Homogeneity - II

true solutions are homogeneous at a molecular level

for solid samples special care on the homogenisation

a formal homogeneity check is described in the „International harmonized protocol...“



Demands on the provider -Stability - I

test material must be sufficiently stable under the conditions of storage and

distribution to the participants for the time period from producing the

samples until the analyses in the participant’s laboratory

this stability has to be tested by the PT provider



Demands on the provider -Stability - II

Analysing a part of the samples after the estimated time necessary for the distribution differences in the results may be due to instability

or to between-batch variability in the organiser’s laboratory

information may be derived from the organiser’s prior experience or obtained from technical literature

accelerated stability testing by worsening the ambient conditions for the samples



Demands on the provider -Stability - III

the organiser has to ensure that the changes due to instability do not significantly affect the evaluation of the laboratories’ performance



Choice of analytical method

Normally the laboratory should use its routine method

the choice might be limited by e.g. legal regulations

organiser should ask for details to conduct a method specific evaluation to give comments on the methods used



Method specific evaluation

too low low correct high too high

0

10

20

30

40

50

60

70

80

ICP-MS

GFAAS

ICP-OES

Photometry

aluminum in groundwater



Determination of the assigned value

one of the most critical features of a proficiency test

inappropriate value will drastically reduce the value of the scheme

the same problem as in the certification of a reference material but the organiser of a proficiency test

cannot expend the same amount of effort



Assigned value –certified reference material

ideal test material for a proficiency test disadvantages

high costs limited availability

in the necessary quantity and concentration range

CRM‘s have to be stable for months and PT often deals with more or less instable samples (foodstuffs, biomedical, environmental samples)



Assigned value –Consensus of „Expert Laboratories” - I

mean of analysis by expert laboratories with high precision reference methods and

traceable materials for calibration if different physico-chemical methods are

used and the same results are obtained, it is more probable that the value is near to the „true“ value



Assigned value –Consensus of „Expert Laboratories” - II

disadvantages very much effort to ensure the accuracy of

the reference measurements „nobody is perfect“

there might be doubts among the participants if the result of the expert laboratories deviates from the mean of the participants



Assigned value – Formulated or “Synthetic” test materials - I materials, spiked with the analyte to a known

extent can be made with extremely accurate amounts by

gravimetric or volumetric methods If material does not contain significant amounts

of the analyte assigned value directly from added amount

If material contains analyte, this amount has to be characterized very well. method to calculate this content from proficiency test

was recently developed by the author



Assigned value – Formulated or “Synthetic” test materials - II

disadvantages difficult to achieve sufficient homogeneity,

especially with solid materials analyte might be bound in a different

chemical form Especially in solid materials the originally

contained analyte might be bound more strongly to the matrix



Assigned value –Consensus of participants - I

easiest and cheapest way used very often

If method for analysis is easy and straightforward good estimate of „true“ value

If a „convention method“ (an empirically defined method) is used, the consensus value is the only possibility



Assigned value –Consensus of participants - II

disadvantages Consensus value might be seriously biased (e.g.

analyses of highly volatile substances) there might be no consensus at all

e.g. if two analytical methods are used, where one is biased

these circumstances are not uncommon in trace analysis

care should be taken to decide whether a consensus value really is good choice



Methods to calculate consensus value – Arithmetic mean

requires an outlier test but these tests are often not very

satisfactory, especially if many outliers are present

outlier tests assume normal distribution which is normally not true in trace analysis



Methods to calculate consensus value – Median

not affected by outlying data makes not full use of the information

content of the data if the distribution is skewed, the median

is biased



Methods to calculate consensus value – Robust mean

„trimmed“ data; a certain part of the data on both tails of the data set is excluded prior to the calculation of the mean

e.g. mean of interquartile range mean of data between the first and the

third quartile of the data set or Huber statistics



Methods to calculate consensus value – Robust mean – Huber statistics

Iterative process define initial value for m as median of all

data all data outside m±1.5*STD are set to

m+1.5*STD or m-1.5*STD new value for m is calculated as arithmetic

mean of the new data repeat until there are no changes



Performance scoring

assigned value is the target for the assessment of laboratories a

accepted range is necessary prescribed range originating from the

demands put on the analysis (fitness for purpose)

calculated from the standard deviation of the data set



Performance scoring –Tolerance range from STD

normally distributed set of data 95,5% of the values inside a range of ±2σ 99,7% of the values inside a range of ±3σ

on a confidence level of 95,5 % all accurate data are inside µ±2σ



Performance scoring – Z-score

the deviation from the assigned value in standard deviation units

the standard deviation is calculated after exclusion of outlier or with robust statistics

s

xscorez

)(



Performance scoring – Classification of the Internat. Harmonized Protocol

|Z-score|2 - satisfactory 2<|Z-score|3 - questionable |Z-score|>3 - unsatisfactory

Z-scores are common practise in the assessment of laboratory results



Z-score - diagramm

-15

-10

-5

0

5

10

75 39 40 36 34 20 71 56 3 85 2 59 43 25 54 90 1 80 44 76 64 29 79 73 15 7 57 32 37 72 52 10 17 95 63 49 4

laboratory code

Z-s

core



Statistical distribution

Data near to the limit of determination are not normal distributed

otherwise there should be negative values with a finite probability

tolerance limits should be asymmetrical (more narrow below the assigned value, more wide above it)



Solution approaches for assymetrical tolerance limits

logarithmic normal distribution take the logarithm of the values prior to

statistical calculations

Modification of Z-scores

2

11222 2

1exp

1

2

1exp

1kkkk

11

1 12

1

kk

= rel. standard deviation

= distribution function of standard normal distribution

1- = confidence level (here: 0,955)

with g = quality limit for Zand k1, k2 =correction factors

0

0

2

1

ZifZk

g

ZifZk

g

Zu



Laboratory assessment

by combination of single value assessment

involves danger of misinterpretation a laboratory can measure one parameter

permanently wrong, but nevertheless is positively assessed



Combined assessment according to Intern. Harmon. Protocol... - RSZ

RSZ (rescaled sum of z-scores) RSZ = z/√m with m = number of scores same scale as z-score negative assessment, if all values are

within the tolerance but a little biased in the same direction

errors with opposite sign cancel each other out



Combined assessment according to Intern. Harmon. Protocol... - SSZ

SSZ (sum of squared z-scores) different scale, because 2-distributed doesn‘t consider the sign of z-scores



Combination of single values assessments

just counting positive and negative assessments of all values

the absolute value of the z-score is not considered

e.g. assessment in the proficiency tests of german water authorities 80 % of the values – |Zu-score|2 80 % of the parameters successful



Test scheme reports

should be distributed to the laboratories as soon as possible normally not later than 1 month after

deadline for the analytical results laboratories need quick feedback for

corrective actions laboratories should be identified in the

report by test specific codes to maintain confidentiality



Test scheme reports – ISO Guide 43 – contents - I

Name and address of the organisation conducting or coordinating the test scheme

Names and affiliations of persons involved in the design and conduct of the test scheme

Date of issue of the report Report number and clear identification of the test

scheme Clear description of item or materials used,

including details of sample preparation and homogeneity testing

Laboratory participation codes and test results



Test scheme reports – ISO Guide 43 – contents - II

Statistical data and summaries, including assigned values and range of acceptable results

Procedures used to establish any assigned value Details of the traceability and uncertainty of any

assigned value Assigned values and summary statistics for test

methods/procedures used by other participating laboratories (if different methods are used by different laboratories)

Comments on laboratory performance by the coordinator and technical advisers



Test scheme reports – ISO Guide 43 – contents - III

Procedures used to design and implement the test scheme (which may include reference to test scheme protocol)

Procedures used to statistically analyse the data

Advice, where appropriate, on the interpretation of the statistical analysis



Certificate

If the proficiency test scheme has regulations for the assessment of the laboratories on the basis of the data (successful / not successful) a certificate should be sent to the laboratory in case of successful participation.

In many cases these certificates are used by the laboratories for demonstrating competence to their customers, i.e. for advertising.



Certificate – example



Confidentiality normally in all PT schemes the identity of all

laboratories are kept confidential public reports about poor performance of a

laboratory in a proficiency test could be the economic ruin of this laboratory

identity should be known only to a small number of persons These persons must be regularly instructed about

there duty to keep this information confidential the coordinating body may be required to report

poor performance to a particular authority participants should be notified of this possibility



Collusion and falsification of results

PT schemes often are not only a help for the laboratories to improve their quality but also a control tool for accreditation bodies, customers and authorities

their may be a tendency among some participants to give a falsely optimistic impression of their capabilities



Collusion

must not be possible

as many concentration levels as possible



Number of multiple measurements

example from reality: routine: single measurement asked in proficiency test:

independent double measurement executed in proficiency test:

40 (!) measurements

therefore: limitation of sample amount, where possible



Level-by-level evaluation for different concentrations

can lead to injusticealuminum

-80-60-40-20

020406080

100120

0,050 0,075 0,100 0,125 0,150 0,175 0,200

concentration in mg/l

tole

ran

ce li

mit

s /

stan

dar

d d

evia

tio

n in

%

a procedure for a common evaluation can be found in a German standard (DIN 38402-45:2003)



Effort for the laboratory

analysis of the samples should not exceed the effort for routine

samples in reality not insignificant

participation fee



Participation fee

normally between 300 US-$ and 1000 US-$ (depending on matrix and parameters)



Benefits - I

regular, external and independent check on data quality

Assistance in demonstrating quality and commitment to quality issues

Motivation to improve performance Support for accreditation/certification to

quality standards Comparison of performance with that of peers



Benefits - II

Assistance in the identification of measurement problems

Feedback and technical advice from organisers (reports, newsletters, open meetings)

Assistance in the evaluation of methods and instrumentation

A particularly valuable method of quality control where suitable reference materials are not available



Benefits - III

Assistance in training staff Assistance in the marketing of analytical

services Savings in time/costs by reducing the

need for repeat measurements A guard against loss of reputation due

to poor performance Increased competitiveness



Benefits - costs

the costs are noticed immediately benefits are difficult to quantify in

monetary terms the succesful participation often is a

important proof of competence and therefore compensate for the costs

of participation

Documents

Interlaboratory Tests