MEASUREMENT OF UNCERTAINTY IN MICROBIOLOGICAL TESTING

Mjikisile Vulindlu

Scientific Services Department, Water Services, City of Cape Town P O Box 16548 Vlaeberg 8018

Tel: +27 21 684 1000 mjikisile.vulindlu@capetown.gov.za

ABSTRACT In view of pressure put on laboratory results today, testing laboratories need to be aware of the uncertainty of measurement of the tests that they perform. Estimated uncertainty of measurement must for any test result be provided when requested by either the client or the court of law. Uncertainty Measurement allows for an assessment of the reliability of results.

INTRODUCTION Measurement of microbiological parameters in water and food samples is not as simple as it is often perceived. It is imperative for testing laboratories and their clients to understand complications associated with sample test procedures. These complications can be evaluated and documented well in advance by the testing laboratory. Unless this is done the testing laboratory is unable to defend legally the results of tests they perform. It should be borne in mind that measurement is not perfect. It is always associated with variations. These variations are called uncertainty. Uncertainty arises from different sources including errors and imperfection in measurement and reproducibility respectively. An error can be defined as the difference between a measurement and the true value. The challenge in measurement is that the error and the true value are unknown. Uncertainty comes from unknown errors. Testing laboratories need to be aware of the uncertainty of measurement of the tests they perform. A good testing laboratory must be able to provide measurement of estimated uncertainty when requested to do so for any test result. The British Standards Institution (BSI) and United Kingdom Accreditation Service (UKAS) have become aware of this and are now encouraging all microbiological testing laboratories in the UK to express their results with a ‘±’ suffix. Results from testing laboratories become more meaningful when they are expressed with this indication of a possible variation. In other words uncertainty indicates the range within which the true value is likely to fall, (CPHLS, 2002). Let us consider the example given by Voysey and Jewell (1999) of Campden & Chorleywood Food Research Associates, U K. When given a measurement, for example, of log count = 4.51. This figure is unlikely to be a true value. It is also correct to say that a true value is not known and so is the true error. This figure can be expressed with a ‘±’ suffix if the uncertainty was measured from previous experience.

1

Calculations from previous work in similar circumstances may have determined the magnitude of errors. If 95% of the errors were smaller than 0.5 log units, i.e ± 0.5, an implication would thus been given, a 95% confidence that the true log count falls within 0.5 of the measurement. Now the measurement can be reported as log count = 4.5 ± 0.5. Here 0.5 is an expanded uncertainty. One must make sure that a quoted uncertainty is clear, quantitative and justifiable. Generally uncertainties are quoted as a 95% confidence intervals so that about 95% of the time the true value will lie inside the quoted interval, but about 5% will lie outside (CPHL, 2002). More and more polluters of controlled water bodies worldwide face legal court action. And testing laboratories must be able to defend their laboratory results in the court of law when required to do so. This then compels the testing laboratories to produce results that are legally and statistically credible. WHY DO WE MEASURE UNCERTAINTY?????????? Uncertainty measurement demonstrates how well the results represent the value of the quantity being measured. It also allows for assessment of the reliability of a result, for example in comparing the results from different sources or samples with a reference value. In cases where the uncertainty of measurement or test results is considered too small to be evaluated, this needs to be justified with a formal evaluation. Sometimes samples have inconsistency in their property and this results in them having large uncertainty. In such case it may be decided that relatively large uncertainty of measurement be ignored due to uncertainty attributed to the sample variation. This assertion cannot be done unless an estimate of the uncertainty of the measurement and the relevant sample variation has been made. The uncertainty figures can be required by either the client or court of law for the interpretation of a result against a legal standard. This data can be used for the competence testing of the trainee staff as well as students in their experiential training. SOURCES OF UNCERTAINTY The overall uncertainty of a measurement arises from numerous components. Uncertainty can be as result of sampling. For examples where samples are taken from one spot. These will not given same results even when analysed under similar conditions. The other example that applies to water samples is when the sample is perfectly mixed. The rationale for sample mixing is to ensure that colony forming units are uniformly spread throughout the samples. In other words perfect mixing ensures equal probability of colony forming units being in each sub-section of the sample. Even under these conditions colony forming units will not be perfectly distributed throughout the sample. This is an unavoidable uncertainty especially when the entity that is being measured is too small. Unavoidable sampling uncertainty can be estimated by the Poisson Distribution. Sometimes uncertainty comes from other errors, normally not associated with microbiological testing. Bacterial culturing has changed dramatically over recent years when emphasis is now put on enzyme substrate technology. This differs from the use of

2

solid/liquid media where the ultimate aim is to get colony forming units. For example new methods such as the Idexx Quanti-tray (MPN method) uses substrate reagents which target specific enzyme in micro-organisms. The preparation of these as well as the preparation of the conventional culture media can contribute in the uncertainty. For example: • Errors due to graduation of the instrument scale, limits of measurement resolution or

threshold of discrimination and operators bias in reading instruments • Values assigned to standards and reference instrument • Changes in characteristics or the performance of equipment since the last calibration • Approximations and assumptions incorporated since the last calibration including

values of constants and other parameters used in the test • Fluctuation in local environmental conditions such as temperature, humidity and air

pressure, or variability in the performance of the individual test and inadequate knowledge of the effect of environmental conditions on the measurement process

EXAMPLES OF UNCERTAINTY COMPONENTS Component Measur

e Control Monitor Can be included in repeatability-

reproducibility Pippetting, plating, diluting

√ √ ⋅ √

Weighing, materials √ √ ⋅ √ Competition ⋅ ⋅ ⋅ √ Incubation time & temperature

√ √ √ √

Media √ √ √ √ Analysts, groups √ √ ⋅ √ Laboratory/group bias √ √ ⋅ ⋅ Matrices √ √ ⋅ √ HOW TO CALCULATE UNCERTAINTY OF MEASUREMENT The laboratory must decide which data should be used to calculate the uncertainty. The calculation of uncertainty is based on the concept of repeatability testing. A good place to start is to use data from External/Internal Quality Assurance scheme. The samples can also be tested in replicates but this does not need to apply to all the samples being tested. The laboratory can chose to replicate every tenth samples. It is advisable to involved more than one analyst when replicate testing is carried out. The Central Public Health Laboratory (CPHL), UK, argues that the use of replicate samples data demonstrates closeness of agreement between the results of successive measurement of the same sample measured under the same condition of measurement. Samples can also be spiked to obtain the required data. Where spiking of samples is used, it is a good practice to use recovery media that will enable statistically accurate counting of colonies. Usually in the range of 20-80 colony forming units (cfu) on a membrane or 100 – 200 cfu on solid culture media. The measurement of uncertainty will take into account the variability of the collated data called variance. The variance is a mean of squared deviation and it provides quantified

3

information about the variability in a set of data (Neter et.al.,1988). This measure allows variability that exists due to repeatability-reproducibility testing of samples. Reproducibility: Closeness of the agreement between the results of successive

measurements of the same measurand under the same conditions of measurement.

Repeatability: Closeness of the agreement between the results of measurements of

the same measurand carried out under changed conditions of measurement.

SAY WHAT AND WHEN The laboratory must state clearly the frequency of variance reviews as well as the review of standard deviation. This can be presented in the form of a table so that the laboratory staff will know the frequency and data to work from. Clients and the accreditation assessors will have more confidence on the data produced by the laboratory. SAMPLES THAT BE USED TO CALCULATE THE UNCERTAINTY OF THE TEST PROCEDURE Test Data Used Review of

Pooled Variance Review of Pooled Standard Deviation

Indicator Organisms eg Total colifroms, faecal coliforms & E.coli

IQC/ spiked samples Every month Annual

Pathogens Spiked samples validation of SOP

Every three months

Annual

Most Probable Number eg. Idexx Quanti-Tray

IQC and spike samples

Every month Annual

Heterotrophic Plate Count

EQA/IQC Every two months

Annual

It is also useful for laboratories to set up control procedures for sources of variability in tests. This indication gives the implication of the laboratory’s commitment in keeping these variability to a minimum. CONTROL PROCEDURES FOR SOURCES OF VARIABILITY IN TEST FOR DETECTION, CPHL, 2002 Test Procedure Sources of Variability Approach Detection of pathogens Appropriate matrices Use different sample types

for spiking Reading biochemical tests or commercial kits

Interpretation of the tests Use of manufacturers’ instructions or colour chart

Media and Reagents Batch variation Quality control of batch

4

CONCLUSION Setting up of these procedures should help minimise variations that exist because of: • Variations between workers when processing samples • Hope to mimimise variation due to the measurement of sample volume/weight. This

arises because of the operator and imprecision of equipment • Use of different batch media influence growth of microorganisms. Comparisons

between old and new batch should be viewed as a good tool to determine variations in culture media.

• Error of measurement and fluctuations in temperature of incubator/waterbath. • Variation between workers when counting colonies. This arises due to the

interpretation of the analyst eg. colour change or description of morphology. REFERENCE: 1. Central Public Health Laboratory, Standard Operating Procedure “ Uncertainty of

Measurement in Testing” 2002 2. Central Public Health Laboratory, Standard Operating Procedure “ Procedure for

Measurement in Testing in Laboratory ” 2002 3. Methods for Microbiological Examination of Food and Animal Feeding Stuffs: General

Laboratory Practices. BS 5763: Part 0: 1996. ISO 7218 : 1996 4. Neter J et al., (1988) Applied Statistic 3rd Edition, Allyn and Bacon, Inc 5. PHLS Guidance note “ Uncertainty of measurement in testing “ 1998 6. Vocabulary of Methodology. Part 3. Guide to the Expression of Uncertainty. BSI PD

6461 : Part 3 : 1995. 7. Voysey P. A. and Jewell K (1998) Uncertainty Associated with Microbiological

Measurement. CCFRA Review No.15: Project No. 29732Campden & Chorleywood Food Research Association

5