Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/50867783

ValidationofAlternativeMethodsfortheAnalysisofDrinkingWaterandTheirApplicationtoEscherichiacoli

ARTICLEinAPPLIEDANDENVIRONMENTALMICROBIOLOGY·MARCH2011

ImpactFactor:3.95·DOI:10.1128/AEM.00020-11·Source:PubMed

CITATIONS

8

4AUTHORS,INCLUDING:

AbdelkaderBoubetra

InstitutScientifiqued'Hygièneetd'Analyse

6PUBLICATIONS77CITATIONS

SEEPROFILE

MaxFeinberg

FrenchNationalInstituteforAgriculturalR…

76PUBLICATIONS1,165CITATIONS

SEEPROFILE

Availablefrom:AbdelkaderBoubetra

Retrievedon:30August2015

APPLIED AND ENVIRONMENTAL MICROBIOLOGY, May 2011, p. 3360–3367 Vol. 77, No. 100099-2240/11/$12.00 doi:10.1128/AEM.00020-11Copyright © 2011, American Society for Microbiology. All Rights Reserved.

Validation of Alternative Methods for the Analysis of Drinking Waterand Their Application to Escherichia coli�

Abdelkader Boubetra,1* Francois Le Nestour,1 Corrie Allaert,2 and Max Feinberg3

Institut Scientifique d’Hygiene et d’Analyse (ISHA), 25 Ave. de la Republique, 91300 Massy, France1; IDEXX Laboratories,c/ Plom, no. 2-8, 3°, 08038 Barcelona, Spain2; and Institut National de la Recherche Agronomique,

Met@risk, 16 Rue Claude Bernard, F-75231 Paris Cedex 05, France3

Received 5 January 2011/Accepted 13 March 2011

In Europe, the Drinking Water Directive of the European Commission indicates which methods (most ofwhich are CEN/ISO-standardized methods) should be used for the analysis of microbiological parameters(European Commission, Environment, Council Directive 98/83/EC of 3 November 1998). According to theDirective, alternative methods “may be used, providing it can be demonstrated that the results obtained are atleast as reliable as those produced by the methods specified.” The prerequisite for the routine use of anyalternative method is to provide evidence that this method performs equivalently to the correspondingreference method. In this respect, the ISO 16140 standard (ISO, ISO 16140. Microbiology of Food and AnimalFeeding Stuffs—Protocol for the Validation of Alternative Methods, 2003) represents a key issue in generating sucha procedure based on an interlaboratory study. A new statistical tool, called the accuracy profile, has beendeveloped to better interpret the data. The study presented here is based upon the enumeration of Escherichiacoli bacteria in water. The reference method may require up to 72 h to provide a confirmed result. The aim ofthis publication is to present data for an alternative method by which results can be obtained in 18 h(Colilert-18/Quanti-Tray) based upon defined substrate technology (DST). The accuracy profile is a statisticaland graphical decision-making tool and consists of simultaneously combining, in a single graphic, � expec-tation tolerance intervals (�-ETIs) and acceptability limits (�). The study presents the validation criteriacalculated at the three levels of contamination used in the trial for a � equal to 80% and a � equal to �0.3 andcombines the accuracy profiles of Escherichia coli for a � of �0.3 log10 unit/100 ml, a � of �0.4 log10 unit/100ml, and a � of 80% or 90%. Several interesting conclusions can be drawn from these data. The accuracy profilemethod has been applied to the validation of the Colilert-18/Quanti-Tray method against reference method ISO9308-1 (ISO, ISO 9308-1. Water Quality—Detection and Enumeration of Escherichia coli and Coliform Bacteria.Part 1. Membrane Filtration Method, 2000), using a � of 80% and a � of 0.4; the alternative method can bevalidated between 1.00 and 2.05 log10 units/100 ml, equivalent to 10 to 112 CFU/100 ml.

Until now, there has been little formal guidance on proce-dures for adopting alternative methods for determining levelsof microbes in water. From a metrological point of view, thefirst step in developing a procedure is to define the measurand.The measurand itself may be simply defined as “the quantityintended to be measured” (in reference 16, see section 2.3).Due to the nature of microorganisms and the well-recognizedconcept of CFU, the currently accepted method for definingthe measurand and for ensuring traceability in microbiologyconsists of using a reference (or official) method. This is usuallya historic method which has been standardized and is recog-nized as reliable by the community of microbiologists andregulatory bodies. We must keep in mind, however, that mi-crobiological methods, even if they have the status of referencemethods, are based in counts of discrete units, and hence theiruncertainties have intrinsic, unavoidable, stochastic compo-nents that make their values higher than the uncertainties ofmost chemical methods. This is still more accentuated in themost probable number (MPN) methods, where the calculated

result is the mode of a statistical distribution of values, butaround this mode there are values with lower probabilities.Ideally, each MPN should be expressed with a confidence in-terval according to this fact. As microbiological methods areempirical (also called “direct”) analytical techniques, the mea-surand is highly dependent upon the operating procedure.Therefore, the measurand, as defined by a “reference”method, can be somewhat different when determined by an“alternative” method. For practical purposes, however, and toundertake the validation, it is necessary to make this approx-imation. This situation is rather typical when alternative andreference methods are compared.

Taking into account the points highlighted above, the pre-requisite for the commercial retail and routine use of anyalternative method is to provide evidence that this methodperforms equivalently to the corresponding reference method.To provide such evidence, the manufacturers of alternativeproprietary kits, the food and beverage industry, the publichealth services, and other authorities require a reliable andcommonly agreed upon procedure for the validation of suchalternative methods. In this respect, the ISO 16140 standard isa key issue in generating such a procedure (13).

As suggested in its title, ISO 16140 (13) separately proposesvalidation protocols for both quantitative and qualitative meth-ods (1). An interesting requirement of the standard for quan-

* Corresponding author. Mailing address: Institut Scientifiqued’Hygiene et d’Analyse (ISHA), 25 Ave. de la Republique, 91300Massy, France. Phone: 33 1 69 75 45 46. Fax: 33 1 64 47 15 44. E-mail:aboubetra@isha-analyse.fr.

� Published ahead of print on 25 March 2011.

3360

titative method validation is the organization of an interlabo-ratory study in accordance with the recommendations of ISO5725 (15a). Some years ago, microbiologists were reluctant toparticipate in collaborative studies due to the assumed insta-bility of samples. Much improvement has occurred with regardto stabilization of samples, and interlaboratory studies are nowcommonly used for proficiency testing programs. This classicalcollaborative approach can now also be applied to methodvalidation in microbiology; sample instability and organiza-tional constraints are no longer an issue (2, 4, 12).

The classical statistical strategy employed for the interpre-tation of data in many validation procedures is based largely onnull-hypothesis testing. This type of analysis aims to demon-strate that an alternative method does not produce resultssignificantly different from those of the reference method. Thisstrategy presents many drawbacks that have been describedextensively in recent publications (7, 18). The most strikingobservation was that the more uncertain the results obtainedby an alternative method are, the easier the validation is. Forthis reason, a new statistical tool, called the accuracy profile,has been developed to better interpret the data of a validationstudy, so that misleading conclusions are avoided. This meth-odology has been extensively applied to chemical analyticalmethods and, more recently, in the field of food microbi-ology (8).

The study presented here is based upon the enumeration ofEscherichia coli bacteria in water. The choice to use the accu-racy profile for this parameter is based largely upon recentchanges introduced by new European regulations and subse-quently by regulation 2073/2005 of the Commission of theEuropean Communities and its amendments for microbiolog-ical criteria (3).

The objective of these regulations is to ensure that drinkingwater is free of pathogens such as viruses, protozoa, and bac-teria. Waterborne pathogens cause diseases such as hepatitis,giardiasis, and dysentery. The analysis of water for the pres-ence of specific harmful viruses, protozoa, and bacteria is time-consuming and thus expensive. In addition, not all analyticallaboratories are equipped and approved to proficiently per-form the required testing. Water testing for the presence ofspecific organisms is therefore limited to investigating specificwaterborne disease outbreaks.

E. coli and coliform bacteria are a broad class of bacteriafound in the environment and also in the feces of humans andother animals. Therefore, the presence of coliform bacteriaand, in particular, E. coli in drinking water may indicate thepresence of harmful, disease-causing organisms. For this rea-son, the enumeration of E. coli cells in water is increasinglyused to assess water quality.

The current reference method has several limiting factors, inparticular, time, as it may require up to 72 h to provide aconfirmed result. The aim of this publication is to present datafor an alternative method by which results can be obtained in18 h. In the context of a validation study, a collaborative studywas organized and data were collected according to the guide-lines of ISO 16140 (13). The interpretation of these data usingthe accuracy profile approach is presented, and the “fitness forpurpose” of this alternative method versus the referencemethod is ascertained.

MATERIALS AND METHODS

Methods for the enumeration of E. coli organisms. (i) Reference method. Thereference method for the enumeration of E. coli organisms is the publishedstandard ISO 9308-1 (15). This method consists of using Tergitol 7-triphenyltetrazolium chloride (TTC) agar after sample filtration. The standard operatingprocedure can be summarized as follows.

● Filter 100 ml of a sample with a sterile membrane as described in ISO 7218(14).

● Carefully place the membrane on Tergitol 7-TTC agar.● Incubate the samples at 36 � 2°C for 21 � 3 h.● If no typical colonies are present, incubate the samples at 36 � 2°C for an

additional 24 � 2 h.

When presumptive coliform colonies (lactose-positive colonies which show ayellow color development in the medium under the membrane) are present, aconfirmatory step is required. Selected colonies of presumptive E. coli andnon-E. coli coliform bacteria are subcultured onto a nonselective medium andincubated at 37 � 1°C for 24 � 2 h. Confirmation involves testing the coloniesfor oxidase activity and the production of indole. The colonies which are oxidasenegative and indole negative are presumed to be non-E. coli coliforms, whereascolonies which are oxidase negative and indole positive are presumed to be E.coli.

(ii) Alternative method. The alternative method (Colilert-18/Quanti-Tray) isbased upon the Defined Substrate Technology (DST) of IDEXX Laboratories,Inc. (5). Colilert-18/Quanti-Tray simultaneously detects and enumerates totalcoliform and E. coli bacteria in water. When total coliform bacteria metabolizethe nutrient indicator, o-nitrophenyl galactopyranoside (ONPG), the sampleturns yellow. When E. coli metabolizes the nutrient indicators ONPG and methyl-umbelliferyl-�-D-glucuronide (MUG), the sample turns yellow and fluorescesunder UV light. Only E. coli detection and enumeration are used in the presentstudy.

One of the outputs of this study is the quantification limit for the Colilert-18method. The Colilert-18 operating procedure can be summarized as follows.

● Add the contents of one blister pack to a 100-ml room temperature watersample in a sterile vessel.

● Cap the vessel, and shake it until the reagent is dissolved.● Pour the sample-reagent mixture into a Quanti-Tray and seal it in an

IDEXX Quanti-Tray sealer.● Incubate the sealed tray at 36 � 2°C for 18 to 22 h.● Read the results according to the result interpretation table, and count the

number of positive wells.

No confirmations are needed. The most probable number (MPN) can be calcu-lated from the number of positive wells (see the Appendix) or read in the tableprovided with the trays to convert the number of positive wells to MPN format.The values in this table agree with those calculated with the FDA’s Bacteriolog-ical Analytical Manual (BAM) method (19), used in the Appendix.

Experiment. (i) Experimental design. The experimental design used for theinterlaboratory study is described in ISO 16140 (see reference 13, section 6.3.3and Annex H). The aim of the design is to comparatively determine the perfor-mance characteristics (accuracy and precision) of an alternative method againstthe corresponding reference method. The design consists of at least eight par-ticipating laboratories producing usable results. The first step of the interlabo-ratory study is to select a single well-mixed representative water sample. Thewater sample selected for this study was artificially contaminated with E. colistrain ESC.1.131, an environmental strain isolated from water. Samples werecontaminated at three nominal levels (expressed as CFU/100 ml): control, 0(sterile); low, between 1 and 10; medium, between 10 and 50; and high, between50 and 200. The zero-contamination level was prepared for control purposes onlyand was not included in the calculations. Prior to inoculation, the absence of E.coli and coliform bacteria in the water sample was confirmed by the organizinglaboratory according to NF EN ISO 9308-1 (15).

Each batch of sample was divided into 100-ml aliquots that were transferred tosterile vials, which were subsequently closed and sealed with tape. Each vial wasindividually well mixed prior to shipment to a participant. Each participantreceived two samples per contamination level and was required to make dupli-cate analyses of each sample. A total of 16 enumerations were performed by eachlaboratory.

The stability of inoculated samples was determined over a 3-day period usinga prototype sample stored at 4 � 2°C. The results of the stability study arepresented in Table 1. Following log transformation, data were subjected to aone-way analysis of variance (ANOVA), and stability was found to be satisfactory

VOL. 77, 2011 VALIDATION OF METHODS TO CONTROL DRINKING WATER 3361

over a 48-h period. Samples were packaged at �4°C in thermostatic boxescontaining a temperature control probe (TomProbe, catalog no. MD30100; AESChemunex) prior to shipment by express mail to the participant. In order to beincluded in the study, results for each sample had to be reported to the InstitutScientifique d’Hygiene et d’Analyse (ISHA) within 48 h.

Eleven participating laboratories (I � 11) were selected to participate in thecollaborative study. Samples were labeled according to the following codingsystem: one letter, from A to K, for the identification of the laboratory and arandom number, from 1 to 8, anonymously assigned by the organizing laboratoryin order to identify the level.

One week before the trial, participants received detailed instructions on theoperating procedure and the necessary quantities of analytical reagents and kitsrequired for the study. All participants agreed to perform the analyses within24 h of receiving the samples. Receipt of samples was acknowledged within 24 h,and all samples were analyzed within 48 h according to the instructions providedby the organizing laboratory.

In addition, the organizing laboratory also provides a test protocol and a datasheet for recording experimental data and critical experimental conditions withineach laboratory.

During the interlaboratory study, only one type of matrix is required, as apreliminary study has previously been undertaken to fully define the types ofmatrices for which the method is applicable. The results of this preliminary studyare not presented here. Subsequently, each participating laboratory receivesthree subsamples at the three levels of contamination and performs duplicateanalyses with each method (alternative and reference). In all, each participantreturns 12 (3 � 2 � 2) analytical results, not including the results for the controlsample.

(ii) Statistical processing. Data are gathered by the organizing laboratory andprocessed in order to calculate validation criteria, such as repeatability, repro-ducibility, and bias between the two methods. Classical interpretation of dataconsists of testing hypotheses for each validation criterion. This strategy is oftenmisleading and can result in contradictory conclusions. For this reason, it wasdecided to apply the new strategy of the accuracy profile to aid data interpreta-tion.

The accuracy profile is a statistical and graphical decision-making tool aimedat helping the analyst to conclude whether an analytical procedure is valid. Itconsists of simultaneously combining, in a single graphic, � expectation toleranceintervals (�-ETIs) and acceptability limits (�) (see the Appendix for definitionsof terms). �-ETIs (or average tolerance intervals) are defined as intervals thatcover, on average, a certain percentage of a distribution. In practical terms,�-ETIs can be claimed to contain, on average, for example, 80% of futuremeasurements. �-ETIs should not be confused with confidence intervals, whichcharacterize only statistical parameters, such as an average, as �-ETIs relate toindividual future observations.

Analysts are often interested in estimating the average value in a population.Information about the population average, in the form of a sample estimate, canbe deduced by drawing an interval or range of values around the sample averagewhich is likely to include the true population average. Such ranges are generallyreferred to as confidence intervals. However, on occasion, the range of values ina population is of greater importance than the average. In such cases, anothertype of interval, a tolerance interval, may be useful. Average tolerance limitsdefine the bounds of an interval which contains, on average, a specified propor-tion (�) of the measurements.

In contrast, acceptability limits are defined as the allowable difference that canbe accepted between the reference and alternative methods without misinter-preting a result. For example, in many cases, a difference of 0.3 log or 0.4 log10

unit/100 ml between the result obtained by a reference method and that given by

an alternative method affects the interpretation with regard to the conformity ofa sample.

In so far as validation must cover the complete application domain of themethod, the accuracy profile combines both tolerance intervals and acceptabilitylimits calculated at several levels of contamination across the application domainof the method, thus meeting the criteria for validation.

The theory, calculation, and application of accuracy profiles to chemical anal-yses are described in detail elsewhere (9, 10, 11). When applied to microbiolog-ical analyses, some modifications are necessary. The construction of the accuracyprofile can be summarized as a sequence of nine steps, listed below (8). Withinthe calculation, i is the identification index of a participating laboratory, and 1 �

i � I, where I is the total number of laboratories participating in the trial. Afurther identifier (j) is the identification index of a replicate, and 1 � j � J, whereJ is the number of replicates, which is assumed to be the same for each labora-tory-level combination. Finally, k is the identification index of a level, and 1 �

k � K, where K is the number of contamination levels. According to the recom-mendations of ISO 16140 (13), I should be greater than 8, J should equal 2, andK should equal 3.

Construction of the accuracy profile. The construction of an accuracy profileinvolves the following steps. (i) Define the acceptability criterion (�), usually�0.2 or �0.3 decimal log unit/100 ml, for the alternative method. It is typical toselect a single value for � for all accuracy profiles, but it is possible to choosedifferent values depending on the level of contamination. (ii) Collect the analyt-ical results (in CFU/100 ml) obtained by the reference method within the inter-laboratory trial. For each level of contamination, calculate the median result[T(k)] obtained with the reference method and log transform the data. Thesevalues are called reference or target values. (iii) Collect the results (in CFU/100ml) obtained by the alternative method and log transform the data. These dataare denoted X. (iv) For each level, k, using xijk, calculate the reproducibilitystandard deviation (sR). The principle behind this calculation is that the totalvariance of all replicates of one level is modeled according to a random-effectANOVA, where the random effect corresponds to the laboratory. This methodconsists of splitting total variance into the within-laboratory variance (sr

2), alsocalled repeatability variance, and the between-laboratory variance (sL

2 ). Thisclassical statistical procedure is fully described in ISO 5725-2 (15b). Finally, thereproducibility standard deviation for one level of contamination can be calcu-lated using equation 1:

s�k�R � �s�k�L2 � s�k�r

2 (1)

(v) For each level, calculate the gross average [ �x�k�] of measurements made withthe alternative method. (vi) For each level, calculate the absolute bias accordingto equation 2:

�x�k� � T�k� (see step 2) (2)

This is an estimate of the trueness of the alternative method compared to thereference method. (vii) For each level, calculate the limits of the �-expectationtolerance interval (�-ETI) according to the method of Mee (17). �-ETI is theinterval within which the average proportion of future � results (�%) will fall.Calculations are detailed in the work of Rozet et al. (18). Finally, �-ETI isexpressed by equation 3, where s(k)R is the standard deviation of reproducibilityand k(k)M is its coverage factor for level k [see “Definitions of terms,” above, forthe k(k)M calculation].

x�k� � k�k�M � s�k�R (3)

(viii) For each level, calculate the differences between the limits of the toleranceinterval and the target value T(k) according to equation 4.

x�k� � k�k�M � s�k�R � T�k� (4)

(ix) Generate a graphical representation of calculated results as follows.

● On the horizontal (x) axis, plot the target values [T(k)] in decimal log units(logs).

● On the vertical (y) axis, simultaneously plot the bias (equation 2), theacceptability limits (��), and the tolerance interval limits (equation 4), allexpressed in log10 units.

In this context, the acceptability criterion (��) is expressed as an acceptabledifference, as we are dealing with logarithms, whereas in fact this difference canbe interpreted as a ratio. Acceptability criteria represent the maximum accept-able differences between a result obtained by an alternative method and thatgiven by the reference method.

TABLE 1. Stability study of enumerations of E. coli bacteria indrinking water over 3 days

Day

No. of E. coli organisms (CFU/100 ml) at indicatedlevel of contamination

Low Medium High

1 2 64 10011 67 110

2 12 65 1209 66 100

3 8 66 1003 63 120

3362 BOUBETRA ET AL. APPL. ENVIRON. MICROBIOL.

All calculations in this study were performed using Microsoft Excel, andspecific worksheets were prepared for this purpose. These worksheets maybe downloaded from http://www.paris.inra.fr/metarisk/downloads/software_programs/excel_templates). The interpretation of the accuracy profile is asfollows. If across the validation domain of the method, all �-expectation toler-ance intervals (�-ETIs) are included within the acceptability limits, the methodis declared valid over this range. Where any �-ETI value exceeds one of theacceptability limits, the method is not valid and the validity domain must bereduced.

This can be interpreted as follows. According to the definition, a �-ETI shouldcontain, on average, �% of the predicted future results. Therefore, the analystand end user can be confident that �% of future results will fall between thelimits of this interval. As long as this percentage is included in the acceptabilitylimits, the analyst can be confident that his measurements are comparable tothose obtained by the reference method, with an acceptance of �� log10 units.

RESULTS

Reference material preparation. Within the scope of thisvalidation study, the Colilert-18 method was submitted forvalidation according to the ISO 16140 procedure (13). Resultswere collected during a collaborative study organized by theInstitut Scientifique d’Hygiene et d’Analyse (ISHA, Massy,France) and supervised by the AFNOR Certification Board.

Raw data (expressed as numbers of CFU/100 ml) are pre-sented in Table 2.

Since microbial counts are not normally distributed, it wasdecided to transform the data into decimal logarithms, as istypical with such analyses. For each level of contamination,the reference value, T (or target value), was calculated as themedian result obtained with the reference method. For thelow, medium, and high contamination levels, target values inCFU/100 ml and their corresponding log10 values (in paren-theses) were 12 (1.079), 64 (1.806), and 120 (2.079), respec-tively. These values are somewhat different from the expectednominal values, demonstrating that the reference methodand/or reference sample preparation procedure also repre-sents a source of uncertainty. Counts obtained for the alterna-tive method were also transformed into log10 units.

TABLE 2. Raw count of E. coli bacteria collected during ourinterlaboratory studya

Level ofcontamination Laboratory

No. of CFU/100 ml by:

Reference method Alternative method

Duplicate 1 Duplicate 2 Duplicate 1 Duplicate 2

Low A 5 10 9 10B 19 10 18 14C 10 8 14 11D 14 12 10 14E 14 11 15 11F 9 6 8 3G 9 13 12 9H 4 11 15 7I 9 12 18 11J 13 9 9 8K 15 8 9 12

Medium A 39 29 53 36B 72 52 62 53C 65 64 95 78D 67 66 48 78E 52 56 34 38F 30 50 62 89G 55 51 56 59H 33 44 88 50I 53 54 66 62J 46 65 59 74K 45 52 59 48

High A 105 112 202 130B 139 142 145 118C 105 124 201 145D 146 124 200 200E 80 78 50 50F 70 90 130 130G 131 136 145 130H 89 90 130 200I 130 133 118 202J 130 112 202 130K 90 74 118 165

a The study consisted of 3 levels of artificially contaminated samples, 11 lab-oratories, and duplicate measurements obtained with the reference and alterna-tive methods. Level 0 was excluded from calculations.

FIG. 1. Linearity check for Escherichia coli data. Medians are indicated as vertical dashed lines and correspond to the target (T) values. In theequation, Alt and Ref are abbreviations for the alternative and reference method names.

VOL. 77, 2011 VALIDATION OF METHODS TO CONTROL DRINKING WATER 3363

Linearity. Linearity was achieved graphically as illustrated inFig. 1 by simultaneously plotting the logarithm results obtainedby each method on the same sample. It can be seen that thealternative method gives results which are proportional tothose of the reference method, and this complies with thedefinition of linearity. Additionally, the slope of the linearregression line between both methods is close to 1 and con-firms the linearity of the data. It is not useful to perform anyconformity hypothesis testing on this linear regression becausewithin the framework of the accuracy profile approach, nohypothesis testing is performed. This is intended, because withhypothesis testing, usually only the null hypothesis is verified,whereas a true estimate of the performance of the test requiresdefining an acceptance limit for each alternative hypothesis. Itis therefore preferable to globally use the acceptance limit(��), which allows a more informed decision to be as de-scribed below. Although this graphical interpretation mayseem rather subjective, it is considered sufficient.

In Fig. 1, the medians of the measurements obtained at eachlevel with the reference method are also represented. Thesevalues are used to define the target (T) values and to calculatethe trueness of the alternative method.

Accuracy profile. For this study, acceptability limits were setat two values: �0.3 and �0.4 log10 unit/100 ml. In terms of thenumber of CFU/100 ml, 0.3 corresponds to a factor of 2 in thecloseness of agreement of results generated by the alternativemethod compared to the reference method. For example,when an acceptability limit of �0.3 is applied, if the referencemethod gives a result of 10 CFU/100 ml (or 1 log unit), it isdeemed acceptable that the alternative method gives extremeresults at 5 or 20 CFU/100 ml. Likewise, when using an ac-ceptability limit of �0.4 log10 unit/100 ml, this maximum in-terval becomes 4 to 25 CFU/100 ml. It should be noted thatthese values are maximum acceptable limits, and it is expectedthat performance of the alternative method will be at least asgood as that of the reference method.

These levels of acceptance may appear to be rather lenientin some analytical domains, such as food chemistry, but corre-spond well to the actual decision rules that are applied toregulatory controls when traditional microbiological analysesbased on bacteria growing on agar media are used. Addition-ally, these thresholds take into account all possible sources ofuncertainty, for example, changes within the sample duringhandling and storing, dilution inaccuracies, sample heteroge-neity, matrix effects, the physiological state of the bacteria, theability of bacteria to grow and develop a colony, and manyother factors, such as laboratory effects, not included here.

The proportion (�) of future results falling within the �-ETIwas also set at two levels: 80% and 90%. For the alternativemethod at a given concentration, on average, �% of results forthe alternative method are comprised between the limits of the�-expectation tolerance interval. If this tolerance interval isincluded within the limits of acceptability, the method isclaimed to be valid.

Table 3 presents the validation criteria calculated at thethree levels of contamination used in the trial for a � of 80%and a � of �0.3. Figure 2 combines the accuracy profiles of E.coli for a � of �0.3 or �0.4 log10 unit/100 ml and a � equal to80% or 90%, and several interesting conclusions can be drawnfrom these data.

● For a � of 80% and a � of �0.4, the alternative methodcan be validated between 1.00 and 2.05 log10 units/100 ml(i.e., 10 and 112 CFU/100 ml).

● For a � equal to 80% and a � equal to �0.3, the toleranceinterval limits are contained within the acceptability limitsonly for low and medium contamination levels, and thealternative method cannot be validated over all studieddomains.

● The bias (the difference between the target value and theaverage result) is small but varies from 0.02 to 0.09 log10

unit/100 ml as the bacterial concentration increases. Thismay explain why the upper tolerance interval limit exceedsthe acceptability limit for higher contamination levels.

● When any �-ETI limit intersects any acceptability limit,the alternative method is determined not to be valid aboveor below the corresponding concentration. This point ismarked by a vertical arrow in Fig. 2 and corresponds to aconcentration of 1.96 log10 units/100 ml (about 92 CFU/100 ml). This value can be denoted the upper limit ofquantification (ULOQ) of the alternative method.

● With regard to the lower limit of quantification (LLOQ),the lowest level of the validation domain can be used, as itis not possible to extrapolate to values which were notactually included in the study in determining LOQs. How-ever, it can be assumed that alternative-method quantifi-cation performance is superior to this limit.

DISCUSSION

The accuracy profile method has been applied to the vali-dation of the Colilert-18/Quanti-Tray against the referencemethod ISO 9308-1 (15). Using a � of 80% and a � equal to0.4, the alternative method can be validated between 1.00 and2.05 log10 units/100 ml, equivalent to 10 to 112 CFU/100 ml.

The bias which is shown in Fig. 2 may be largely due to thedifference in the principles of enumeration between the refer-

TABLE 3. Validation criteria and statistical results fora � equal to 80%

Parameter Equation

No. of log10 CFU/100 ml witha � of 80% and a � of 0.3 at

indicated contamination levela

Low Medium High

Target value 1.000 1.716 2.049No. of participants (I) 11 11 11Avg of the level (alternative) 1.024 1.771 2.142Repeatability SD (sr) 0.141 0.093 0.099Between-lab SD (sL) 0.092 0.081 0.141

Reproducibility SD (sR) 1 0.168 0.123 0.172

Coverage factor (kM) 1.367 1.376 1.396TI SD (sIT) 0.173 0.127 0.178

Absolute lower TI limit 3 0.794 1.601 1.902Absolute upper TI limit 1.254 1.941 2.382

Bias 2 0.024 0.055 0.093

Lower TI limit (� � 80%) 4 �0.206 �0.115 �0.147Upper TI limit (� � 80%) 0.254 0.225 0.333

Lower acceptability limit (��) �0.3 �0.3 �0.3Upper acceptability limit (��) 0.3 0.3 0.3

a �, tolerance probability; �, acceptability limit in log10 units.

3364 BOUBETRA ET AL. APPL. ENVIRON. MICROBIOL.

ence method and the alternative method. The referencemethod is based upon the enumeration of bacterial colonies onan agar medium, while the alternative method is based uponthe most probable number (MPN) approach, the results ofwhich derive from a calculation.

It may be an interesting exercise to adjust the observed biasby applying a correction factor, although the values with thiscorrection are no longer useful for method validation pur-poses. As the slope of the regression line between the refer-ence and the alternative methods is 1.02, as illustrated in Fig.1, a correction factor of 2% can be proposed. This correctionfactor was applied to all log-transformed data, and validationcriteria were recalculated (Fig. 3). It can be seen that, withthese revised data, the alternative method can be deemed asvalid over all studied domains when a � of 80% and a � of �0.3are chosen. The ULOQ is then 2.05 log10 units/100 ml, but theLLOQ remains unchanged.

From the data presented, it can be concluded that the Colilert-

18 method, compared to the reference method with an accep-tance level of �0.3 log10 units/100 ml, is appropriate for theapplication domain ranging from 1.5 up to 2.1 log10 units/100ml.

Some questions raised by this study remain. (i) The lowestand highest target values used in the study were, respectively,approximately 10 and 110 coliform bacteria in 100 ml. Whendealing with distribution water, actual control samples maycontain lower contamination levels below 1 CFU/100 ml. As itis not possible to guarantee sample stability at such a low levelof contamination, this was not selected for the interlaboratorystudy. This may be used as a criticism of collaborative studieswithin the field of water microbiology. However, it must beremembered that collaborative studies based on calibratedsamples are commonly used for proficiency testing schemes inorder to control laboratory trueness. (ii) Data were collected inthe framework of an interlaboratory study, and TIs were cal-culated by including a laboratory effect; it may explain why they

FIG. 2. Accuracy profiles of alternative methods for a � equal to 80% and 90% and a � equal to �0.3 or �0.4 log10 unit. The arrow indicatesa possible upper limit of quantification (ULOQ). TI, tolerance interval.

FIG. 3. Accuracy profile after correction of log-transformed data with a 2% correction factor.

VOL. 77, 2011 VALIDATION OF METHODS TO CONTROL DRINKING WATER 3365

are wide. Thus, it can be expected that an individual laboratorymay achieve better performance if individual accuracy profilescan be drawn. It must be remembered that the ISO 16140procedure (13) is an attempt to validate an alternative method,regardless of the laboratory that will use it in the future. (iii)One key issue when building an accuracy profile is to correctlyselect the parameters � and �. The first parameter (�) dependson chosen criteria being related to the foreseen use of themethod, such as the significance of the microorganism as hy-giene criteria or as health risk criteria or a requirement of theend user; the second (�-ETI) is chosen to get an interval whichcontains a given percentage (usually 80% or 90%) of futuremeasurements, on average. These two parameters are inde-pendent, and they must be selected by different persons. Therole of the acceptability limit is to inform the end user of aperformance method in comparison to the reference methodduring the decision-making process. For instance, in the caseof hygiene control, this must be related to the potential patho-genicity of coliform bacteria in water. If distribution watermust contain less than 2 CFU/100 ml (0.3 log) and the accept-ability limit is 0.3 log10 unit/100 ml, this means that a watersample containing up to 4 CFU/100 ml is considered nonhaz-ardous. In contrast, if � is set at 80%, one-fifth of all qualitycontrol samples can be rejected at the LOQ. In the remainderof the validation domain, the percentage of acceptable resultsis much higher. For example, it can be calculated that, beforethe result is corrected, the probability of generating unaccept-able results at a level of 1.71 is 0.2% below the acceptabilitylimit of �0.3 and 0.6% above �0.3 log10 unit/100 ml.

In conclusion, the accuracy profile combined with an inter-laboratory study can be efficiently used as a decision supporttool for method validation. It is also a diagnostic tool in un-derstanding analytical problems linked to a method; for exam-ple, the presence of a bias could be related to the nature of themeasurement process and corrected via a simple correctionfactor.

APPENDIX

Definitions of terms. Definitions of terms used herein are as follows.(i) � is the chosen percentage of values that will be included in thecalculated interval. Notice that � has a meaning here different fromthat in the test of the hypothesis, where � usually means the probabilityof accepting a given alternative hypothesis. (ii) A � expectation toler-ance interval (�-ETI) is defined as an interval that covers an averagepercentage of a variable distribution. For instance, a �-ETI can beclaimed to contain 90% of future measurements, on average. A �-ETIcan be expressed as �x � kM � sr, where kM is the coverage factor,given by the equation

kM � Qt�v,1 � �

2 ��1 �1

I � J � G2

sr is the repeatability standard deviation, Qt is the percentile of aStudent t test distribution, � is the chosen probability (usually 80 or90%), I is the number of laboratories, J is the number of replicates, �is the number of degrees of freedom, and G is given by the equation

G � � H � 1J � H � 1

where H � sL2 /sr

2 � sR2 /sr

2 � 1, SR2 is the reproducibility variance, and

Sr2 is the repeatability variance. The number of degrees of freedom, �,

is given by the formula

� ��H � 1�2

�H �1J�

I � 1 �

1 �iJ

I � J

(iii) The acceptability criterion, �, is defined as the maximum accept-able difference between a result obtained by the alternative methodand the reference (true) value given by the reference method. Since weare dealing with logarithms, this difference can be comprehended as aratio. For example, the fact that the acceptability criterion (�) is equalto 0.3 log10 unit means that results (logarithms) by the alternativemethod, log(Nalt), that are inside the interval log(result by the refer-ence method) � 0.3 [or log(Nref) � 0.3] are acceptable. (iv) Theinterval log(Nref) � the acceptability criterion is named the accept-ability interval. Notice that since 10�0.3 is equal to 0.501 and 100.3 isequal to 1.995, an alternative expression for � � 0.3 could be 0.501 �Nalt/Nref � 1.995. The acceptability criterion (or the acceptabilityinterval) must be defined for each microbiological criterion selected.

Most probable number calculation. The MPN is the number thatmakes the observed outcome most probable and the solution for �(concentration) in the following equation:

�j � 1

Kgjmj

1� � exp��mj�� �

j � 1

K

tjmj

where exp(x) means ex, K is the number of dilutions, gj is the numberof positive tubes (tubes with growth) in the jth dilution, mj is theamount of the original sample put in each tube in the jth dilution, andtj is the number of tubes in the jth dilution (based on the FDA’smethod in the Bacteriological Analytical Manual [20]).

When the number of dilutions, K, equals 1, the MPN can be calcu-lated directly, without iteration, as

MPN �Um

ln� tt � g� �

Uvd

ln� tt � g�

where U denotes the amount of sample used as the unit (for example,100 for the MPN/100 ml), v denotes the volume of each well (ml), andd denotes the dilution (ml original sample/ml inoculated dilution).

REFERENCES

1. Anonymous. 2002. AFAQ AFNOR certification. Exigences relatives auxetudes preliminaire et collaborative menees par un laboratoire expert. In-ternal report, revision 7. AFAQ AFNOR, Saint-Denis, France.

2. Augustin, J. C., and V. Carlier. 2006. Lessons from the organization of aproficiency testing program in food microbiology by interlaboratory compar-ison: analytical methods in use, impact of methods on bacterial counts andmeasurement uncertainty of bacterial counts. Food Microbiol. 23:1 1–38.

3. Commission of the European Communities. 2005. Commission regulation(EC) no. 2073/2005 of 15 November 2005 on microbiological criteria forfoodstuffs. European Commission, Brussels, Belgium. http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri�OJ:L:2005:338:0001:0026:EN:PDF.

4. Corry, J. E. L., B. Jarvis, S. Passmore, and A. Hedges. 2007. A critical reviewof measurement uncertainty in the enumeration of food micro-organisms.Food Microbiol. 24:1 230-253.

5. Eckner, K. F. 1998. Comparison of membrane filtration and multiple-tubefermentation by the Colilert and Enterolert methods for detection of water-borne coliform bacteria, Escherichia coli, and enterococci used in drinkingand bathing water quality monitoring in southern Sweden. Appl. Environ.Microbiol. 64:8 3079-3083.

6. European Commission, Environment. 1998. Council directive 98/83/EC of 3November 1998 on the quality of water intended for human consumption.European Commission, Brussels, Belgium.

7. Feinberg, M. 2007. Validation of analytical methods based on accuracyprofiles. J. Chromatogr. A 1158:174–183.

8. Feinberg, M., D. Sohier, and J. F. David. 2009. Validation of an alternativemethod for counting Enterobacteriaceae in foods based on accuracy profile.J. AOAC Int. 92:2 527-537.20

9. Hubert, P. K., et al. 2004. Harmonization of strategies for the validation ofquantitative analytical procedures. A SFSTP proposal—part I. J. Pharm.Biomed. Anal. 36:579–586.

10. Hubert, P., et al. 2007. Harmonization of strategies for the validation ofquantitative analytical procedures. A SFSTP proposal—part II. J. Pharm.Biomed. Anal. 45:70–81.

11. Hubert, P., et al. 2007. Harmonization of strategies for the validation of

3366 BOUBETRA ET AL. APPL. ENVIRON. MICROBIOL.

quantitative analytical procedures. A SFSTP proposal—part III. J. Pharm.Biomed. Anal. 45:82–96.

12. In’t Veld, P. H. 1998. The use of reference materials in quality assuranceprogrammes in food microbiology laboratories. Int. J. Food Microbiol. 45:35–41.

13. ISO. 2003. ISO 16140. Microbiology of food and animal feeding stuffs—protocol for the validation of alternative methods. ISO, Geneva, Switzerland.

14. ISO. 2007. ISO 7218. Microbiology of food and animal feeding stuffs—general requirements and guidance for microbiological examinations. ISO,Geneva, Switzerland.

15. ISO. 2000. ISO 9308-1. Water quality—detection and enumeration of Esch-erichia coli and coliform bacteria. Part 1. Membrane filtration method. ISO,Geneva, Switzerland.

15a.ISO. 1986. ISO 5725. Precision of test methods—determination of repeat-ability, 2nd ed. ISO, Geneva, Switzerland.

15b.ISO. 2002. ISO 5725-2. Accuracy (trueness and precision) of measurementmethods and results. Part 2. Basic method for the determination of repeat-ability and reproducibility of a standard measurement method—corrigen-dum. ISO, Geneva, Switzerland.

16. Joint Committee for Guides in Metrology, International Bureau of Weightsand Measures. 2007. Guide ISO/CEI 99. Vocabulaire international demetrologie—concepts fondamentaux et generaux et termes associes (VIM).ISO, Geneva, Switzerland.

17. Mee, R. W. 1984. �-expectation and �-content tolerance limits for balancedone-way ANOVA random model. Technometrics 26:251–253.

18. Rozet, E., et al. 2007. Improvement of the decision efficiency of the accuracyprofile by means of a desirability function for analytical methods validation.Application to a diacetyl-monoxime colorimetric assay used for the deter-mination of urea in transdermal iontophoretic extracts. Anal. Chim. Acta591:239–247.

19. U.S. Food and Drug Administration. 2011. Bacteriological analytical manual(BAM). U.S. Food and Drug Administration, Washington, DC. http://www.fda.gov/Food/ScienceResearch/LaboratoryMethods/BacteriologicalAnalyticalManualBAM/default.htm.

20. U.S. Food and Drug Administration. 2006. Bacteriological analytical man-ual. Appendix 2. Most probable number from serial dilutions. FDA, Wash-ington, DC. http://www.fda.gov/Food/ScienceResearch/LaboratoryMethods/BacteriologicalAnalyticalManualBAM/ucm109656.htm.

VOL. 77, 2011 VALIDATION OF METHODS TO CONTROL DRINKING WATER 3367