PTM Overview and Consulting - Pàgines de la UABjornades.uab.cat/workshopmrama/sites/jornades.uab.cat.workshopmrama/... · Regulate products Buy or sell food, beverage products, or

Copyright © 2009 AOAC Research Institute

THE AOAC RESEARCH INSTITUTE PERFORMANCE TESTED METHODSSM

PROGRAM

Validation of Rapid Microbiological Methods


Outline of Workshop

About AOAC

Validation Overview

PTM Validation Components

Qualitative Study Design

Quantitative Study Design

Environmental Surface Testing

Report and Review

Certification and Publication

Renewal and Modification

About AOAC


AOAC Vision and Mission

That there be worldwide confidence in analytical results

To serve the communities of analytical science by providing fit-for-purpose methods and services for assuring quality measurements


AOAC

Solves problems by providing science based solutions

Globally recognized

Independent third party

Establishes standards

Voluntary consensus through stakeholder panels and working groups

Official and regulatory methods

Conducts conformity assessment

Official Methods of AnalysisSM

Performance Tested MethodsSM


AOAC

AOAC does not:

Regulate products

Buy or sell food, beverage products, or proprietary technologies

Promote specific food and beverage products

Do risk assessments

Set tolerance levels

Own a laboratory or provide laboratory services


AOAC INTERNATIONAL

Founded in 1884 in Philadelphia

Founded the Official Methods of AnalysisSM

Program

Originally Association of Official Agricultural Chemists

In 1965 became Association of Official Analytical Chemists

In 1991 became AOAC INTERNATIONAL

AOAC represents all of its method communities

Chemistry and microbiology methods

~ 3500 members worldwide

1/3 of members overseas (mostly Europe)

~2700 Official Methods


AOAC Research Institute

Incorporated in 1991

Administers the AOAC Performance Tested MethodsSM

program

Commercial and proprietary methods

Chemistry and microbiology methods

Harmonized with AOAC Official MethodsSM

program

>100 Current certificates

International membership

PTM Certification Mark has international recognition


AOAC RI Performance Tested MethodsSM

(PTM) Program

Experience validating a wide variety of method techniques

Cultural methods

Immunoassay

Lateral flow

Polymerase chain reaction (PCR), RT-PCR

Enzyme methods

Bioluminescence

Chemiluminescence

Electrical resistance

ID methods


AOAC RI Performance Tested MethodsSM

(PTM) Program

Experience validating methods for a wide variety of microorganisms

Listeria

monocytogenes

& Listeria

spp.

Salmonella

E. coli, E. coli O157:H7, &

STEC

Coliforms

Total plate counts

Campylobacter

Vibrio

Staphlyococcus

aureus

& Staph

enterotoxin

Yeasts and molds

Bacillus anthracis


Up to Date List of PTM Approved Methods

Validation Overview


Programs for Rapid Methods

Performance Tested MethodsSM

Administered by AOAC Research Institute

Official Methods of AnalysisSM

Administered by AOAC INTERNATIONAL


Method Validation Guidelines

AOAC Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis

J. Assoc. Off. Anal. Chem.

85, 1187-1203 (2002)

Available at www.aoac.org

http://www.aoac.org/


PTM Program Technical Requirements

Method Developer Responsibilities:

Inclusivity

Exclusivity

Matrix Study

Robustness

Stability

Lot-to-Lot Variation

Independent Laboratory Responsibilities:

Matrix Study

1 Laboratory

1-4 Foods or surfaces (claim dependent)


OMA Program Technical Requirements

Pre-Collaborative Study

Inclusivity

Exclusivity

Matrix Study

Collaborative Study

Matrix Study

At least 10 labs for qualitative methods

At least 8 labs for quantitative methods

1-6 Foods

AOAC, USDA, FDA, ISO, Health Canada reference methods

Reproducibility


PTM Program Validation Procedure

Consulting

Required

Waiver can be granted upon approval of managing director

Application

New methods and modifications

Data Collection

Method developer and independent studies

Review

Performance Tested Methods status granted or declined



Consulting

Method Developer Submits:

Consulting application and fee

Claim

Target analyte(s)

Matrices

Method instructions (package insert, user guide, SOP)


Consulting

AOAC RI Provides:

Overview of current technical requirements

AOAC Guidelines

Appropriate reference methods

Specific validation protocols

Reviewed and approved by General Referee

Validation flowcharts


Consulting

Benefits

Understanding for both sides

AOAC has better understanding of the method

Method developer has better understanding of AOAC requirements

Can significantly reduce method developer time and cost

Most current technical updates and requirements


PTM Application

Method Developer Submits:

Application

Package insert, user manual, directions for use

Product labels

QA/QC synopsis

Application fee


PTM Study

Method Developer Study

Inclusivity and exclusivity

Matrix Study

Lot-to-lot

Stability

Robustness


PTM Study

Independent Laboratory Study

Contracted by AOAC RI

Method developer provides method materials and training

Performs matrix study

Report


Review

Method developer submits complete study report and package insert (PI)

Reviewed by general referee and 2 expert reviewers

Method developer addresses reviewers’ comments/questions as necessary

Report and PI revisions submitted as necessary

Decision to grant or decline approval, based on reviewer consensus


Certification

License Agreement and Use of the Mark

List on PTM Approved Methods website

Annual Review & Certification Renewal


Publication

Certificate issued

Certification Mark issued

Certification Report published in Inside Laboratory Management

Study report published in Journal of AOAC INTERNATIONAL


Harmonization with Official Methods of AnalysisSM

Method developer can use PTM study to satisfy

the OMA pre-collaborative requirement

Both

PTM and OMA requirements must be met

Allows for company to perform one large study

rather than two separate ones


Harmonization Collaborative Study

Collaborative Study

Qualitative

10 labs minimum with valid data

1-

6 matrices, depending on claim

3 levels (uninoculated, low, and high)/matrix

6 test portions/level/matrix/method

Quantitative

8 labs minimum with valid data

1-

6 foods, depending on claim

4 levels (uninoculated, low, medium, and high)/matrix

2 test portions/level/matrix/method


Harmonization Flowchart

Application for

harmonized study

RI prepares PTM & OMA protocols

Sponsor collects

internal data, IL performs

study

Sponsor writes report

Report reviewed by GR, Stat, 2

ER & 1 Comm. H

Approved study =

PTM certification

Collaborative study (OMA)

& review

Published in AOAC OMA and JAOACI

PTM manuscript published in

JAOAC

Certification report

published in ILM

Approved study = OMA First Action

OMA Final Action


Harmonization with MicroVal

MicroVal

is a European organization for the validation and certification of alternative methods for the microbiological analysis of food and beverages.

http://www.microval.org/

EN-ISO 16140 –

“Protocol for the validation of the alternative methods”

forms the basis for the European

certification of alternative methods, thus fulfilling European legislation.

http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=30158

http://www.microval.org/

http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=30158



Method developer can use MicroVal

Expert Laboratory study to satisfy the PTM inclusivity, exclusivity and matrix study requirements

Both

PTM and MicroVal

requirements must be met

Since all matrix study work is done by the Expert Lab, an additional independent lab study is not required

Additional PTM studies (lot-to-lot, stability, robustness) are performed by the method developer



Allows for company to perform one large

study rather than two separate ones

Can use common expert reviewers, but still

include GR for AOAC

Each validating organization retains its own

acceptance criteria


Validation Complete

Questions?

For additional information, go to http://www.aoac.org/

http://www.aoac.org/




Inclusivity

Exclusivity

RobustnessStabilityMatrix Study

Lot-to-lot


Responsibilities

Method Developer

Inclusivity

Exclusivity

Robustness

Lot-to-lot/stability

Matrix Study

AOAC-RI

Coordinates independent laboratory testing

Matrix Study


Inclusivity

To ensure the test kit can detect a variety of target organisms

Serovar, species, genus or class

At least 50% should be food or environmental isolates

Source: ATCC, NTCC, etc.

Origin: chicken, cheese, etc.

Cultures grown in test kit media

Test at approximately ten fold

higher than the LOD

Inclusivity


Exclusivity

To determine the test kit’s ability to discriminate target organisms from non-target organisms

Report source and origin of all isolates

Cultures grown in non-selective media that supports growth

Does not need to be the same for all isolates

If an isolate generates a positive result

Re-test isolate in test method

selective enrichment

Report both results

Exclusivity


Robustness

Introduce minor variations to the method and observe effects

Critical steps in the method

Steps that an end user could likely vary

Incubation temperature

Incubation time

Reagent volume

Sample volume

Robustness


Lot-to-lot and Stability

To ensure the manufacturing is consistent between lots and to support the shelf life of the test kit

Can combine lot-to-lot and stability

Test 3 lots

Newly manufactured

Middle of term

Near expiration date

Stability

Lot-to-lot


Matrix Study

Candidate method

Reference method

Claim

Allocation of matrix testing

Choosing Foods

Sample preparation

Analysis

Paired or unpaired samples

Matrix Study


Matrix Study

Candidate Method

Method of analysis submitted for validation. The method can be proprietary or noncommercial.

Reference Method

The appropriate official method that is applicable to the analyte

and sample type that the

candidate method is intended to detect. Candidate methods are compared to reference methods when available.


Matrix Study

AOAC Official Methods of Analysis

USDA FSIS Microbiological Laboratory Guidebook

US FDA Bacteriological Analytical Manual

ISO Methods

Health Canada

Other (Compendium, SMEDP, etc.)


Claim Number of Matrices

Number of Categories

Variety At least 10 At least 5

Selected At least 5 At least 2

Food Category At least 5 1

Matrix Study Claim


Matrix Study Allocation of Matrix Testing

Option 1

Method developers required to validate all claimed matrices

Independent lab evaluates 20% of total claimed matrices

Same strain/serovar

for each matrix as was tested in MD lab (if artificially contaminated)

Option 2

AOAC RI coordinates all

matrix study testing

Performed in an independent lab


Matrix Study Allocation of Matrix Testing

Number of Claimed Matrices

Number of Matrices Evaluated by Independent

Lab

1-5 1

6-10 2

11-15 3

16-20 4


Matrix Study Choosing Foods

Likely to contain target analyte

Associated with outbreaks

Where analyte

can survive

pH, water activity, spices, etc.

Target market for test method

Trial run of all foods before beginning validation study


Matrix Study Choosing Foods

Use naturally contaminated samples whenever possible

Salmonella

in poultry products

Campylobacter

in carcass rinses

Artificially contaminated samples can be prepared if necessary

A different isolate should be used for each matrix tested

Details in Annex A of the Micro Guidelines


Matrix Study Competitive Flora

At least one food type must be tested with competing flora (natural or artificial)

Naturally present in raw matrices

Always done in environmental surface studies

Helps simulate conditions found in nature

If artificially contaminating, competing organism should be at least ten times higher than target


Matrix Study Cultures for artificial contamination

Matrix study should use foodborne

isolates

Single strain per matrix, not a pool of strains

Internationally recognized source

ATCC or equivalent

Laboratory isolate

Confirmation documentation required

Biochemical

Serological


Matrix Study Cultures for artificial contamination

Lyophilized cultures

Unless known, recommended to do aging study of organism in matrix

Preparation

Historical counts

Turbidity

Known counts


Matrix Study Food Preparation

Purchase

Sufficient amount to cover a paired or unpaired study, plus MPN analysis and aerobic plate counts

Screening

Recommended to use both the test method and reference method

If negative for target, prepare food for inoculation

Chop, cut, grind, melt, etc.

Use same lot of material for inoculated samples and uninoculated

control samples


Matrix Study Artificial Contamination

Liquid

Add diluted organism to large

quantity and mix, stir, etc.

Dry

Add lyophilized organism to

food and mix: roll, shake, etc.



Solid/moist

Add diluted organism drop-wise over bulk quantity of food, homogenize as appropriate

Spray



Bulk inoculation required

Make sure to maintain integrity of matrix when inoculating with prepared culture

Be careful not to dilute matrix with the inoculum


Matrix Study Food Stabilization

Perishable foods

Store 2-4 days at 2-6°C

Quantitative Listeria

store for 1 day at 2-6°C

Room temperature, shelf stable foods

Store minimum of 2 weeks at room temp

Test periodically to ensure level is stable

Frozen foods

Store minimum of 2 weeks at -20 to -30°C


Matrix Study Analysis

Perform test method according to method instructions

Perform reference method according to most current published method

Perform MPNs

according to reference method

Determine aerobic plate count (APC)

All test portions, MPNs, and APC are to be initiated the same day


Matrix Study Paired –

common enrichment

25g test portionenrichment

Candidate MethodPresumptive Result

Reference MethodIsolate and confirm

CONFIRMED RESULT


Matrix Study Unpaired –

separate test portions

Candidate Method25g test portion enrichment

Transfer step,If necessary

Run AssayPresumptive Result

Isolate and confirm

Candidate MethodCONFIRMED RESULT

Reference Method25g test portion enrichment

Isolate and confirm

Reference MethodCONFIRMED RESULT


Matrix Study Confirmations

All samples, regardless of presumptive result, must be subjected to isolation procedures for confirmation of target colonies

Confirm according to the reference method protocol, or as directed by the test method instructions if included

Confirm the number of colonies as outlined in reference method

Qualitative Method Study Design


Qualitative Method

“Method of analysis whose response is either presence or absence of the analyte

detected

either directly or indirectly in a certain amount of sample.”

–

AOAC Guidelines


Validation Study Components

Inclusivity

Exclusivity

Stability RobustnessMatrix Study

Lot-to-lot


Inclusivity Qualitative

To ensure the test method can detect a variety of target organisms


At least 50% of food origin

Minimum of 50 isolates

100 for Salmonella


Inclusivity


Exclusivity Qualitative

To determine the test method’s ability to discriminate target organisms from non-target organisms


Report source and origin

of all isolatesExclusivity


Robustness Qualitative

Introduce minor variations to the method and observe effects




Incubation time

Reagent volume

Sample volume

Robustness


Robustness Qualitative

Qualitative

Use 2 target isolates and 1 non-target

5 replicates of each for a total of 15 results

Culture organisms

Target -

in test method enrichment media

Non-target -

in non-selective enrichment media

Analyze with test method


Lot-to-lot and Stability Qualitative

To ensure the manufacturing is consistent between lots and to support the shelf life of the test method components


Test 3 lots

Newly manufactured

Middle of term


2 target isolates and 1 non-target


Stability

Lot-to-lot


Matrix Study Qualitative

Target Contamination Levels

Natural or artificial contaminationto achieve appropriate levels

Important to test within the range of the limit of detection of the test method

Goal is to achieve fractional results

for at least one level for at least one method (test or reference or both)

5-15 positive out of 20 replicates

Matrix Study



Target contamination level

Low: 0.2-2 cfu/25g

High: 2-5 cfu/25g

Uninoculated: 0 cfu/25g

Adjust levels as necessary to achieve fractional results

Data from one level

is acceptable, as long as results are fractional

Often do 2 levels just in case

20 replicate test portions each for high and low levels, 5 for uninoculated

Per level per method if unpaired samples



If natural contamination is found:

2 different lots, 20 replicate test portions per lot

20 test portions per lot per method if unpaired

Fractional recovery required for at least one lot

Food may be temperature abused or diluted with uncontaminated product to achieve appropriate levels



Use a large scale Most Probable Number (MPN) to confirm contamination levels

3 x 100g, 3 x 10g, 3 x 1g, 3 x 0.1g

MPN = statistical estimate of contamination level

Determine using reference method

Use same lot as test portions

Initiate on same day as test portions


MPN

10

0 g

10

0 g

10

0 g

10

g

10

g

10

g

1 g

1 g

1 g

0.1

g

0.1

g

0.1

g

100g sample + 900ml media



0.1g sample + 9.9ml media


MPN


Matrix Study Confirmation of qualitative levels -

MPN

Record number of positive flasks/tubes

Calculate MPN Index using 3 values

Level 100g 10g 1g 0.1g

High 3/3 3/3 0/3 0/3Low 2/3 1/3 1/3 0/3

Level 100g 10g 1g 0.1gHigh 3/3 3/3 0/3 0/3Low 2/3 1/3 1/3 0/3


MPN Calculations Number of positive tubes

100g 10g 1g MPN/25g

3 2 2 5.253 2 3 7.253 3 0 6.03 3 1 11.53 3 2 27.5

Using MPN table, determine MPN Index


Statistical Analysis Qualitative: Paired and Unpaired

Test for significant difference -

Chi-square (χ2)

χ2 results greater than

3.84 indicate a significant

difference between two methods at a 95%

confidence level

Sensitivity

Specificity


Qualitative Statistics -

Paired Contingency Table

McNemar’s

χ2

= (|b

– c| – 1)2/ b

+ c

Sensitivity rate = a

/ a + b

Specificity rate = d

/ c

+ d

Test Method Result

Positive Negative Total

Reference Method Result

Positive a b a + b

Negative c d c + d

Total a + c b + d


Qualitative Statistics-Paired Raw Data

SampleTest

MethodPresumptive

TestMethod

Confirmed

ReferenceMethodResult

1 - - -

2 + - -

3 + + +

4 + + +

5 - - -

6 + + +

7 + + +

8 - - -

9 - - -

10 - - -

11 - + +

12 + + +

13 + + +

14 + + +

15 + - -

16 + + +

17 - - -

18 + + +

19 + + +

20 + + +

Total pos 13 12 12

Uninoc.

1 - - -

2 - - -

3 - - -

4 - - -

5 - - -


Qualitative Statistics-Paired Contingency Table

McNemar’s

χ2

= (|1 – 0| – 1)2/ 1 = 0

Sensitivity = 11 / 12 = 0.92 or 92%

Specificity = 8 / 8 = 1 or 100%

Test Method Result

Positive Negative Total

Reference Method Result

Positive 11 1 12

Negative 0 8 8

Total 11 9


Qualitative Statistics-Unpaired Contingency Table

Conf. pos Conf. neg

Alternative Method

Pres. pos A B

Pres. neg C D

Reference Method E F

Mantel-Haenszel

χ2

= (n-1)(AF – (B+C+D)E)2

(A+B+C+D)(A+E)(B+C+D+F)(E+F)

where n = A

+

B

+ C

+ D

+ E

+ F

Relative sensitivity = A/E


Qualitative Statistics-Unpaired Performance Parameters

Relative sensitivity = A/E

Proportion of presumptive positive results that

confirmed positive for the alternative method

relative to the proportion positive results for the

reference method


Qualitative Statistics-Unpaired Raw Data

SampleTest

MethodPresumptive

TestMethod

Confirmed

ReferenceMethodResult

1 - - +

2 + - -

3 + + +

4 + + -

5 - - +

6 + + +

7 + + -

8 - - +

9 - - +

10 - - -

11 - + +

12 + + -

13 + + -

14 + + +

15 + - +

16 + + +

17 - - -

18 + + +

19 + + +

20 + + -

Total pos 13 12 12

Uninoc.

1 - - -

2 - - -

3 - - -

4 - - -

5 - - -


Qualitative Statistics-Unpaired Contingency Table

Conf. pos Conf. neg

Alternative Method

Pres. pos 11 2

Pres. neg 1 6

Reference Method 12 8

Mantel-Haenszel

χ2

= (40 -

1)(88 -

108)2 = 0.1

(20)(23)(17)(20)

Relative sensitivity = 11/12 = 0.92 or 92%


Data Presentation

Matrix

Inoculating strain

Level of contamination

MPN (cfu/25g)

Number of test portions

Reference method results

Alternative method results

Presumptive

Confirmed

Chi square

Sensitivity rate

Specificity rate

Quantitative Method Study Design


Quantitative Method

“Method of analysis whose response is the amount of the analyte

measured either directly

(e.g. enumeration in a mass or volume) or indirectly (e.g. color absorbance, impedance, etc.) in a certain amount of sample.”

–

AOAC Guidelines


Validation Study Components

Inclusivity

Exclusivity

Stability RobustnessMatrix Study

Lot-to-lot


Inclusivity Quantitative

To ensure the test method can detect a variety of target analytes




Test cultures in non-selective broth

No inclusivity for total count methods

Inclusivity


Exclusivity Quantitative

To determine the test method’s ability to discriminate target organisms from non-target organisms


Report source and originof all isolates

No exclusivity for total count methods

Exclusivity


Robustness

Introduce minor variations to the test method and observe effects




Incubation time

Reagent volume

Sample volume

Robustness


Robustness Quantitative

Use 1 target isolate and 1 non-target

1 target isolate at high and low level


Culture organisms in non-selective broth

Analyze using test method


Lot-to-lot and Stability Quantitative

To ensure the manufacturing is consistent between lots and to support the shelf life of the test kit


Test 3 lots

Newly manufactured

Middle of term


Use 1 target isolate and 1 non-target

5 replicates of target at a high level and 5 replicates at a low level

Stability

Lot-to-lot


Matrix Study Quantitative

Target inoculum

levels

Natural or artificial contamination at appropriate levels

If inoculating, important to have 10 fold between levels

Uninoculated: 0 cfu/g

Low: 10-100 cfu/g

Medium: 100-1000 cfu/g

High: 1000-10,000 cfu/g

5 replicate test portions per level

5 per level per method if unpaired samples

Matrix Study


Matrix Study Quantitative

If natural contamination is found:

Quantitative method

3 different lots –

5 replicate test portions per lot

3 different contamination levels

Food may be temperature abused or diluted with uncontaminated product to achieve appropriate levels


Statistical Analysis Quantitative

Perform log10

transformation of microbial counts

Perform outlier tests

Cochran and Grubbs or equivalent

Plot data

Test method on y-axis and reference method on x-axis

Calculate repeatability, sr

Calculate relative standard deviation for repeatability, RSDr

Perform significance tests

ANOVA, paired t-test or independent t-test


SampleCFU/g

RMLog10

RMCFU/g

TMLog10

TM1 1500 3.18 1000 3.00

2 750 2.88 830 2.92

3 750 2.88 900 2.95

4 2400 3.38 920 2.96

5 1500 3.18 1000 3.00

Quantitative Statistics Medium Inoculation Level –

Raw Data


Quantitative Statistics Method Agreement

Plot data

Test method on y-axis and reference method on x-axis

Use log10

values

Plot all levels on one graph

Provide regression line, line formula and correlation coefficient

Do not plot data if there is a missing value, i.e. >1100 or TNTC or <10

Missing data and corresponding value are removed from data analysis


Quantitative Data 3 levels, ten fold apart


Quantitative Statistics Repeatability (Standard deviation)

SampleLog10RM

Log10TM

1 3.18 3.00

2 2.88 2.92

3 2.88 2.95

4 3.38 2.96

5 3.18 3.00

Mean 3.10 2.97

Calculate sr

and RSDr

(RSDr

= Sr

/mean) for each method using the log10

values

Ref Methodsr

= 0.22RSDr

= 0.71

Test Methodsr

= 0.03RSDr

= 0.010


Quantitative Statistics

t-Test or Analysis of Variance (ANOVA)

To determine if a significant difference between the alternative method mean and the reference method mean can be detected

Performed for each contamination level per matrix

Typically a 95% confidence interval

If p > 0.05, no significant difference detected


Quantitative Statistics

Resulting data from t-test in Excel


Data Presentation

Matrix

Inoculating organism

Contamination level

Organisms/g

Log10

organisms/g

Mean

Repeatability, sr

Relative standard deviation, RSDr

P-values from t-test or ANOVA comparison of means

Environmental Surface Testing Study Design



Test kits used in the analysis of environmental surfaces

Typically qualitative methods

Same parameters used for food testing are required for surface testing:

Inclusivity

Exclusivity

Robustness

Lot to lot/stability

Matrix study

Inclusivity

Exclusivity

RobustnessStabilityMatrix Study

Lot-to-lot



Reference methods

Listeria

–

USDA FSIS MLG

Salmonella

–

FDA BAM

Guideline

“Study Protocol for Validation of Environmental Surfaces Claims for Target Organisms.”

Draft

document


Matrices

Stainless steel

Rubber

Wood

Food grade painted surfaces

Plastic

polyethylene, polypropylene, polycarbonate

Ceramic, glazed earthen material or glass

Sealed concrete

Cast iron (coated to prevent rusting)

Air filter material


Claim

Environmental Surfaces – All 9 surfaces required

5 using sponge – 4” x 4” surface area

4 using swab – 1” x 1” surface area

Selected Surfaces – 2-8 Surfaces

Each surface listed

Individual Surface type


Can add stabilizer to inoculum

Dilute in non-nutritive buffer

Inoculate surface

At least one surface must be contaminated with a competitor at a level 10 times greater than the target

Dry overnight

Recovery levels can vary

Strain/surface combinations

Laboratory conditions (humidity, temperature)

Preliminary testing is critical

Surface Inoculation


Sample surface

Swab or sponge

Sponge/swab entire sampling area in both a horizontal and vertical motion

Swab/sponge held in neutralizing broth (DE broth) at room temperature for 2 hrs prior to testing

Add enrichment media to collection device

Surface Sampling


Performing Study

If unpaired samples, side by side surface areas are tested, one for each method

Perform remainder of test method and reference method

Estimate initial level by determining plate count of inoculum

(MPNs

not applicable)

At least one level by at least one of the methods must achieve fractional recovery 5-15 positive test portions out of 20


Performing Study

All samples, regardless of test method results must continue through confirmation process

Chi-square statistics and performance parameters are determined as per food studies based on whether test portions are paired or unpaired

Present data as per food studies omitting MPN results

Report and Review


Report

Data collection is complete

Independent lab has submitted report

Internal lab testing is complete

Analyze data

See qualitative and quantitative sections for specific

statistical information

Compile into single report


Report Format

Template is provided with the validation

outline

Format is in place to facilitate subsequent

publication in JAOAC

Indicate if method is PTM only or if harmonized

with OMA


Report Format

Method / Kit Title:AOAC Performance Tested MethodSM

XXXXXX

AbstractMethod AuthorsSubmitting CompanyIndependent LaboratoryReviewers

1 Scope of method1.1Target analyte1.2 Matrices 1.3 Summary of validated performance claims

2 Definitions2.1 Limit of Detection2.2 Accuracy2.3 Precision

3 PrincipleGeneral description of the scientific principle of the test method.

4 General InformationGeneral information about the target analyte, matrix, occurrence of the analyte

in nature and its effect.


5 Test Kit Information5.1 Kit Name5.2 Catalog Number 5.3 Ordering Information

5.3.1 USA.—

address with telephone, fax , e-mail, and website addresses.5.3.2 Europe.—

other offices if applicable.5.4 Test Kit Reagents

5.4.1 Reagent one.—

brief description.5.4.2 Reagent two 5.4.3 Etc. as needed.

6 Additional supplies and reagentsCite supplies and reagents as needed.—

brief description, specifications, and source.7 Apparatus

7.1 Heating block.—

brief description, specifications, source.7.2 Temperature-controlled waterbath.—

brief description, specifications, source.8 Standard Reference MaterialsExample: ECRC 96-1555 (available from the E.coli

Reference Center, Pennsylvania State University, University Park, PA., USA)9 Standard SolutionsExample: Tris-borate EDTA electrophoresis buffer (TBE).—

Dissolve 108 g Tris, 9.3 g NA2

EDTA, 55 g boric acid in ca 800 mL

H20. Adjust pH to 8.2 with concentrated HCl

and bring final volume to 1 L with H20.

Report Format


Report Format

10 Safety Precautions

Example: Electrophoresis gels contain ethidium

bromide which is a potential mutagen. Observe all of the manufacturer’s precautions for handling and disposal of these gels.

11 General Preparation 11.1 Assemble the gel electrophoresis apparatus according to the manufacturer’s instructions. 11.2 Turn on the heating blocks and set temperatures to 37o and 95oC.

12 Sample Preparation

Describe how samples are processed.13 Analysis

13.1Title of first major step in analysis such as ExtractionGeneral description of process if needed.

13.1.1 Step-by-step description of process 13.1.2 Etc as needed.

13.2 Title of second major step in analysis such as Sample IncubationGeneral description of process if needed.

13.2.1 Step-by-step description of process 13.2.2 Etc as needed.

14 Interpretation and Test Result Report

Description of interpretation procedure and how results are reported.


Report Format

15 Method Developer Validation Studies

Validation studies in the conducted in the Method Developer laboratory15.1 Title of first study such as “Robustness Testing”General description of the study such as: The effects of perturbations in four method parameters were investigated: 1)

temperature [69.5°C / 70.5°C]; 2) reduced sample volume (45 µL / 50 µL); 3) time delay (0 min / 15 min) in cooling rack; and 4) sample volume (3 µL / 5 µL).

15.1.1 Methodology15.1.2 Results

15.2 Title of the second study such as “Matrix Study”15.2.1 Methodology15.2.2 Results

16 Independent Validation Study

Validation study conducted by an independent laboratory under the direction of the AOAC Research Institute. 16.1 Title of study such as “Matrix Study”

16.1.1 Methodology

16.1.2 Results

17 Discussion

General discussion of the all of the studies with particular emphasis on unexpected results, new knowledge about the analyte, matrix, or analytical technique.

18 Conclusion

19 References


Report Format

Abstract

Method Authors

Submitting Company

Independent Lab

Reviewers

Introduction

Materials and methods

Summary of results

Discussion

Conclusion

References

Acknowledgements


Review

Method Developer submits

Study report

Package insert (directions for use)

QA/QC description

Package labels


Review

Report, Package Insert, and review form sent to:

PTM: GR and 2 expert reviewers

Harmonized PTM/OMA: GR, 2 expert reviewers, 1 Committee H member, and statistical advisor

2 weeks allowed for initial review

Reviewer comments are sent to author

Technical consultant available to answer questions and assist with responses


Review

Report and/or package insert revised as necessary and resubmitted to reviewers

One week allowed for review

Decision or additional comments sent to author

Same time schedule as above for any additional reviews, if needed

Review complete once consensus (for or against) is achieved


Potential Problems

Fractional recovery requirement not met

Methods not followed exactly

Did not use correct or current reference method

Did not follow current validation policies

Data not acceptable

Test method did not perform as well or better than the reference method


Potential Resolutions

Can retest if:

Fractional recovery is not achieved

Method(s) not followed exactly

Out of date or incorrect methods or policies were followed

Matrices can be removed from claim if test method is unsuccessful

To be determined by expert reviewers



Approval Criteria

Applicant’s performance data supports and confirms all claims made in the method’s descriptive insert

Independent laboratory’s performance data corroborates the applicant’s performance data within the statistical limits specified in the testing protocol

Test method must perform as well or better than the reference method


Approved

Upon approval, the method author will receive the following

Approval letter

Certification mark

Certificate

Web entry

License agreement

Request for ILM article/certification report


Use of Certification Mark

License agreement is executed covering the Institute- owned PTM certification mark

Authorizes the method developer to use the mark on the approved test method, packaging, insert, informational and promotional materials

Denotes that the method was evaluated by the AOAC RI and confirms claims

Mark must be discontinued if certificate is not renewed


Certificate Information

Certificate of Performance TestedSM

Status

Certificate No. (number)

The AOAC Research Institute hereby certifies that the performance of the test method designated as:

Full Test Method Namemanufactured byCompany Name

Company AddressCountry of Company

has been reviewed under the AOAC Research Institute's Performance Tested MethodsSM Program, and found to perform as stated by the manufacturer. This certificate authorizes the manufacturer to display the AOAC Performance TestedSM certification mark along with the statement - "THIS TEST KIT'S PERFORMANCE WAS REVIEWED BY AOAC RESEARCH INSTITUTE AND WAS FOUND TO PERFORM TO THE MANUFACTURER'S SPECIFICATIONS" - on the above mentioned test kit until December 31, (Current Calendar Year).

Signed for AOAC Research Institute: Date:________________

Scott G. Coates Managing Director


Sample Web Entry


Publications

Inside Laboratory Management

Summary of study

Serves as certification report

Journal of AOAC INTERNATIONAL

Complete study

Web site

http://www.aoac.org/testkits/testedmethods.html

http://www.aoac.org/testkits/testedmethods.html

http://images.google.com/imgres?imgurl=http://www.dmsc.moph.go.th/net/lib/New/jour16/no9/%25BB%25A1AOAC_vol85no4july-august02.gif&imgrefurl=http://www.dmsc.moph.go.th/net/lib/v169.htm&h=215&w=158&sz=29&tbnid=eAJ5VMEr25UJ:&tbnh=101&tbnw=74&start=43&prev=/images%3Fq%3DAOAC%26start%3D40%26hl%3Den%26lr%3D%26safe%3Doff%26sa%3DN



Renewal

Applicant certifies that no changes have been made to the test method since the original PTM approval and the method performs as originally evaluated

Submit Method Modification Review Form if changes have been made

Granted for 1 year


Renewal

Changes in test method

Notification -

responsibility of the licensee to notify

AOAC-RI of any test method changes

Failure to appropriately notify the AOAC RI of

changes may result in the cancellation of PTM

certificate


Modification

Changes made in test method which effect:

Test method instructions

Labeling changes

Deletion or restatement of validated claims or procedures

Increase/decrease stability claims

Test method performance

Changes to method components

Changes to enrichments/procedure

Changes to equipment

Changes to claim

Changes to manufacturing process


Modification

Review Levels

Level 1

Internal AOAC RI review

Change does not alter the validated performance of the test method

Level 2

Licensee data required

Reviewed by GR

Level 3

Licensee and independent data required

Reviewed by GR and 2 Expert Reviewers


Modification

Testing requirements

Level 1

Document review, no additional testing required

Level 2

Matrix study required, at least 3 matrices

Modified method compared to reference method

Data collected by licensee


Modification

Testing requirements

Level 3

Licensee and independent lab data required

Same as for new PTM study

Matrix extension

Matrix study required

Independent lab data required if total

claimed matrices for

the method exceeds 4


Emergency Response Validation (ERV) Program

Purpose is to respond immediately to emerging food contamination crisis

Salmonella

contamination in peanut butter

Melamine in infant formula

Provides a rapid one-matrix, one-analyte

validation of

candidate method

Candidate methods evaluated using blind-coded samples prepared by independent lab

Special “Certificate of Validation”

granted upon approval

PTM certified and OMA validated methods eligible (where applicable)


Cancellation of PTM Certificate

If a licensee has:

Not complied with original agreement relative to the use of the mark

Not responded adequately to complaints of poor performance reported by users

Modified the test method without notifying the RI

Failed to submit a renewal application

Requested PTM status to be discontinued


Complaints

Formal complaints from applicant to be

directed to the AOAC RI Managing Director in

writing

Formal complaints from test method user

discussed with the method licensee

Failure to address user complaints will result in the

potential cancellation of PTM certificate


Potential Problems

Incomplete method instructions

Too vague

Important parameters not described

Method not fully adapted to new matrices

Clinical method often applied to foods and other matrices that vary greatly in protein, fat, and other components

Manufacturing and quality control problems


Summary

Background

Validation Overview


Qualitative Study Design

Quantitative Study Design


Report and Review



Proposed Revisions to the AOAC Guidelines for Microbiological Method Validation


AOAC RI Disclaimer

The proposed revisions contained in this presentation are being reviewed but have not been approved by the AOAC Methods Committee on Microbiology nor the Committee on Statistics

This presentation is applicable to the Performance Tested Methods

Program and the

Official Methods

of Analysis Program

Copyright © 2009 AOAC Research Institute147

Proposed Revisions

New terms and concepts for Qualitative Methods

POD –

Probability of Detection

Example analysis of qualitative SLV data

LPOD –

Cross-laboratory Probability of Detection

Repeatability

Reproducibility

Example analysis of qualitative collab

data

Study design changes

MPN estimation


POD: Probability of Detection

The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given bacterial level or concentration.

POD = x/N

For example, if 17 out of 20 replicates at a given concentration are positive by the candidate method, then the candidate method POD is 17/20 = 0.85 or 85% at that concentration.

Report POD values with 95% confidence intervals


POD: Probability of Detection

PODC

= POD of candidate method

PODR

= POD of reference method

PODCP

= POD of candidate method presumptive result

PODCC

= POD of candidate method confirmed result

Replaces sensitivity and specificity

No distinction between matched and unmatched analyses

Does not require a reference method


Graph POD Values vs. Concentration

Listeria Detection

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 2 4 6 8 10 12

Concentration (MPN/25g)

Prop

ortio

n D

etec

ted

(PO

D)

CP CC C R


dPOD: Difference of Two POD Values

dPOD

values are differences between two POD values:

dPODC

= PODC - PODR

dPODCP

= PODCP - PODCC

Calculate 95% confidence intervals on dPOD

values

If the confidence interval on dPOD

does not contain zero, then the two POD values are statistically different.


LPOD: Cross Laboratory POD

Calculated as the mean POD across laboratories in a collaborative study:

LPOD = ∑x / ∑N

LPODC , LPODR , LPODCP , LPODCC

Calculate 95% confidence intervals of LPOD values

Graph LPOD vs. Concentration


dLPOD: Difference of Two LPOD Values

dLPOD

values are differences between two LPOD values:

dLPODC

= LPODC

– LPODR

dLPODCP

= LPODCP

– LPODCC

Calculate standard deviation of dLPOD, sdPOD

Calculate 95% confidence intervals of dLPOD

values

If the confidence interval on dLPOD

does not contain zero, then the two LPOD values are statistically different.


Repeatability

Calculate repeatability standard deviation, sr

sr

= √LPOD(1-

LPOD)

Calculate repeatability standard error, SEr

SEr

= sr

/ √n


Reproducibility

Calculate the reproducibility standard deviation, sR

sR

= √∑

(PODl

–

LPOD)2/ L-1

Calculate 95% confidence interval on sR


Comparison of Repeatability and Reproducibility

Compare SEr

to sR

If SEr

falls within the 95% confidence interval of sR

, then the laboratory effect is not significant.

The contribution of the laboratory effect to the variation in POD values across laboratories can be estimated by

slab

= √(sR2

– (sr2/n))

If sR

< Ser , then report slab

as 0%.


Study Design Changes

Matrix Study –

Qualitative Methods

Single Lab

At least 3 levels: POD = 0, POD = 0.25-0.75, POD = 1.0

20 replicates each level

Collab

Minimum of 8 laboratories

At least 3 levels: LPOD = 0, LPOD = 0.25-0.75, LPOD = 1.0

12 replicates each level


MPN Estimation

SLV

5-tube 3-level MPN

2-

or 3-fold serial dilutions

Can use reference method replicates in

calculation of MPN –

formula provided

Collab

Use reference method replicates from all labs


Reminder

The proposed revisions contained in this presentation are being reviewed and have not been approved by the AOAC Methods Committee on Microbiology nor the Committee on Statistics

Questions?

Special Thanks to our AOAC

Statistical Advisors:

Paul Wehling, Robert LaBudde, and

Daryl Paulson

Documents

PTM Overview and Consulting - Pàgines de la UABjornades.uab.cat/workshopmrama/sites/jornades.uab.cat.workshopmrama/... · Regulate products Buy or sell food, beverage products, or