Verification of performance characteristics of … Method Verification Workshop, 23 Sep 2008 1 Verification of performance characteristics of standardised methods Franz Ulberth European

AOAC Method Verification Workshop, 23 Sep 2008 1

Verification of performance

characteristics of standardised

methods

Franz Ulberth

European Commission

Joint Research Centre

Institute for Reference Materials and Measurements

http://www.irmm.jrc.be

http://ec.europa.eu/dgs/jrc/index.cfm


Where does the JRC fit

in the European Commission?

Commissioner Potočnik

Science and Research

Joint Research Centre (JRC)

President Barroso 27 Commission Members

Research DG (RTD)


The Joint Research Centre

Centre of scientific and

technical reference

for the conception,

development and

monitoring of EU policies


7 Institutes in 5 Member States 2800 staff

IE - Petten The Netherlands

-Institute for Energy

IRMM - Geel Belgium

- Institute for Reference Materials and

Measurements

ITU - Karlsruhe Germany

- Institute for Transuranium Elements

IPSC - IHCP - IES - Ispra Italy

- Institute for Environment and Sustainability

- Institute for Health and Consumer Protection

- Institute for the Protection and Security of the

Citizen

IPTS - Seville Spain

- Institute for Prospective Technological Studies

JRC-IRMM 2007

Mission of JRC Institute for Reference Materials

and Measurements

The mission of the IRMM is to promote a common

and reliable European measurement system

in support of EU policies

IRMM - CONFIDENCE IN MEASUREMENTS®


Mutual recognition of measurement data

• to avoid unnecessary duplication of efforts

• to foster confidence in certificates of analysis

• to promote international trade

– driven by ISO/IEC 17025 and accreditation bodies

– ISO/IEC 17025 requires uncertainty and traceability

statements

“confidence and acceptance of

accredited laboratory data”


Pillars of analytical quality control

Quality of analytical data

Measures to build trust

Re

fere

nce

ma

teri

als

Va

lida

ted

me

tho

ds

Pro

ficie

ncy te

stin

g


Standard development

../../pages/contact.php


Standard

A document established

• by consensus, and

• approved by a recognized body, that provides, for

common and repeated use,

• guidelines or characteristics for activities or their

results, aimed at the

• achievement of the optimum degree of order in a

given context.

ISO/IEC Guide 2 (1996)


Standards as knowledge resource

• Easily available

• Complete documentation

• Proven performance

• Acceptance

• Consensus

• Market driven

http://www.fil-idf.org/Content/Default.asp?PageID=422


Method performance criteria

• applicability (matrix and concentration range)

• calibration, linearity, working range, sensitivity

• limit of detection

• limit of quantification

• precision (repeatability, reproducibility)

• trueness (recovery)

• selectivity / specificity

• ruggedness

• (measurement uncertainty)


EURACHEM Guide: The Fitness for Purpose of Analytical

Methods: A Laboratory Guide to Method Validation and

Related Topics

• It is often assumed that standard methods can be used

straight off the shelf and the published performance data

achieved straight away by whoever uses the method. This is

not a safe assumption.

• Competence might be established in terms of the analyst’s

ability to achieve the levels of performance stated in the

method, such as repeatability, limit of detection, etc.

Performance verification of standardised methods (I)


Performance verification of standardised methods (II)

IUPAC Harmonized Guidelines for Single Laboratory

Validation of Methods of Analysis

• A laboratory using a collaboratively studied method,

which has been found to be fit for the intended purpose,

needs only to demonstrate that it can achieve the

performance characteristics stated in the method

• The laboratory should undertake precision studies, bias

studies (including matrix variation studies), and possibly

linearity studies


ALAAC Guide

The purpose of the guide is to

define the activities that are

required to fulfil method

verification based on

analytical method performance

characteristics


Exercise

• Design an experimental plan to demonstrate that your laboratory has

the competence to implement a standardised method for the analysis

of aflatoxin B1 in peanut butter and achieve equivalent performance

characteristics as given in AOAC 999.07

AOAC Official Method 999.07

Aflatoxin B1 and Total Aflatoxins in Peanut Butter, Pistachio Paste, Fig Paste, and

Paprika Powder

Immunoaffinity Column Liquid Chromatography with Post-Column Derivatization

First Action 1999, Final Action 2008

Test portion is either extracted with MeOH–H2O (8 + 2) or MeOH–H2O (8 + 2) plus hexane (or

cyclohexane). Ex tract is filtered, diluted with water to a specified solvent concentration, and

applied to an affinity column containing antibodies specific to aflatoxins B1, B2, G1, and G2.

Aflatoxins are removed from the affinity column with MeOH and are quantified by reversed-

phase liquid chromatography (LC) with post-column derivatization (PCD) involving bromination.

PCD is achieved with either electrochemically generated bromine (Kobra cell) or with

pyridinium hydrobromide perbromide (PBPB) and determined by fluorescence detection.


Considerations

• Selection of suitable test materials

• Selection of suitable reference standards

• Deviations from collaborative study testing conditions

with respect to sampling, sample pre-treatment, and

matrix effects

• Number of experiments to be carried out (e.g.

calibration, precision and bias study)

• Statistical data treatment

• Regulatory aspects


Work flow

Produce a list of potential

deviations from collab

study conditions

Are

conditions

equivalent?

Evaluate/take into

account/correct

influence of

deviating

conditions

Do test items vary

with respect to

composition and

analyte level(s)

Carry out precision and

study on a

representative test item

(matrix/analyte(s) at one

or several levels)

Carry out precision and

study on several

representative test items

(different

matrices/analyte(s) at

one or several levels)

NO

YES

YESNO


Work flow (cont’d)

Suitable

reference

material(s)

available?

Blank matrix(ces)

available for

spiking

Replicate analyses of

reference material(s)


spiked blank matrix(ces)


spiked sample(s)

NO

NO

YES

YES


Sources of suitable reference materials

Certified Reference Materials

• NIST http://ts.nist.gov/measurementservices/referencematerials/index.cfm

• IRMM http://irmm.jrc.ec.europa.eu/html/reference_materials_catalogue/catalogue/index.htm

• COMAR http://www.comar.bam.de/

Quality Control Materials

• FAPAS http://www.fapas.com/tmsales.cfm

http://ts.nist.gov/measurementservices/referencematerials/index.cfm

http://ts.nist.gov/measurementservices/referencematerials/index.cfm

http://irmm.jrc.ec.europa.eu/html/reference_materials_catalogue/catalogue/index.htm

http://irmm.jrc.ec.europa.eu/html/reference_materials_catalogue/catalogue/index.htm

http://www.comar.bam.de/

http://www.fapas.com/tmsales.cfm


Boundary conditions

• Actual testing conditions do not differ to a large extent

from those during the collaborative study:

– Sample pre-treatment

– Instrumentation

– Immunoaffinity columns

• Peanut butter considered a uniform product

• Blank materials for recovery (spiking) experiments and

certified calibration materials available

• Well characterised quality control materials available (low

level and high level contamination)


Study design

Calibration using CRM solutions

Evaluation of calibration function

Precision study: analysis of blank and spiked

materials, and of naturally contaminated materials

Bias study: analysis of well characterised naturally

contaminated materials

Statistical evaluation


Selectivity/Specificity

Typical chromatogram of aflatoxins in pistachio paste after immunoaffinity clean-up

(contamination level 1 ng/g aflatoxin B1)

54.5

55

55.5

56

56.5

57

57.5

58

58.5

59

59.5

0 2 4 6 8 10 12 14 16

time in min

mV

B1

B2

G1

Operating conditions:

Injection volume: 200 µL

Column: LC-18 (25 cm and 4.6 mm inner diameter)

Flow rate: 1 mL/minute

Mobile phase: water-methanol-acetonitrile (6+3+2 [v/v/v]),

containing 120 mg KBr and 350 µL HNO3 of (c {HNO3}=4 Mol/L)

Derivatisation: electrochemical bromination (KOBRA-cell)

Detection: Fluorescence (Ex.:365 nm, Em.:435 nm)


Aflatoxin B1 in peanut butter by HPLC

Calibration

- 5 levels

- 2 dilutions per level

- 2 injections/dilution

Conc.

[ng/ml]Inject.

R. Time

[min]Peak Area Average RSD %

0.404 1 10.83 185287 187855 1.10

0.404 2 10.75 190367

0.404 1 10.88 187967

0.404 2 10.86 187797

1.212 1 10.79 576293 580462 1.23

1.212 2 10.81 576428

1.212 1 11.02 577982

1.212 2 10.97 591145

2.02 1 10.74 969239 980551 2.14

2.02 2 10.82 960628

2.02 1 11.05 1008660

2.02 2 10.98 983677

2.828 1 10.74 1346150 1364427 1.82

2.828 2 10.78 1339951

2.828 1 11.03 1387377

2.828 2 11.03 1384228

3.636 1 10.72 1714052 1749339 2.09

3.636 2 10.76 1722359

3.636 1 11.01 1788018

3.636 2 11.03 1772928


Calibration function

MS Excel functions

Regression Statistics

Multiple R 0.999

Standard Error 5757.4

Observations 5

Coefficients P-value Lower 95% Upper 95%

Intercept -4206.85 0.660961167 -24026.4358 15612.73576

Slope 483531.436 1.60028E-27 474991.468 492071.4035

ANOVA

df SS MS F Significance F

Regression 1 6.10565E+12 6.1057E+12 14150.01252 1.6003E-27

Residual 18 7766901932 431494552

Total 19 6.11342E+12

0

500000

1000000

1500000

2000000

0 0.5 1 1.5 2 2.5 3 3.5 4

Aflatoxin B1 (ng/mL)

Are

a

-50000

-40000

-30000

-20000

-10000

0

10000

20000

30000

40000

0 0.5 1 1.5 2 2.5 3 3.5 4

Aflatoxin B1 (ng/mL)

Resid

uals


Weighted regression

Weights=1/s2

Conc.

[ng/ml]Average Variance

w w*x w*y (x-xw) (y-yw) w(x-xw)^2 w(x-xw)(y-yw)

0.404 187855 4.31E+06 232.19 93.80 43617187.2 -0.100436 -48833.97382 2.342140852 1138797.74

1.212 580462 5.13E+07 19.49 23.62 11312799 0.707564 343773.5262 9.757262665 4740613.673

2.02 980551 4.42E+08 2.26 4.57 2220492.49 1.515564 743862.5262 5.201490452 2552972.543

2.828 1364427 6.17E+08 1.62 4.58 2210307.01 2.323564 1127738.026 8.746047147 4244879.432

3.636 1749339 1.34E+09 0.75 2.71 1303661.42 3.131564 1512650.776 7.308250341 3530130.565

Sum 256.31 129 60664447 33 16207394

Slope 485903.191

Intercept -8418.4836

sy.x.w 7929.92987


Comparison of calibration models

• Unweighted regression

• Weighted regression

Average

(area) Known Predicted U U% Lower Upper

187855 0.404 0.397 0.0350 8.81 0.362 0.432

580462 1.212 1.209 0.0281 2.33 1.181 1.237

980551 2.020 2.037 0.0254 1.25 2.011 2.062

1364427 2.828 2.830 0.0281 0.99 2.802 2.859

1749339 3.636 3.627 0.0348 0.96 3.592 3.661

Average

(area) Known Predicted U U% Lower Upper

187855 0.404 0.404 0.0350 8.66 0.369 0.439

580462 1.212 1.212 0.0354 2.92 1.177 1.247

980551 2.020 2.035 0.0375 1.84 1.998 2.073

1364427 2.828 2.825 0.0410 1.45 2.784 2.866

1749339 3.636 3.617 0.0450 1.24 3.572 3.662

2

2

0

11

)xx(

)xx(

np*

b

s)c(u

i

pred


Spiking (recovery) experiments

Design:

5 Sub-samples of blank material

5 Sub-samples spiked at 1.01 ng/g

(low spike)

5 Sub-samples spiked at 4.04 ng/g

(high spike)

Sample Inject.R. Time

[min]Peak Area

Conc.

[ng/g]

Recov

ery

[%]

Average

Recovery

[%]

SDRSD

[%]

Spike 1 1 10.88 185964 0.81 80 84 2.07 2.47

Spike 1 2 10.84 191822 0.83 82

Spike 2 1 10.84 195155 0.85 84

Spike 2 2 10.82 198041 0.86 85

Spike 3 1 10.83 194510 0.84 84

Spike 3 2 10.84 205081 0.89 88

Spike 4 1 10.84 195261 0.85 84

Spike 4 2 10.83 198270 0.86 85

Spike 5 1 10.77 192658 0.84 83

Spike 5 2 10.86 192735 0.84 83

Spike 6 1 10.88 781332 3.30 82 83 2.08 2.52

Spike 6 2 10.81 816842 3.45 85

Spike 7 1 10.78 803146 3.39 84

Spike 7 2 10.89 799423 3.37 83

Spike 8 1 10.83 758903 3.20 79

Spike 8 2 10.75 808367 3.41 84

Spike 9 1 10.86 782667 3.30 82

Spike 9 2 10.85 789750 3.33 82

Spike 10 1 10.85 760450 3.21 79

Spike 10 2 10.81 809207 3.41 85

Low spike High spike

Mean [ng/g] 0.85 3.34

SD 0.021 0.084

RSD 2.47 2.52

Recovery (%) 84 83


Naturally contaminated material

Design:

4 sub-samples of

well characterised

peanut butter

containing 0.8 and

3.4 ng/g Afla B1

Sample Inject.R. Time

[min]Peak Area

Conc.

[ng/g]

Recovery

[%]

Average

Recovery

[%]

SDRSD

[%]

nat. cont. 1 1 10.61 27535 0.76 95 103 5.39 5.24

nat. cont. 1 2 10.64 32152 0.88 110

nat. cont. 2 1 10.62 30517 0.84 104

nat. cont. 3 1 10.59 28790 0.79 99

nat. cont. 4 1 10.59 29247 0.80 100

nat. cont. 4 2 10.58 30867 0.84 106

nat. cont. 5 1 10.63 28643 0.79 98

nat. cont. 5 2 10.59 32044 0.88 109

nat. cont. 6 1 10.64 118451 3.09 91 93 3.80 4.09

nat. cont. 6 2 10.61 123997 3.23 95

nat. cont. 7 1 10.59 112742 2.94 87

nat. cont. 7 2 10.67 120109 3.13 92

nat. cont. 8 1 10.62 117933 3.08 91

nat. cont. 8 2 10.69 129078 3.36 99

nat. cont. 9 1 10.69 122073 3.18 94

nat. cont. 9 2 10.63 125263 3.27 96

Low High

Mean [ng/g] 0.82 3.16

SD 0.008 0.087

RSD 0.92 2.74

Recovery (%) 103 93


Verification of precision

To demonstrate that repeatability is consistent with the

repeatability standard deviation obtained in the course of the

collaborative exercise

Mean [ng/g] # Labs sr RSDr Mean [ng/g] # Repl. sr RSDr F Fcrit (95 %)

Spiked 0.9 15 0.09 10 0.85 5 0.02 2.5 0.233 4.56

Spiked 3.6 13 0.11 3 3.34 5 0.08 2.5 0.764 4.86

Contaminated 0.8 15 0.05 6 0.82 4 0.01 1.0 0.160 4.89

Contaminated 1.5 14 0.10 7

Contaminated 3.4 14 0.13 4 3.16 4 0.09 2.8 0.669 5.04

AOAC 999.07 Verification study


Demonstrating control of bias

To demonstrate that bias of the method is under control

Mean [ng/g] # Labs sr sR Mean [ng/g] # Repl. sr Δ 2σD

Contaminated 0.8 15 0.05 0.26 0.82 4 0.01 0.02 0.51

Contaminated 3.4 14 0.13 0.65 3.16 4 0.09 0.24 1.28

AOAC 999.07 Verification study

|Δ| < 2√ (sR2 - sr

2 + sw2/n)


Conclusions

• Verification gives objective proof that laboratory has

competence to carry out standardised procedure in

compliance with specified performance criteria

• Intended purpose of the method governs

performance criteria to be verified

• Verification results can be used to estimate

measurement uncertainty


www.jrc.cec.eu.int

www.jrc.irmm.be

[email protected]

Useful links and contacts

Documents

Verification of performance characteristics of … Method Verification Workshop, 23 Sep 2008 1 Verification of performance characteristics of standardised methods Franz Ulberth European