Analyzing the Power and Error of Listeria monocytogenes Growth Challenge Studies Mark Powell U.S....

Analyzing the Power and Error of Listeria monocytogenes Growth

Challenge StudiesMark Powell

U.S. Department of Agriculture

Washington, DC

IAFP 2009, Grapevine, TX, 12-15 July

Introduction

• For ready-to-eat (RTE) foods that do not support growth of L. monocytogenes, food safety criteria limit of 100 colony forming units (cfu)/g.– EC Regulation 2073/2005– FDA (2008) draft compliance policy guide– Codex (2009) microbiological criteria

• For RTE foods that do support growth of L. monocytogenes, “zero tolerance” (i.e., not detected in a regulatory sample).

• Design and interpretation of challenge studies to determine whether RTE are unable to support growth of L. monocytogenes.

Introduction

• Type I (F+) error (α): probability that H0 is rejected when true.

• Type II (F-) error (β): probability that H0 is not rejected when Ha is true.

• Power = (1-β). • By convention, α ≤ 0.05 and (1-β) ≥ 0.8

Fixed Exceedance Values

• To distinguish real growth from measurement uncertainty, L. monocytogenes challenge study protocols apply a fixed exceedance value: difference (δ) < M.

• EU/CRL (2008): difference between the initial and final sample median concentrations < 0.5 log10 cfu/g for all batches tested (Mm = 0.5 log).

• CCFH (2009): ≤ (on average) 0.5 log10 cfu/g increase for at least the expected shelf life (Mxbar = 0.5 log).

• FDA (2008): < 1 log10 increase during replicate trials (assume Mxbar = 1 log).

Fixed Exceedance Values

• M ~ ISO “expanded uncertainty” (U)

• x ± U = x ± 2σx

– where σx = std. error of meas. uncertainty

• 2 (k factor) ≈ z(1-0.05/2) = 1.96

– α = 0.05; 2-sided interval

• If σx = 0.25 log, → M = 0.5 log

• If σx = 0.50 log, → M = 1.0 log

Variance of a difference

• If two means are independent:

– where:

– Assuming equal σx and n:

222

00 txtxtxtx

nxx /22

2n if ONLY 2 2

0 x

xtxtx n

Quantitative Measurement Uncertainty

• σx ≠ 0.25 or 0.5 or any other fixed value.• EC, FDA, and CCFH reference ISO

Method 11290-2 for enumerating L. monocytogenes in RTE foods.

• Scotter et al (2001): std dev reproducibility (sR) = 0.17 - 0.45 log cfu/g in food samples.

• sR: an intra-laboratory measure of quantitative measurement uncertainty.

Challenge Study Designs Differ

• Number of sampling times• Number of batches• Experiment-wise α depends on:

– Number of comparisons– Whether multiple comparisons are

independent or dependent.• Independent: (μfinal – μinitial) X multiple

batches• Dependent: μ(t) – μ(t0) within a batch

Challenge Study Designs Differ

• EU/CRL (2008): k = 2 sampling times (initial and final), b ≥ 3 batches, sample size (n) = 3 samples per sampling time.– c ≥ 3 multiple, independent pair-wise comparisons.– std dev w/in batch < 0.3 log at t0.

• FDA cites Scott et al. (2005): k = 5-7 sampling times, sample size (n) = 2-3 samples per sampling time.– c = k-1 dependent pair-wise comparisons per trial

(μ(t) – μ(t0)).– No minimum number of batches.

Type I error for fixed exceedance value (Mxbar)

• For a single comparison test of H0: δ ≤ 0:

• For multiple independent comparisons:

• For multiple dependent comparisons, Monte Carlo simulation, with α = proportion (F+)

• Based on Scotter et al (2001), consider σx from 0.15 log cfu/g to 0.50 log cfu/g

nMMtxtxp x

xx

2

0

2,0|1

c 11

Type I error for difference in means fixed exceedance value (Mxbar) = 0.5 log cfu/g

** p<0.01

std. dev. (log

cfu/g)

sample size (n) = 2 sample size (n) = 3

independent comparisons (c) independent comparisons (c)

1 2 3 4 5 6 1 2 3 4 5 6

p(type I error) ≤ α

0.15 ** ** ** ** ** ** ** ** ** ** ** **

0.20 0.01 0.01 0.02 0.02 0.03 0.04 ** ** ** ** 0.01 0.01

0.25 0.02 0.04 0.07 0.09 0.11 0.13 0.01 0.01 0.02 0.03 0.04 0.04

0.30 0.05 0.09 0.14 0.18 0.22 0.25 0.02 0.04 0.06 0.08 0.10 0.12

0.35 0.08 0.15 0.21 0.27 0.33 0.38 0.04 0.08 0.12 0.15 0.19 0.22

0.40 0.11 0.20 0.28 0.36 0.43 0.49 0.06 0.12 0.18 0.23 0.28 0.32

0.45 0.13 0.25 0.35 0.44 0.51 0.58 0.09 0.17 0.24 0.30 0.36 0.42

0.50 0.16 0.29 0.40 0.50 0.58 0.65 0.11 0.21 0.30 0.37 0.44 0.50

Type I error for difference in means fixed exceedance value (Mxbar) = 0.5 log cfu/g

** p<0.01

std. dev. (log

cfu/g)

sample size (n) = 2 sample size (n) = 3

dependent comparisons (c) dependent comparisons (c)

1 2 3 4 5 6 1 2 3 4 5 6

p(type I error) ≤ α

0.15 ** ** ** ** ** ** ** ** ** ** ** **

0.20 0.01 0.01 0.02 0.02 0.03 0.03 ** ** ** ** 0.01 0.01

0.25 0.02 0.04 0.06 0.07 0.08 0.10 0.01 0.01 0.02 0.02 0.03 0.03

0.30 0.05 0.09 0.11 0.14 0.16 0.18 0.02 0.04 0.05 0.06 0.08 0.09

0.35 0.08 0.13 0.17 0.21 0.23 0.26 0.04 0.07 0.09 0.12 0.14 0.15

0.40 0.11 0.18 0.23 0.27 0.31 0.33 0.06 0.11 0.14 0.17 0.20 0.22

0.45 0.13 0.22 0.28 0.33 0.36 0.39 0.09 0.14 0.19 0.23 0.26 0.29

0.50 0.16 0.25 0.32 0.37 0.41 0.45 0.11 0.18 0.24 0.28 0.32 0.34

Power of F-test for One-Way ANOVA

• SAS© PROC Power

– where:– Fω = non-central F dist– Fcrit = critical value of the F dist with k-1 and k(n-1) df

– ω (non-centrality parameter) = – H0: μi = μ for all i– Ha: μmax – μmin = δ– Power depends on δ and growth pattern under Ha

,,|111 21 dfdfFF crit

2

1

2 /.

k

iin

Pattern that maximizes power for δ = 1 log

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6

time

log1

0 cf

u/g

Pattern that minimizes power for δ = 1 log

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6

time

log1

0 cf

u/g

Power curves for one-way ANOVA F-test (α = 0.05, δ = 1 log cfu/g)

with sample size n = 2 and sampling times k = 2-7

0.0

0.10.2

0.3

0.4

0.50.6

0.7

0.80.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Std Dev (log10 cfu/g)

Po

we

r

n=2 k=7

n=2 k=6

n=2 k=5

n=2 k=4

n=2 k=3

n=2 k=2

max

min

Power curves for one-way ANOVA F-test (α = 0.05, δ = 1 log cfu/g)

with sample size n = 3 and sampling times k = 2-7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Std Dev (log10 cfu/g)

Po

we

r

n=3 k=7

n=3 k=6

n=3 k=5

n=3 k=4

n=3 k=3

n=3 k=2

max

min

Conclusions

• Applying any fixed acceptance criteria exceedance value (e.g., less than a 0.5 log or 1 log increase) to distinguish real growth from quantitative measurement uncertainty over different experimental designs and/or measurement uncertainty values implies highly inconsistent type I error probabilities.

Conclusions

• None of the L. monocytogenes growth challenge study designs currently being considered are likely to provide an F-test with α = 0.05 and power ≥ 0.8 to detect a 1 log increase in mean concentration over the entire range of measurement uncertainty values for enumeration of L. monocytogenes reported in food samples in a validation study of ISO Method 11290-2.

Conclusions

• Satisfying these conventional experimental design criteria would require a larger sample size, lower measurement uncertainty, or both.

Thank you