Upload
syoolove
View
221
Download
0
Embed Size (px)
Citation preview
7/30/2019 Food Safety Risk Assessment Tool
1/15
A simple, spreadsheet-based, food safety risk assessment tool
Thomas Ross*, John Sumner
Centre for Food Safety and Quality, School of Agricultural Science, University of Tasmania, GPO Box 252-54, Hobart,
Tasmania 7001, Australia
Received 21 June 2001; received in revised form 13 November 2001; accepted 18 January 2002
Abstract
The development and use of a simple tool for food safety risk assessment is described. The tool is in spreadsheet software
format and embodies established principles of food safety risk assessment, i.e., the combination of probability of exposure to a
food-borne hazard, the magnitude of hazard in a food when present, and the probability and severity of outcomes that might
arise from that level and frequency of exposure. The tool requires the user to select from qualitative statements and/or to provide
quantitative data concerning factors that that will affect the food safety risk to a specific population, arising from a specific food
product and specific hazard, during the steps from harvest to consumption. The spreadsheet converts the qualitative inputs into
numerical values and combines them with the quantitative inputs in a series of mathematical and logical steps using standard
spreadsheet functions. Those calculations are used to generate indices of the public health risk. Shortcomings of the approach
are discussed, including the simplifications and assumptions inherent in the mathematical model, the inadequacy of data
currently available, and the lack of consideration of variability and uncertainty in the inputs and outputs of the model. Possibleimprovements are suggested. The model underpinning the tool is a simplification of the harvest to consumption pathway, but
the tool offers a quick and simple means of comparing food-borne risks from diverse products, and has utility for ranking and
prioritising risks from diverse sources. It can be used to screen food-borne risks and identify those requiring more rigorous
assessment. It also serves as an aid to structured problem solving and can help to focus attention on those factors in food
production, processing, distribution and meal preparation that most affect food safety risk, and that may be the most appropriate
targets for risk management strategies. D 2002 Elsevier Science B.V. All rights reserved.
Keywords: Food safety; Hazard analysis; Qualitative risk assessment; Relative risk; Spreadsheet
1. Introduction
Formal risk assessment techniques have been devel-
oped and exploited in many areas of human enterprise
and activity for decades (NRC, 1983, 1994, 1996;
Morgan, 1993; Bernstein, 1996). The application of
risk assessment techniques to food safety issues is
being strongly promoted by national and international
organisations (CAST, 1994; Kindred, 1996; ILSI,
1996; WHO/FAO, 1999) and several authors have
reviewed the application of risk assessment methods
to food safety (Jaykus, 1996; Kindred, 1996; Lam-
merding, 1997; ICMSF, 1998; Voysey and Brown,
2000).
0168-1605/02/$ - see front matterD 2002 Elsevier Science B.V. All rights reserved.P I I : S 0 1 6 8 - 1 6 0 5 ( 0 2 ) 0 0 0 6 1 - 2
* Corresponding author. Tel.: +61-3-6226-1831; fax: +61-3-
6226-2642.
E-mail address: [email protected] (T. Ross).
www.elsevier.com/locate/ijfoodmicro
International Journal of Food Microbiology 77 (2002) 3953
7/30/2019 Food Safety Risk Assessment Tool
2/15
Risk assessment involves the identification of a
hazard and the methodical description of a system,
and its failures, which could give rise to that hazard,
including all possible routes whereby that hazard canarise. This qualitative description can be made quanti-
tative by expressing in mathematical terms, the system,
and the relationships between those elements that
contribute to the risk. Full quantitative assessment of
the risk can be achieved if the distributions of values of
the factors in the system that contribute to the risk are
known. Approaches that use all of this information, the
so-called stochastic, or probabilistic treatments,
are the preferred option for risk assessment (Vose,
1996;Cassin et al., 1998a). Methods for microbial food
safety risk assessment are being developed by various
organisations (FAO, 1995; CAC, 1999; Kindred, 1996;
ILSI, 1996; Buchanan, 1997; PCCRARM, 1997) and,
since the mid-1990s, a number of microbiological risk
assessments have been presented. These were sum-
marised by Schlundt (2000). Others have since been
released for public comment/peer review (WHO/FAO,
2000a,b,c, 2001; FDA/FSIS, 2001a,b).
The effort expended to assess a specified risk
should be commensurate with the magnitude of that
risk and, in general, there is a large difference in effort
and rigour between qualitative and quantitative risk
assessment. The latter are typically expensive, labourintensive and technically demanding processes taking,
in some cases, many person-years to complete (FSIS,
1998; FDA/FSIS, 2001a). Despite this, many food
safety risk assessments have concluded that there are
insufficient data to enable a reliable numerical esti-
mate of risk within narrow confidence limits. Pre-
screening of the risk by simpler methods can aid
decisions about the value of investing resources in
fully quantitative risk assessments.
Risk managers may have difficulty comparing risks
from different sources for risk management prioritisa-tion. The fundamental objective of risk assessment is
to provide support for decisions, and there are a
number of decision-support tools to assist in deter-
mining whether a pathogen is, or could be, an impor-
tant hazard in a given food/food process combination.
These include various semi-quantitative scoring sys-
tems, decision trees,etc. (see, e.g. Notermans andMead,
1996; Todd and Harwig, 1996; ICMSF, 1996; Van
Schothorst, 1997). van Gerwen et al. (2000) presented
a step-wise approach and developed a computerised
expert system, named SIEFE, for quantitative micro-
biological risk assessment of food products and pro-
cesses that begins to address this problem. Schemes to
assist qualitative risk assessments have also been devel-oped (Corlett and Pierson, 1992; Huss et al., 2000).
While the approaches of Corlett and Pierson (1992)
and Huss et al. (2000) are valuable in categorising risk
and in directing broad mitigation strategies, neither
provides good discrimination of risk (e.g. neither could
be used to assess an as-yet undocumented risk), nor of
the effect of contributions to risk of individual risk-
affecting factors. Consequently, those schemes do not
focus attention on steps where control could most
effectively be applied.
In this paper, we describe a simple and accessible
food safety risk calculation tool intended as an aid to
determining relative risks from different product/
pathogen/processing combinations, that addresses
some of the shortcomings identified above.
2. Methods and materials
2.1. Development of the decision support tool
The decision-support tool was developed to assist
in translating an academic understanding of the riskassessment approach and philosophy into a useful tool
for ranking the risk from different food/hazard combi-
nations. In particular, the tool was intended to make
the techniques of food safety risk assessment more
accessible to non-expert users, both as a decision-aid
and an educational tool.
It was recognised that the tool would have to
incorporate all factors that affect the risk from a
hazard in a particular commodity including:
(1) Severity of the hazard.(2) Likelihood of a disease causing dose of the
hazard being present in a meal.
(3) Probability of exposure to the hazard in a
defined period of time.
In turn, it was recognised that a number of factors
affect each of the above. Disease severity is affected by:
(a) the intrinsic features of the pathogen/toxin, and
(b) the susceptibility of the consumer.
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 395340
7/30/2019 Food Safety Risk Assessment Tool
3/15
Exposure to the hazard will depend on how much
is consumed per meal by the population of interest,
how frequently they consume the food, and the size of
the population exposed.Probability of exposure to an infectious dose will
depend on:
(a) serving size,
(b) probability of contamination in the raw pro-
duct,
(c) initial level of contamination,
(d) probability of contamination at subsequent
stages in the farm-to-fork chain, and
(e) changes in the level of the hazard during the
journey from farm to fork, including, e.g.
simple concentration and dilution, growth or
inactivation of pathogens.
The tool was developed to assist users to describe
the product/pathogen/pathway of interest to them. In
prototypes, the user was prompted to choose from a
list of qualitative answers in response to each of a
series of simple questions, so that risk could be
estimated or compared without recourse to numerical
data. After trials, it was realised that relying on a
small and finite range of qualitative answers greatly
limited the ability of the tool to discriminate levelsof risk. Consequently, the capacity for users to pro-
vide numerical answers to some questions was in-
cluded.
2.2. User interface
The user interface represents a generic conceptual
model of the factors that contribute to food safety
risk.
The model was developed in MicrosoftR Excel
spreadsheet software, using standard mathematicaland logical functions. The List Box macro tool,
an inbuilt MicrosoftR Excel function available on the
Forms toolbar, was used to automate the conver-
sion from qualitative inputs to quantities for use in
calculations. The list box tool allows users to select
from a range of options by simply mouse-clicking
on their choice. The software converts that selection
into a numerical value.
The user is required to answer 11 questions,
related to the three main factors identified in Section
2.1. The underlying mathematical model equates
each descriptor with a numerical value or weight-
ing. The weightings currently used in the model are
detailed in Table 1. Some of the weightings arearbitrarily defined, while others are based on known
mathematical relationships, e.g. from days to weeks,
or years. To assist users to make objective and
reproducible responses, and to maintain transparency
of the model, examples of the subjective descriptors
are provided, or the weighting factors applied to
the descriptors are shown in the user interface
(see Fig. 1). Alternatively, where the options pro-
vided do not accurately reflect the situation being
modelled, users can enter a numerical value that is
appropriate.
Different iterations of the spreadsheet model were
tested by food safety managers. Through that process,
ambiguities in the structure of the questions were
revealed. Thus, the questions were modified to make
their intent clearer.
Questions 1 and 2 consider the susceptibility of the
population of interest and the severity of the illness.
The hazard severity is arbitrarily weighted by factors
of 10 for increasing levels of severity. The weighting
factors for susceptibility of various population sub-
groups include values for relative risk of infection/
intoxication for a variety of hazards. That weighting isloosely based on epidemiological data for relative
susceptibility to listeriosis calculated by Jurado et al.
(1993), Jones et al. (1994) and Nolla-Salas et al.
(1993). Absolute risk is based on the population size,
the proportion of the population consuming the food
and how frequently people eat the food. This infor-
mation is selected in Questions 35. The selection of
a sub-population from the general population (Ques-
tion 2) automatically reduces, by the proportions
indicated in Table 1, the total population estimated
to be exposed.Using Australian population age structure data
(ABS, 2000) and 1998 data from the US Center
for Disease Prevention and Control (cited in FDA/
FSIS, 2001a) for the proportion of listeriosis cases
in defined age categories, we also estimated the
relative susceptibility by age. The proportion of the
population in these categories in Australia was
estimated from ABS (2000), and also by Paoli
(1999, pers. comm.) for North American populations.
Both estimates were in the range of 1520%, consis-
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 41
7/30/2019 Food Safety Risk Assessment Tool
4/15
Table 1
Weighting values used in the current model (V.1)
Comment
1. Hazard severitySEVERE hazardcauses death to most victims 1 arbitrary weighting factors
MODERATE hazardrequires medical intervention
in most cases
0.1
MILD hazardsometimes requires medical attention 0.01
MINOR hazardpatient rarely seeks medical attention 0.001
2. How susceptible is the consumer?
GENERALall members of the population 1 100% of population
SLIGHTe.g., infants, aged 5 20% of population
VERYe.g., old, very young, diabetes, alcoholic etc. 30 3% of population
EXTREMEe.g. AIDS, transplants recipients,
cancer patients, etc.
200 0.1% of population
arbitrary weightings, but based on relative
susceptibility to listeriosis, populationestimates based on Australian health statistics
3. Frequency of consumption
daily 365 simple algebra
weekly 52
monthly 12
a few times per year 3
once every few years 0.3
4. Proportion of population consuming
all (100%) 1 arbitrary weighting factors
most (75%) 0.75
some (25%) 0.25
very few (5%) 0.05
5. Size of population of interest User selected or specified
6. Proportion of product contaminated?
Rare (1 in a 1000) 0.001 0.01% of samples
Infrequent (1%) 0.01 1% of samples
Sometimes (10%) 0.1 10% of samples
Common (50%) 0.5 50% of samples
All (100%) 1 all samples
OTHER user input
7. Effect of process
The process RELIABLY ELIMINATES hazards 0 arbitrary weighting factorsThe process USUALLY (99% of cases) ELIMINATES hazards 0.01
The process SLIGHTLY (50% of cases) REDUCES hazards 0.5
The process has NO EFFECT on the hazards 1
The process INCREASES (10 ) the hazards 10The process GREATLY INCREASES (1000 ) the hazards 1000
8. Is there a potential for recontamination?
NO 0 arbitrary weighting factors
YESminor (1% frequency) 0.01
YESmajor (50% frequency) 0.50
OTHER user input
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 395342
7/30/2019 Food Safety Risk Assessment Tool
5/15
tent with that of Lindqvist and Westoo (2000), Hitch-
ins (1996) and Buchanan et al. (1997). The propor-
tions in different susceptibility classes were used to
modify the number of cases predicted, as describedbelow.
The frequency of contamination of food and the
implications of subsequent processing and handling
are considered in Questions 6 9 and Question 11.
Some factors, such as processing or cooking, may
completely eliminate the risk. The model includes,
however, the possibility that re-contamination may
occur subsequently, and re-introduce risk. Subsequent
pathways of cross-contamination are not explicitly
considered in the model.
Neither the concentration of the hazard nor thesize of the serving is considered explicitly in the
model. Both factors are included indirectly in the
response to Question 10, which requests an estimate
of the increase required for the initial contamination
level to reach ID50.1 In the calculation of relative risk,
for pathogens believed to have a high infectious
doses, the distribution of pathogen loads in the food
system has little effect (WHO/FAO, 2001). Rather, it
is the total load of such pathogens in the foodsupply that determines the overall population health
risk.
The model multiplies the factors to calculate var-
ious measures of risk, described below. Some esti-
mates consider only the probability of illness, while
others also consider the severity to produce an esti-
mate of the risk of the illness and the numbers of
consumers affected. As a descriptor is selected or
changed, the risk estimates are automatically recalcu-
lated.
2.3. Structure of the tool and mathematical bases
Full details of the logic and equations leading to
the risk estimates are detailed below.
Four measures of risk are calculated.
To simplify the description of the calculation of
these values, it is useful to describe some intermediate
calculations. These are the following.
PDD: Probability of a disease-causing dose being
present in a portion of the product of interest. This is
1 The dose expected to result in 50% of the exposed population
becoming ill.
Table 1 (continued)
Comment
9. How much increase from level at processing is required to reach an infectious or toxic dose for the average consumer?
none 1 arbitrary weighting factorsslight (10-fold increase) 0.1
moderate (100-fold increase) 0.01
significant (10,000-fold increase) 0.0001
OTHER user input
10. How effective is the post-processing control system?
WELL CONTROLLEDsystems in place, audited, well-trained staff 1 arbitrary weighting factors
CONTROLLEDsystems in place, audited, well-trained staff 3
NOT CONTROLLEDno systems in place, untrained staff 10
GROSS ABUSE OCCURS 1000
NOT RELEVANTlevel of risk agent does not change 1
11. Effect of preparation for meal
Meal preparation RELIABLY ELIMINATES hazards 0 arbitrary weighting factorsMeal preparation USUALLY ELIMINATES (99%) hazards 0.01
Meal preparation SLIGHTLY REDUCES (50%) hazards 0.50
Meal preparation has NO EFFECT on the hazards 1.00
user-input
OTHER value
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 43
7/30/2019 Food Safety Risk Assessment Tool
6/15
defined as whichever is the larger value of the
product of
the proportion of product contaminated
value of Question 6 the effects of
processing on the probability of contamination
value of Question 7 the effect of
post processing handling=storage
value of Question 9 the increase in the
initial level of the factor required to reach an
infectious dose value of Question 10 the
effect of preparation prior to eating
value of Question 11
or
the proportion of product re contaminated
value of Question 8 the effect of
post processing handling=storage
value of Question 9 the increase in the initial
level of the factor required to reach an infectious
dose value of Question 10 the effect of
preparation prior to eating value of Question 11
The probability of a portion of food being con-taminated with a toxic dose cannot exceed 1. Accord-
ingly, if the value of the above calculations exceeds 1,
it is set equal to 1.
Pexp: Probability of exposure to the product per
person per day, given by:
the frequency of consumption
value of Question 3 proportion of the
population that consumes the product
value of Question 4
Exposure: Total number of portions of the product
of interest eaten per day in the general population,
given by:
the frequency of consumption
value of Question 3 proportion of the
population that consumes the product
value of Question 4 the total population
considered value of Question 5
The first measure: Probability of illness per con-
sumer per day is calculated as:
PDD Pexp
This value is not strictly a measure of risk, because
it does not include the severity of the illness resulting
from exposure to the hazard.
The second measure Total predicted illnesses/
annum in population of interest does not differentiate
severity either, but provides another measure that
might be more readily understood. Total predicted
illnesses/annum in population of interest is calcu-
lated as:
365 i:e: days per year Probability of illness
per consumer per day as described above
fraction of population considered in at risk
class part of Question 2 the total population
value of Question 10
The Comparative Risk in the population of
interest is a measure of relative risk and is independ-
ent of the size of the population, but does consider the
proportion of the population consuming. It is calcu-
lated as:
Probability of illness per day per consumer of
interest as described above Hazard severity
Question 1 Proportion of population
consuming Question 4 Proportion of total
population in population of interest
part of Question 2
and can be used to rank the relative risk of the
pathogen/product/process combination and consump-
tion patterns, independent of population size. Whenspecific sub-populations are selected at Question 2, the
estimate of the absolute number of cases among the
total population is amended by the weighting factors
shown in Table 1 for relative susceptibility to infection,
and also the proportion of the total population in that
sub-group. The model is constructed, and the weight-
ing factors selected, so that the Comparative Risk
can never exceed 1. A Comparative Risk of 1
represents the situation where every person in the
population consumes the product of interest daily,
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 395344
7/30/2019 Food Safety Risk Assessment Tool
7/15
and that each portion of the product contains a lethal
dose of the hazard.
The Comparative Risk measure is cumbersome,
and its numerical value is not readily understood as ameasure of risk. Relatively small changes in one of
the answers can lead to alarming changes in the
predicted number of cases. Furthermore, the specifi-
cation of the numerical value of risk is misleading, as
it provides no information regarding the confidence
one should place in that numerical estimate. To
provide a more user-friendly and robust index of
relative risk, the Risk Ranking measure was devel-
oped based on the Comparative Risk estimate.
The Risk Ranking value is scaled logarithmically
between 0 and 100, where 0 represents no risk, and 100
represents the opposite extreme where every member
of the population eats a meal that contains a lethal dose
of the hazard every day. To set the Risk Ranking
scale, we chose a probability of mild food-borne illness
of less than or equal to one case per 10 billion people
(greater than current global population) per 100 years
as a negligible risk. The Comparative Risk estimate
that corresponds to this value is 2.75 10 17. Weequated the Risk Ranking corresponding to this
level to zero. Analogously, we set the upper limit of
Risk Ranking at 100, corresponding to a Compa-
rative Risk of 1. All the estimates generated by themodel are based on the multiplication of factors, many
set at factor of 10 differences. The end-points of the
Risk Ranking scale lead to an increment of six Risk
Ranking units, corresponding approximately to a
factor of 10 difference in the absolute risk estimate.
Thus, Risk Ranking is defined as:
If
Comparative Risk Q 2.75 10 17 thenRisk Ranking= 0 or else Risk Rank-
ing=(100/17.56) (17.56 + LOG10(Compara-tive Risk)).
In the spreadsheet, the result is rounded to the
nearest integer value, reflecting the level of discrimina-
tion we believe appropriate given the bases of the tool.
2.4. Evaluating the tool
To relate the Risk Ranking scale to practical ex-
perience, we use predicted rates of food-borne illness
in Australia, estimated by ANZFA (1999), and the
estimates of Mead et al. (1999) for food-borne illness
in the USA, to generate Risk Ranking values.
To evaluate the performance of the tool, severalscenarios were modelled and compared to actual data
or other risk assessments. Specifically, conditions
leading to an outbreak of hepatitis A from consump-
tion of oysters in Australia in 1997 were simulated
using the tool, and compared to the epidemiological
data reported by Conaty et al. (2000).
Secondly, the data and assumptions of the quanti-
tative risk assessment of Cassin et al. (1998b), for the
risk of illness from enterohaemorrhagic E. coli in
hamburgers in north American culture, were used to
derive the answers to the questions of the risk assess-
ment spreadsheet. The results of both assessments
were compared.
3. Results
The model interface is shown in Fig. 1.
3.1. Risk ranking
ANZFA (1999) calculated the incidence of food-
borne disease in Australia as f 4,000,000 cases perannum. The vast majority of these cases were consid-
ered to pass unreported. Thus, we set Hazard Severity
(Question 1) to minor hazard. The ANZFA (1999)
estimates are for the total population: we set Question 2
to general. We manipulated other inputs so that the
Total Predicted Illnesses per annum in the Population
of Interest equalled f 4,000,000. Australia was
selected in Question 5. Under these, and all other
conditions leading to a total predicted 4,000,000 minor
food-borne illnesses among the Australian population
off
20 million, the Risk Ranking was 64.Mead et al. (1999) estimated that there were
76,000,000 cases of food-borne disease per year in
the USA, of which 325,000 resulted in hospitalisation
and 5000 caused deaths. Thus, we performed three
assessments. In the first, the Hazard Severity (Question
1) was set to minor hazard and the other questions
adjusted to yield an estimate of 76,000,000 illnesses. In
the second assessment, Hazard Severity (Question 1)
was set to moderate hazardrequiring medical inter-
vention in most cases, and the other questions manip-
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 45
7/30/2019 Food Safety Risk Assessment Tool
8/15
Fig. 1. User interface. The risk model user interface, using Australian populations as an example. Users mouse-click on their choice in each lis
required. As choices are made and values entered, the risk estimates are automatically recalculated. The values shown are those used in Case
7/30/2019 Food Safety Risk Assessment Tool
9/15
ulated to yield an estimate of 325,000 illnesses. In the
third, the Hazard Severity was set to severe haz-
ardcauses death to most victims, and the other
questions manipulated to yield an estimate of 5000cases. In all cases General population was selected
in Question 2, and other selected under Question 5,
with the USA population estimated at 270,000,000.
The Risk Ranking estimates for these three scenar-
ios were 65, 63 and 58, respectively.
3.2. Case study 1: viruses in oysters
An outbreak of hepatitis A involving consumption
of oysters occurred in Australia in 1997. The outbreak
is discussed in Conaty et al. (2000) from which it can
be estimated that production from the affected area
was approximately 280 bags of oysters per week. A
bag of oysters contains approximately 200 serves of
six oysters. The first positive samples were identified
from oysters harvested on 24 December and the area
was closed to further harvest on 14 February, indicat-
ing that contaminated oysters were harvested for up to
7 weeks. Thus, we estimate that the population was
exposed to 390,000 servings of potentially contami-
nated oysters. If spread over the entire Australian
population over an entire year, this would correspond
to 0.02 serves per person per annum. To represent thislevel of exposure in the model we selected Once
every few years at Question 3, and Very Few at
Question 4. (Note that, even though the exposure
occurred only over a 7-week period, we assume that
it was spread over an entire year, and that even though
the exposure was restricted to a geographic region,
that all Australian consumers were potentially
exposed, consistent with the above estimate of expo-
sure level.)
Conaty et al. (2000) report that of 63 samples of
one dozen oysters, 6 tested positive for hepatitis Avirus using a PCR method. From this, we assumed
that 5% of servings of six oysters were HAV-positive
(Question 5).
The level of contamination was not reported.
Where detection of enteric viruses in shellfish has
occurred the levels of contamination are low, ranging
from 0.3 to 200 plaque forming units (PFU) per 100 g
of shellfish (Jaykus et al., 1994; Rose and Sobsey,
1993; CAST, 1994), a typical serving size. Rose and
Sobsey (1993) presented a dose response model
relating the probability of infection with HAV to the
amount ingested. It suggests that the ID50 for HAV isf 500 PFU. Assuming an exponential doseresponse
relationship, 1 PFU would be expected to lead toinfection in 1 in 500 consumers. Thus, it appears
likely that not all contaminated serves would have a
high probability of causing infection. To implement
this relationship at Question 7, we selected OTHER
and entered 65 (the ID50 divided by the geometric
mean of the contamination per serving) at Question
10. The values used are summarised in Table 2.
Australia-wide during the outbreak period (20
January to 4 April), there were 444 cases of hepatitis
A associated with consumption of oysters, the vast
majority of which were believed to be due to oysters
from the contaminated area (Conaty et al., 2000).
Under the assumptions given above, the number of
cases predicted by the spreadsheet model was 225,
and the Risk Ranking was 52.
3.3. Case study 2: comparison to risk estimate of
Cassin et al. (1998b)
Cassin et al. (1998b) developed a process risk
model from which to determine the effectiveness of
a range of strategies to reduce the risk and incidence
Table 2
Values used in the assessment of risk from viral contamination of
oysters in Australia in an outbreak
Risk criteria Input appropriate
to outbreak
Dose and severity
1. Hazard severity Moderateoften requires
medical attention
2. Susceptibility Generalall population
Probability of exposure
3. Frequency of consumption once every few years
4. Proportion consuming very few (5%)
5. Size of population Australia (19,500,000)
Probability of infective dose
6. Probability of contamination Other (5% of servings)
7. Effect of Process Has no effect on the hazard
8. Possibility of recontamination None
9. Post-process control Not relevant
10. Increase to infective dose other (65)
11. Effect of treatment
before heating
Not effective in
reducing hazard
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 47
7/30/2019 Food Safety Risk Assessment Tool
10/15
of E. coli O157:H7 infections from hamburgers in the
North American cuisine.
It was difficult to make a direct comparison with
the results of Cassin et al. (1998b). Many of the valuesthat were required to be entered in the spreadsheet
were not given explicitly by those authors, but were
intermediate calculations in their model. However,
using data and statistics for the general population
presented in that report, we entered the values shown
in Table 3 into the spreadsheet.
We considered the total population, for whom
infection with E. coli O157:H7 will generally cause
mild disease (Questions 1 and 2). For more susceptible
individuals, medical attention will be required. The
population of the USA is approximately 270,000,000
(Question 5). Walls and Scott (1996) reported that on
any day in the USA, 9% of the population consume a
hamburger meal or, equivalently, that in any week 63%
of the population eat a hamburger meal. We entered this
as most people consume a hamburger weekly
(Question 4). It was difficult to determine the predicted
level of contamination during processing. We estimate
a contamination rate on carcass meat of < 1%. Cassin et
al. (1998b) discussed various factors that affect the
contamination rate during the processing of meat, and
concluded that overall, a reduction in the initial con-
tamination of between 10 and 300 could be expected.
We chose The process usually eliminates. . . at
Question 7. However, in their calculations, Cassin et
al. (1998b) predicted that 2.9% of the packages of retailground beef of size 3001000 g are contaminated. We
implemented this directly at Question 8 which over-
rides the contamination changes predicted from the
answers to Questions 6 and 7.
The geometric mean of the contamination levels
estimated by Cassin et al. (1998b) is f 20 CFU/
package. The average serving size is 83 g for adults.
Thus, based on the average package size, average
serving size and average contamination level, we
estimate the average CFU/meal serving as 3 at retail.
We have assumed that the USA has good temperature
control and handling systems for raw meat and have
selected Controlled for Question 9. By analogy
with shigellosis, the ID50 for O157:H7 was estimated
by Cassin et al. (1998b) as f 2000 CFU. Thus, we
assumed that a f 1000-fold increase in dose would
be required to cause infection in the average case.
Cassin et al. (1998b) cited the results of MacIntosh
et al. (1994) for hamburger cooking preference. Nine-
teen percent of consumers were reported to prefer
rare or medium rare cooked meat products. We
assume that all other degrees of cooking result in the
elimination of the pathogen, and that of the remaining20% preferring less thoroughly cooked meat, the
cooking eliminates 75% of the pathogens present.
We implement this as cooking eliminates 95% of the
pathogens present in all hamburgers consumed (Ques-
tion 11). The values used are summarised in Table 3.
The model predicts a per-meal risk of 6.2 10 7,and predicts 45,800 cases per year in the USA. Cassin
et al. (1998b) estimated the risk per meal to be 3.7 and
5.5 10 5 for children and adults, respectively, fromtheir model. On the assumption that half of the
10,000 20,000 cases annually of E. coli O157:H7illness in the USA are due to consumption of ham-
burgers, Cassin et al. (1998b) estimated the per-meal
risk at between 5.7 10 7 and 1.2 10 6. TheRisk Ranking estimate is 58.
4. Discussion
The spreadsheet tool was originally developed as a
means of quickly assessing the performance of various
Table 3
Values used in the assessment of risk from enterohaemorrhagic E.
coli in hamburgers in north America
Risk criteria Input
Dose and severity
1. Hazard severity Moderateoften requires
medical attention
2. Susceptibility Generalall population
Probability of exposure
3. Frequency of consumption weekly
4. Proportion consuming most
5. Size of population Other (270,000,000)
Probability of infective dose
6. Probability of contamination Other (1% of servings)
7. Effect of process Usually eliminates
the hazard
8. Possibility of recontamination Other (2.9%)
9. Post-process control Controlled
10. Increase to infective dose Other (1000)
11. Effect of treatment
before heating
Other (0.05)
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 395348
7/30/2019 Food Safety Risk Assessment Tool
11/15
conceptual models for food safety risk assessment. It
was quickly realised that the spreadsheet itself was a
valuable risk assessment and risk communication tool,
and the conceptual model and the spreadsheet userinterface then were developed in parallel. Refinement
of the conceptual model was based largely on exper-
imentation with the model. This involved trying to
recreate scenarios for which data to describe the model
inputs, and epidemiological data by which to evaluate
the corresponding model outputs, were available. For
example, early iterations of the model required a
number of correction factors so that numerical
predictions of cases of illness matched those reported.
Other risk assessment models have used such factors to
make the predicted number of cases better match the
observed rates of illness (Farber et al., 1996; FDA/
FSIS, 2001a). Experience with the model enabled the
refinement of the questions posed and data required so
that the correction factors were eliminated from the
model. Elimination of correction factors is impor-
tant because a major tenet of the risk assessment
approach is that all assessments should be trans-
parent, i.e. the basis of all calculation should be made
explicit (CAC, 1999).
The spreadsheet interface has also been improved
through feedback from a diverse range of users. We
emphasise, however, that while the tool is presentedas an example of how food safety risk assessments
can be simplified and its benefits made more acces-
sible to risk managers, the tool is not definitive. It can
still be improved, and cannot be expected to be
appropriate to all food safety risk assessment prob-
lems. We discuss some of the shortcomings and
tangential benefits of the model below.
4.1. Evaluation of performance
We compared the predictions of the model toindependently obtained epidemiological data and esti-
mates for food-borne illness in Australia and USA to
calibrate the Risk Ranking value. The estimates
obtained suggest that Risk Ranking in the range
6065 describes the status quo for all microbial food-
borne disease in Australia and USA. We consider
those to be representative of many developed nations.
This gives a reference point from which to evaluate
Risk Ranking values for other product/hazard/path-
way combinations. It should be noted that the Risk
Ranking is independent of population size, but
reflects the relative risk to an individual within the
selected population. Thus, the Risk Ranking can be
used potentially to compare the risks across diversefoods, hazards and cultures.
The USA data enabled the Risk Ranking to be
estimated from different disease end-points (e.g. esti-
mated total illness, estimated hospitalisations, esti-
mated deaths) and revealed that the Risk Ranking
value depended on the end-point chosen. Perhaps
surprisingly, then, the Comparative Risk estimated
from the USA fatality estimates was 10-fold lower
than the Comparative Risk estimates based on total
estimated cases or total cases requiring hospital treat-
ment. In the conceptual model underpinning the tool,
the weighting applied for disease severity (arbitrarily)
assumed death to be 1000 times more serious than
a mild case of illness not requiring medical attention.
It is clearly difficult to deduce an objective, quantita-
tive, measure to compare the severity of death to that
of mild food-borne illness. The Risk Ranking
values based on USA data but using different disease
end-points suggest, however, that the weighting fac-
tors for illness severity used in the model are inappro-
priate (see discussion further on).
The prediction of the model for a scenario based on
a food-borne disease outbreak in Australia was 225cases, within a factor of two of the observed number
of cases (f 440). The inputs to the model were as
consistent with the events surrounding the outbreak as
was possible given the available data. Where no data
was available, reasonable assumptions or estimates
based on analogous data or experience were used so
that, in the scenario modelled, there was little oppor-
tunity to manipulate inputs to achieve the specific
results.
Similarly, using as inputs data taken from Cassin et
al. (1998b) yielded results that were consistent withthe results of that stochastic risk assessment. The per-
meal risk predicted by the spreadsheet model was 1 in
16 million, within the range predicted by Cassin et al.
(1998b) of 1 in 17 million to 1 in 830,000 meals
consumed. Clearly, these two examples do not prove
that the model is reliable. In our experience, however,
the spreadsheet model predictions are usually within
an order of magnitude of independent estimates of the
number of cases of food-borne illness for specific
product/hazard/pathway combinations. This level of
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 49
7/30/2019 Food Safety Risk Assessment Tool
12/15
accuracy is expected on the basis of the models re-
liance on multiplication of a series of weighting fac-
tors, many of which are in 10-fold increments. Further
examples of the models performance and utility as arisk management aid are presented in Sumner and
Ross (this issue), and provide perspective on the rela-
tion of risk ranking values with recognised hazards.
4.2. Limitations/weaknesses
The creation of the model was largely a reactive
process, i.e., during testing against available epide-
miological data, when the model failed, the source of
the failure was analysed and the model modified to
overcome that deficiency. Despite the apparent utility
of the model, we have not been able to systematically
and objectively evaluate the models performance,
because there are few detailed data sets describing
exposure and food-borne disease incidence.
There are other limitations and weaknesses. Some
are general problems associated with risk assessment
modelling, while others are specific to the tool pre-
sented here.
Even though we have attempted to make the
questions unambiguous, the intent of the question
can still be misinterpreted. For example, Question 4
refers to the proportion of any population that con-sumes the product. It does not need to be adjusted by
the user when a sub-population is selected in Question
2, because the spreadsheet model automatically modi-
fies the size of the population exposed when a sub-
population is selected at Question 2. Similarly, the
answer to Question 10 is intended to be based on the
ID50 for a healthy member of the normal population,
irrespective of whether a susceptible population is
selected at Question 2. Again, as described in Meth-
ods and materials, the calculations in the spreadsheet
make adjustment for the selection in Question 2.In modelling any complex and variable system, it
is necessary to balance the need to make simplifying
assumptions against the loss of detail that ensues. In
general, the Australian and USA statistics infer a risk
of mild food-borne illness of one case per person
every 5 to 10 years, roughly equivalent to a risk of 1
in 500011,000 meals. While the risk of outbreaks is
much less, food safety managers are often more
interested in understanding the sets of specific circum-
stances that lead to these relatively rare events of
food-borne illness outbreaks. Using a small number of
descriptors of those conditions hinders discrimination
of small, but potentially critical differences, so that
important information can be lost in the averagingprocess that results.
Another problem associated with these low levels
of discrimination is that many choices automatically
lead to at least a factor of 10 difference in the estimated
risk. The option within some questions for the user to
enter a specific value other than those offered arose
from the realisation that the model could not make
accurate predictions, unless a wider range of values, or
narrower intervals between levels, were available.
Following from the above, it must be emphasised
that some of the weighting factors employed in the
model are arbitrarily derived. Other weights may be
more appropriate. For example, the weighting of
relative susceptibility to illness of consumers with
known predisposing conditions (Question 2) is cur-
rently based on the relative risk of listeriosis. While
those factors may be broadly appropriate to suscept-
ibility to infections, they may be irrelevant to the risk
of intoxications from microbial, or other toxins.
Earlier, we referred to the weights applied to the
disease severity descriptors. One way to make these
weights more objective is to express the severity of
diseases in terms of days of quality or life lost, a non-specific approach to measuring the health burden of
illness that is increasingly advocated in the domain of
public health (HCP, 2000). One such measure is
disability adjusted life years (DALY), which enables
the integration of different disease end-points.
Using this approach, the difference in weights
given to life-threatening food-borne disease compared
to mild gastrointestinal forms was suggested to be
too small (calculations not shown). As discussed
earlier, the Risk Ranking estimates based on differ-
ent disease end-points for the USA data similarlyraised the question whether the weights applied to
disease severity were appropriate. Weight factors
based on DALYs would also simplify the comparison
of illness from diverse sources, e.g. the acute effects
of food-borne infections compared to the chronic
effects of intoxications from chemical residues,
increasing the applicability and universality of the
proposed model. The weights and values used in the
spreadsheet for these, and other variables, can be
easily changed as necessary or appropriate. Care
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 395350
7/30/2019 Food Safety Risk Assessment Tool
13/15
should be exercised, however, that such changes do
not lead to unrealistic values in some of the inter-
mediate calculations in the model.
Stochastic approaches to risk modelling are pre-ferred because risk involves the element of probability
(Cassin et al., 1998a). A limitation of the tool is that
while it provides an estimate of the most probable
outcome, it does not provide information about the
level of confidence we have in that estimate, or more
importantly, the probable range of illnesses for different
scenarios. A possible refinement of the model would be
to allow users to enter a range of values, or distribution
of values that would offer some of the benefits of sto-
chastic modelling, but still in a relatively simple tool.
4.3. Peripheral benefits of the tool
Apart from its use for ranking perceived risks, the
spreadsheet tool helps to focus the attention of the users
on the interplay of factors that contribute to food-borne
disease. The model can be used easily to explore the
effect of different risk-reduction strategies, or the
extent of change required to bring about a desired
reduction in risk. Users must remember, however, that
some of the weighting factors are arbitrarily derived.
Consequently, the predicted effect may not reflect
reality but only the assumptions on which the modelis based, and users should ensure that the model is
appropriate to their risk assessment question.
Whether the mathematical model underlying the
tool is correct or not, we found the spreadsheet tool to
be a powerful aid for teaching the principles of risk
assessment. The model forces users to think about
factors affecting food safety and can help train food
safety managers to think in terms of risk, and the
interaction among factors that contribute to risk, rather
than in absolute terms such as zero tolerance of
hazards. Using the model to recreate scenarios quicklyreveals where data critical to estimating risk are lack-
ing, and so can be used to prioritise research needs.
5. Conclusion
The motivation for the development of the risk
assessment spreadsheet was to facilitate risk manage-
ment prioritisation. Its application, thus, is similar to
the Level 1 risk assessment proposed by van Gerwen
et al. (2000). The model is intended to be generic but
robust, and to include all elements that affect food
safety risks. We propose that the tool can be used by
risk managers and others without extensive experi-ence in risk modelling and as a simple and quick
means to develop a first estimate of relative risk. It can
also be used as a training and risk communication aid
to help determine data needs.
The tool is preliminary, and should be seen as a
prototype, not a definitive model. The tool also
requires that users understand the models limitations.
Despite those limitations, the model includes all
elements required to estimate the risk of illness from
foods. It can be modified to suit the specific question
of the risk assessor or risk manager, and we have
indicated possible developments and refinements to
improve the utility of the tool.
Tools such as these can help managers to think
about how risks arise and change and, thus, to help to
decide where interventions might be applied with
success. We consider the tool as a useful and con-
venient aid to help risk managers reach food safety
decisions more objectively. The spreadsheet can be
downloaded from: http://www.agsci.utas.edu.au/
downloads/ratool.zip.
Acknowledgements
The authors wish to acknowledge the helpful and
constructive comments of Dr. D. Jordan, New South
Wales Agriculture, Dr. D. Schaffner, Rutgers Univer-
sity; Dr. E. Todd of Michigan State University and Mr.
A. Fazil of Health Canada that led to improvements in
the model structure and interface. The spreadsheet tool
had its inception in food safety risk assessments
conducted for Australias Dairy Research and Develop-
ment Corporation, SafeFood NSW and Seafood
Services Australia. TR also thanks Dr. R. Chandlerand Mr. C. Chan for the impetus and encouragement
they provided to develop early prototypes of the tool.
We are also indebted to Meat and Livestock Australia
for ongoing support for microbial food safety research.
References
ABS (Australian Bureau of Statistics), 2000. Download (14 July
2000) from: http://www.abs.gov.au/Ausstats/.
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 51
7/30/2019 Food Safety Risk Assessment Tool
14/15
ANZFA, 1999. Food Safety Standards Costs and Benefits. AGPS,
Canberra.
Bernstein, P.L., 1996. Against the Gods: The Remarkable Story of
Risk. Wiley, New York, 383 pp.
Buchanan, R.L., 1997. National advisory committee on microbio-logical criteria for foods Principles of risk assessment for ill-
nesses caused by foodborne biological agents. J. Food Prot. 60,
14171419.
Buchanan, R.L., Damert, W.G., Whiting, R.C., van Schothorst, M.,
1997. Use of epidemiologic and food survey data to estimate a
purposefully conservative dose response relationship for Liste-
ria monocytogenes and incidence of listeriosis. J. Food Prot. 60,
918922.
CAC (Codex Alimentarius Commission), 1999. Proposed Draft
Principles and Guidelines for the Conduct of Microbiological
Risk Assessment (At step 5 of the Procedure). ALINORM, 99/
13, Appendix IV, pp. 5864.
Cassin, M.H., Paoli, G.M., Lammerding, A.M., 1998a. Simulation
modeling for microbial risk assessment. J. Food Prot. 61, 1560
1566.
Cassin, M.H., Lammerding, A.M., Todd, E.C.D., Ross, W., McColl,
R.S., 1998b. Quantitative risk assessment for Escherichia coli
O157:H7 in ground beef hamburgers. Int. J. Food Microbiol. 41,
2144.
CAST, 1994. Foodborne pathogens: risk and consequences. Council
for Agricultural Science and Technology, USA. Task Force Re-
port No. 122.
Conaty, S., Bird, P., Bell, G., Kraa, E., Grohmann, G., McAnulty, J.,
2000. Hepatitis A in New South Wales, Australia from con-
sumption of oysters: the first reported outbreak. Epidemiol. In-
fect. 124, 121 130.
Corlett, D.A., Pierson, M.D., 1992. Hazard analysis and assignmentof risk categories. In: Pierson, M.D., Corlett, D.A. (Eds.),
HACCP: Principles and Applications. Van Nostrand-Reinhold,
New York, pp. 29 38.
FAO (Food and Agriculture Organisation of the United Nations),
1995. Application of risk analysis to food standards issues. Re-
port of the Joint FAO/WHO Expert Consultation. Geneva, 13
17 March. WHO, Geneva.
Farber, J.M., Ross, W.H., Harwig, J., 1996. Health risk assessment
of Listeria monocytogenes in Canada. Int. J. Food Microbiol.
30, 145156.
FDA/FSIS, 2001a. Draft assessment of the relative risk to public
health from foodborne Listeria monocytogenes among selected
categories of ready-to-eat foods. Download from: http://www.
foodsafety.gov/~dms/lmrisk.html.FDA/FSIS, 2001b. Draft risk assessment on the public health im-
pact of Vibrio parahaemolyticus in raw molluscan shellfish.
Download (1 July 2001) from: http://www.foodsafety.gov/.
FSIS, 1998. Risk assessment for Salmonella enteritidis in eggs.
Food Safety Inspection Service. USA, WWW site: http://www.
fsis.usda.gov/ophs/risk/contents.htm.
HCP (European Commission, Health and Consumer Protection Di-
rectorate-General), 2000. First report on the harmonisation of
risk assessment procedures. The Report of the Scientific Steer-
ing Committees Working Group on Harmonisation of Risk
Assessment Procedures in the Scientific Committees Advising
the European Commission in the Area of Human and Environ-
mental Health, 26 27 October. Published on the internet 20/12/
2000. Download from http://europa.eu.int/comm/food/fs/sc/ssc/
out83_en.pdf.
Hitchins, A.D., 1996. Assessment of alimentary exposure to Listeriamonocytogenes. Int. J. Food Microbiol. 30, 7185.
Huss, H.H., Reilly, A., Ben Embarek, P.K., 2000. Prevention and
control of hazards in seafoods. Food Control 11, 149156.
ICMSF (International Commission for the Microbiological Specifi-
cations for Foods), 1996. In: Roberts, T.A. (Ed.), Micro-organ-
isms in Foods: 5. Microbiological Specifications of Food
Pathogens. Blackie Academic and Professional, London, 508
pp.
ICMSF (International Commission on Microbiological Specifica-
tions for Foods), 1998. Potential application of risk assessment
techniques to microbiological issues related to international
trade in food and food products. J. Food Prot. 61, 10751086.
ILSI (International Life Science Institute), North AmericaRisk
Science Institute Pathogen Risk Assessment Working Group,
1996. A conceptual framework to assess the risks of human
disease following exposure to pathogens. Risk Anal. 16, 841
848.
Jaykus, L.-A., Hemard, M., Sobsey, M.D., 1994. Human enteric
viruses. In: Hackey, C.R., Pierson, M.D. (Eds.), Environmental
Indicators and Shellfish Safety. Chapman & Hall, New York, pp.
92153.
Jaykus, L.-A., 1996. The application of quantitative risk assessment
to microbial food safety risks. Crit. Rev. Microbiol. 22, 279
293.
Jones, E.M., McCulloch, S.Y., Reevest, D.S., MacGowan, A.P.,
1994. A 10 year survey of the epidemiology and clinical aspects
of listeriosis in a provincial English city. J. Infect. 29, 91 103.Jurado, R.L., Farley, M.M., Pereira, E., Harvey, R.C., Schuchat, A.,
Wenger, J.D., Stephens, D.S., 1993. Increased risk of meningitis
and bacteremia due to Listeria monocytogenes in patients with
human immuno-deficiency virus infection. Clin. Infect. Dis. 17,
224227.
Kindred, T.P., 1996. Risk analysis and its application in FSIS. J.
Food Prot., 24 30, Supplement.
Lammerding, A.M., 1997. An overview of microbial food safety
risk assessment. J. Food Prot. 60, 14201425.
Lindqvist, R., Westoo, A., 2000. Quantitative risk assessment for
Listeria monocytogenes in smoked or gravad salmon and rain-
bow trout in Sweden. Int. J. Food Microbiol. 58, 181 196.
Macintosh, W.A., Christensen, L.B., Acuff, G.R., 1994. Perceptions
of risks of eating undercooked meat and willingness to changecooking practices. Appetite 22, 8396.
Mead, P.S., Slutsker, L., Dietz, V., McCaig, L.F., Bresee, J.S., Sha-
piro, C., Griffin, P.M., Tauxe, R.V., 1999. Food-related illness
and death in the United States. Emerging Infect. Dis. 5, 607
625.
Morgan, M.G., 1993. Risk analysis and management. Sci. Am.,
3241 July.
Nolla-Salas, J., Anto, J.M., Almela, M., Col, P., Gasser, I ., Plasen-
cia, A., the collaborative study group of listeriosis of Barcelona,
1993. Incidence of listeriosis in Barcelona, Spain, in 1990. Eur.
J. Clin. Microbiol. Infect. Dis. 12, 157161.
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 395352
7/30/2019 Food Safety Risk Assessment Tool
15/15
Notermans, S., Mead, G.C., 1996. Incorporation of elements of
quantitative risk analysis in the HACCP system. Int. J. Food
Microbiol. 30, 157173.
NRC (National Research Council), 1983. Risk Assessment in the
Federal Government: Managing the Process. National AcademyPress, Washington, DC.
NRC (National Research Council), 1994. Science and Judgement in
Risk Assessment. National Academy Press, Washington, DC.
NRC (National Research Council), 1996. Understanding Risk: In-
forming Decisions in a Democratic Society. National Academy
Press, Washington, DC.
PCCRARM (Presidential/Congressional Commission on Risk As-
sessment and Risk Management), 1997. Framework for Envi-
ronmental Health Risk Management. The Presidential/Congres-
sional Commission on Risk Assessment and Risk Management.
213 pp. Available online at http://www.riskworld.com.
Rose, J.B., Sobsey, M.D., 1993. Quantitative risk assessment for
viral contamination of shellfish and coastal waters. J. Food Prot.
56, 1043 1050.
Schlundt, J., 2000. Comparison of microbiological risk assessment
studies published. Int. J. Food Microbiol. 58, 197202.
Sumner, J.L., Ross, T., 2002. A semi-quantitative seafood safety
risk assessment. Int. J. Food Microbiol., this issue.
Todd, E.C.D., Harwig, J., 1996. Microbial risk analysis of food in
Canada. J. Food Prot., 1018, Supplement.
van Gerwen, S.J.C., te Giffel, M.C., vant Reit, K., Beumer, R.R.,
Zwietering, M.H., 2000. Stepwise quantitative risk assessment
as a tool for characterization of microbiological food safety. J.
Appl. Microbiol. 88, 938951.
Van Schothorst, M., 1997. Practical approaches to risk assessment.
J. Food Prot. 60, 14391443.
Vose, D., 1996. Quantitative risk analysis: a guide to Monte Carlo
simulation modelling. John Wiley and Sons, N.Y.
Voysey, P.A, Brown, M., 2000. Microbiological risk assessment: a
new approach to food safety control. Int. J. Food Microbiol. 58,
173180.Walls, I., Scott, V.N., 1996. Validation of predictive mathematical
models describing the growth of Escherichia coli 0157:H7 in
raw ground beef. J. Food Prot. 59, 13311335.
WHO/FAO (World Health Organisation/Food and Drug Organisa-
tion), 1999. Risk assessment of microbiological hazards in
foods. Report of a Joint FAO/WHO Expert Consultation. Gen-
eva, Switzerland. 15 19 March, 1999. World Health Organisa-
tion, Geneva, 24 pp.
WHO/FAO (World Health Organisation/Food and Drug Organisa-
tion), 2000a. Exposure assessment of Listeria monocytogenes in
ready-to-eat foods. Download (28 October 2001) from: http://
www.who.int/fsf/mbriskassess/index.htm.
WHO/FAO (World Health Organisation/Food and Drug Organisa-
tion), 2000b. Exposure assessment of Salmonella Enteritidis in
eggs. Download (28 October 2001) from: http://www.who.int/
fsf/mbriskassess/index.htm.
WHO/FAO (World Health Organisation/Food and Drug Organisa-
tion), 2000c. Exposure assessment of Salmonella spp. in broil-
ers. Download (28 October 2001) from: http://www.who.int/fsf/
mbriskassess/index.htm.
WHO/FAO (World Health Organisation/Food and Drug Organisa-
tion), 2001. Risk characterization of Salmonella spp. in eggs and
broiler chickens and Listeria monocytogenes in ready-to-eat
foods. Download (28 October 2001) from: http://www.who.int/
fsf/mbriskassess/index.htm.
T. Ross, J. Sumner / International Journal of Food Microbiology 77 (2002) 3953 53