Upload
claud-quinn
View
212
Download
0
Tags:
Embed Size (px)
Citation preview
Rapid Risk Assessment(and/or Rapid Risk Analysis)
Are they possible?
Villie Flari,
Food and Environment Research Agency, UK
[email protected] JIFSAN / Fera annual meeting, June 2011
Trigger for this talk…
• … part of my work when at CFSAN, FDA (2007 – 2009)– “Rapid Risk Assessment Framework” project: a tool to apply with CFSAN
emergency response procedures– Ambitious and quite challenging project: aimed to identify whether there is
room for improving the existent strategies, employed at the time by CFSAN/FDA during emergency events, and if yes to suggest and justify alternative approaches
• Review FDA/CFSAN processes to assess public health threats during emergency events.• Identify areas of opportunities for implementing risk assessment thought approaches
with the view to improving the current FDA/CFSAN process.• Prepare a framework document to describe the recommended process.• Develop templates for assessing information, characterizing unknown factors and for
communicating assessment results effectively to risk managers and/or risk communicators.
Project orphan..
Are any changes to current schemes and approaches truly needed?
• What is the outcome we want to estimate – which is our aim?– Preventive approaches and emergency responses for ensuring food safety
and protecting human health
• Overall, not that bad
• Apart form regulatory processes a number of initiatives and support systems nationally and internationally developed over the last 30 years - currently in place (e.g. RASFF)
2008 2008-9 2009 2010 2011
Source: Food Safety Risk Analysis: a guide for national food safety authorities, (FAO Food and Nutrition Paper 87, 2006)
Food safety challenges - constantly change…
Emerging sciences & technologies
Dynamic
Emerging risks – emerging sciences / technologies
Nano-sized materials
Synthetic Biology
Genetically Modified Organisms
?
?
??Unknown,
thus far, emerging risk(s)Tactical and strategic
problems
Designing rapid risk assessments – are they possible?• What is the outcome we want to estimate – which
is our aim?– What do we mean precisely when we term a
document as a “risk assessment” or a “rapid risk assessment”?
– How do we define the term “risk”?• Safety assessment?• Risk assessment?• A collection of (background + epi) information?
Possible scenarios
Hazard
Chemical Biological
Known Scenario 1 Scenario 3 Scenario 5
Unknown Scenario 2 Scenario 4 Scenario 6
Identifiedvehicle (s)
Non identified vehicles (s)
Hazard
Chemical Biological
Known Melamine in infant formula – 2008 (previous relevant case: Melamine in pet food – 2007)
Salmonella in peanut butter – 2009 (previous relevant case: Salmonella in Peter Pan)
Salmonella saintpaul in fresh produce – 2008/2009
Unknown Melamine in pet food - 2007
Scenario 4 Scenario 6
Identifiedvehicle (s)
Non identified vehicles (s)
Looking into historical events & following events “in vivo”
Historical events
– “Melamine in pet food” - March to May 2007
Following events “in vivo”
– “Salmonella saintpaul in fresh produce” – May to August 2008
– “Salmonella typhimurium in peanut butter” - November 2008 to January 2009
What did we learn?
Incomplete knowledge
Limited time
Many dimensions
FIRMS’PRACTICES
HistoricalData (Firm)
Kill steps
Compliance
EPIDEMIOLOGY
Number of Ill Illness onsetrange
Most illnesses> 10/01
Deaths relatedto outbreak
IllnessesOutside USA
Raw dataCase controlStudy resultsEpi curve
Source Vehicle (s)
Complaints
Questionnaires
Packing
ProcessingPlants
SOP
Production
CLINICALDATA
INVESTIGATIVE LABORATORY FINDINGS
INVESTIGATIONFINDINGS
PRODUCTSMANUFACTURED
Shelf lifeof product
Trace back1st level
Trace back2nd level
Best before dateLOT
Internationalsamples
FIRMdata
FDAdata
STATEData
FACTS
OUTBREAK INVESTIGATION
DISTRIBUTIONTRACEFORWARD
DatabasesInspection reports
BACKGROUND INFORMATION
Trace back3rd level
Distributors
TRACEFORWARD
DomesticInternational
Recall
FOOD SUPPLY CHAIN
Market share(specific to firm (s)
General infoSpecific to product
Distributors
Firm’s productTrace back
RISK ASSESSMENT
Lines of investigation of emergency events: relevant information needs R
efDo
c 11R
efDo
c 11B
ackgro
un
d / ratio
nale
Problem i.e. Risk Management question (s)See RefDoc1 and 2 for extended information
Is Risk Assessment required by RM?
Define all lines of investigationSee RefDoc11 for extended information
Evaluate dataAccept, Reject, Withhold data
See RefDoc 14 for extended information
Form hypothesisSee RefDoc 14 for extended information
Risk Managementdecision
Do not accept
hypothesis
Repeat for each line of investigation
Monitor & Organize DataSee RefDoc12 for extended information
Hypothesisaccepted
Data considered enough foraccepting hypothesis?
See RefDoc 14for extended information
Need to redefineproblem?
Risk Managementoptions
Yes
No
Yes
No
Consider all lines of investigation
NoYes
Unc
erta
inty
ana
lysi
sD
ecis
ion
mak
ing
proc
ess
See
RefD
oc 3
& 1
6 fo
r ext
ende
d in
form
ation
YesNo
Problem classifiedAs Class I Recall?
Assess problem against criteria for Class I Recall1
See RefDoc 2 for extended information
Define RA endpoint (s)See RefDoc 8 for extended information
Continue investigation
Introduce RA as line ofinvestigation
See RefDoc 8 for extended information
Risk
Com
mun
icati
on
Test hypothesisSee RefDoc 14 for extended information
EPIDEMIOLOGY
INVESTIGATIVELABORATORY
FINDINGS
INVESTIGATIONFINDINGS
FIRMS’PRACTICES
TRACEFORWARD
FOOD SUPPLY CHAIN
RISK ASSESSMENT
Risk Assessment approach can be employed to ask information about the
denominator of epidemiological and investigative laboratory results: e.g.
“During the “Salmonella typhimurium in the peanut butter” case how many
samples have they been put under test?”
Usually, a lot of attention is paid on the positive microbiological samples,
but not on the total of samples tested. The denominator could give us
information on the actual risk posed by the particular microbiological
hazard.
Risk Assessment would allow one to be able to answer
questions regarding “how risky it is to consume peanut
butter products”, e.g. “if I eat 200 crackers from 200
different packs which was the probability to have been
contaminated by Salmonella typhimurium?”
Risk Assessment would allow to address
questions regarding the effect of risk
factors if certain mitigation methods
would be followed; for example “If the
temperature during a kill step is always
above the threshold of 350 F what would
be the decrease in probable pathogen
concentration in a cookie?”Usually a presence/absence of the
microbiological hazard is given. Colony
Forming Units (CFUs) information is not given.
As a result, Dose Response information is
missing. Requesting CFUs could help risk
assessments in the future as well.
Risk Assessment’s value:
Bringing all the information together – connecting all these lines of
investigation in order to reveal patterns, hypotheses.
Gather information, direct research for the future.
RefD
oc 11
RefD
oc 11
Backg
rou
nd
/ ration
aleB
ackgro
un
d / ratio
nale
Is this what we need?
• Partially yes, but ….
Risk assessments / analyses in emergency responses: why, how?
The STEC outbreak reported from Germany is noteworthy considering its magnitude: 276 HUS cases and two deaths due to HUS reported in just a few weeks. The majority of cases reported are adult women. Usually, about 15% of children with STEC infection present with HUS, with this proportion being much lower among adults. This means that several hundred STEC cases with diarrhoea are likely to be occurring in the current outbreak.
The isolated outbreak strain STEC O104:H4 is very rare. Prior to the current outbreak, only one case has been documented in literature, and this case was a woman in Korea in 2005.The case-control study in Hamburg revealed that raw tomatoes, fresh cucumber and leafy salad are the likely vehicles of infection. Samples of fresh cucumbers taken in Hamburg tested positive for STEC, however, the exact time and place of contamination remains unclear.
It is unclear whether the results from Hamburg can be extrapolated to the whole of Germany. Furthermore, it cannot be excluded that an alternative food item is the vehicle of infection. The definite source of the infection remains to be confirmed. There is currently no indication that either the source of the outbreak or the vehicle of infection has been distributed outside of Germany.
Likely 47% sure?71% sure?
83% sure?
The question then becomes…• Whether approaches and thought processes we
develop for and apply in innovative Risk Assessments (or Analyses…), can help us overcome difficulties such as those we encountered recently in the food safety field.– Chemical: Can we identify the true hazard quickly?– Microbiological: Can we identify the true vehicle(s)/source(s) of an
outbreak quickly?– Can we foresee possible future challenges relevant to new sciences
and technologies that are being developed?– Can we produce reproducible, transparent, fully defensible
assessments / analyses that will withstand media pressure, public criticism?
What did we learn?
The work on this project highlighted the significance a number of aspects that appear to be inherent in an emergency investigations:•Unknown factors, and in particular the value of embracing uncertainty - embracing uncertainties by recognizing them, classifying them, characterizing them can only be beneficial for all involved in the investigation, the scientists, the decision makers and the public.•The art of eliciting expert judgment: subjective information plays a major part in the science on which rapid decisions are made. How do we ensure that we obtain the most informative points of view, and how do we ensure that we capture experts’ uncertainty? How do we combine different opinions, if this is deemed necessary?•Following a problem – solving approach: this type of work may be quite different from simply following a process. Asking “what is the problem, and how the problem could be defined?” is not merely a philosophical question; instead, it is the most important question to be settled in the start of an investigation.
Incomplete knowledge
Limited time
Uncertainty in risk assessments, risk analyses in emergency events: why, how?• What is the desired outcome – which is our aim?• Rapid Risk Assessment
– Risk is the uncertainty of the outcome• Outcome(s) Risk Assessment endpoint(s)
– Which are the chances of my child dying if s/he consumes x amount of contaminated food?
– Which is the proportion of Denmark population who may suffer from severe adverse affects (i.e. >10 days of hospitalisation) if they consume x amount of contaminated food?
• Uncertainty of outcome(s)– Probability distributions - Quantitative (Hard data and/or
expert judgment) – + Qualitative (acquired via expert judgment) information
ANDYHART
What did we learn?
The work on this project highlighted the significance a number of aspects that appear to be inherent in an emergency investigations:•Unknown factors, and in particular the value of embracing uncertainty - embracing uncertainties by recognizing them, classifying them, characterizing them can only be beneficial for all involved in the investigation, the scientists, the decision makers and the public.•The art of eliciting expert judgment: subjective information plays a major part in the science on which rapid decisions are made. How do we ensure that we obtain the most informative points of view, and how do we ensure that we capture experts’ uncertainty? How do we combine different opinions, if this is deemed necessary?•Following a problem – solving approach: this type of work may be quite different from simply following a process. Asking “what is the problem, and how the problem could be defined?” is not merely a philosophical question; instead, it is the most important question to be settled in the start of an investigation.
Incomplete knowledge
Limited time
Expert judgment in risk assessments in emergency responses: why, how?
• Who is to be involved?– Best expertise possible
• Lists of experts available• Calibration of experts only in retrospect (time-consuming)
– Fit for purpose method• Qualitative information?• Quantitative information?• Both?
Uncertainty of experts to be captured Trained risk analysts Table top exercises
ROGERCOOKE
What did we learn?
The work on this project highlighted the significance a number of aspects that appear to be inherent in an emergency investigations:•Unknown factors, and in particular the value of embracing uncertainty - embracing uncertainties by recognizing them, classifying them, characterizing them can only be beneficial for all involved in the investigation, the scientists, the decision makers and the public.•The art of eliciting expert judgment: subjective information plays a major part in the science on which rapid decisions are made. How do we ensure that we obtain the most informative points of view, and how do we ensure that we capture experts’ uncertainty? How do we combine different opinions, if this is deemed necessary?•Following a problem – solving approach: this type of work may be quite different from simply following a process. Asking “what is the problem, and how the problem could be defined?” is not merely a philosophical question; instead, it is the most important question to be settled in the start of an investigation.
Incomplete knowledge
Limited time
Problem i.e. Risk Management question (s)See RefDoc1 and 2 for extended information
Is Risk Assessment required by RM?
Define all lines of investigationSee RefDoc11 for extended information
Evaluate dataAccept, Reject, Withhold data
See RefDoc 14 for extended information
Form hypothesisSee RefDoc 14 for extended information
Risk Managementdecision
Do not accept
hypothesis
Repeat for each line of investigation
Monitor & Organize DataSee RefDoc12 for extended information
Hypothesisaccepted
Data considered enough foraccepting hypothesis?
See RefDoc 14for extended information
Need to redefineproblem?
Risk Managementoptions
Yes
No
Yes
No
Consider all lines of investigation
NoYes
Unc
erta
inty
ana
lysi
sD
ecis
ion
mak
ing
proc
ess
See
RefD
oc 3
& 1
6 fo
r ext
ende
d in
form
ation
YesNo
Problem classifiedAs Class I Recall?
Assess problem against criteria for Class I Recall1
See RefDoc 2 for extended information
Define RA endpoint (s)See RefDoc 8 for extended information
Continue investigation
Introduce RA as line ofinvestigation
See RefDoc 8 for extended information
Risk
Com
mun
icati
on
Test hypothesisSee RefDoc 14 for extended information
RefD
oc 1
RefD
oc 1
Investigators, AnalystsDecision Makers,Risk Managers
1: Report trigger (s)
2: Phrase question (s) of investigation
Questions consideredunambiguous by
investigators, analysts?
Assessment of problem, i.e. risk management question (s),
begins
Yes
No
Tem
plate flo
w ch
artT
emp
late flow
chart
Backg
rou
nd
/ ration
aleB
ackgro
un
d / ratio
nale
Defining the problem - question
It is best to define the problem (i.e. risk management question (s)) precisely to ensure
that answers provided fit the purpose.
“To identify the food vehicle responsible for causing an outbreak of
Salmonella typhimurium infections with the following PulseNet
strains1:JPXX01.1818, JPXX01.1825, JPXX01.0459”.
Problem (i.e. risk management question (s)) is dynamic; thus, it may need to be
redefined in subsequent phases of the investigation.
“To a) identify distribution patterns of Peanut Corporation of America
products, b) elicit expert opinion on manufacturing practices and c) review
epidemiological data associated with Salmonella typhimurium, particularly
strains JPXX01.1818, JPXX01.1825, JPXX01.0459 in collaboration with CDC
with the view to prioritizing products according to the risk they may impose
to consumers”.
Advantages:
Diminishing ambiguity in provided answers.
Allowing investigators, analysts who participate in the emergency case to get
familiar with important information of the investigation irrespective of their role
and/or expertise and have the opportunity to look at the big picture of the
investigation.
RefD
oc 1
RefD
oc 1
Tem
plate flo
w ch
artT
emp
late flow
chart
Backg
rou
nd
/ ration
aleB
ackgro
un
d / ratio
nale
What did we learn?• Following a problem – solving approach: this type of work may
be quite different from simply following a process. Asking “what is the problem, and how the problem could be defined?” is not merely a philosophical question; instead, it is the most important question to be settled in the start of an investigation
• “Identifying, defining and then respecting the natural shape of the problem does not come naturally to agencies, since these are based on an organizational structure that was formed years ago and most of the approaches they follow are process-based” – Sparrow, 2008
• Perhaps, such an approach becomes even more challenged during an investigation of an emergency event as the organization is under extra pressure to deliver results and facilitate relevant decision making.
People
Perceptions Biases
Following a problem-solving approach
“Solving a problem simply means representing it so as to make the solution transparent”
Herbert A. Simon: The sciences of the artificial, 1996
Pathology findings:Acute/subacute kidney
failurein cats and dogs
03/15/2007
Chemistry findings:Melamine present in
contaminated wheat gluten03/25/2007
Hypothesis:More than one compounds
responsiblein association with other
agents for illnesses and deaths
Toxicology findings:Melamine not toxic
for mammals, in particular forthe recorded adverse effects
03/27/2007
Hypothesis:Melamine responsible
for illnesses and deaths
Chemistry findings:Further compounds identifiedin contaminated wheat gluten
ureidomelamine, urea,ammeline,
ammelide, cyanuric acid03/30/2007
Hypothesis:Other compound(s)
responsiblefor illnesses and deaths
Toxicology findings:There may be some additional
linksbetween melamine and uric
acid04/03/2007
Toxicology findings:No compound, of those identified
in contaminated wheat glutenis considered as toxic to mammals
04/04/2007
Melamine
Ammeline
Urea
Ammelide
Cyanuric Acid
Ureidomelamine
Pathology findings:Oral toxicology experiments indicate
formation of crystals due tocombination of chemicals, e.g.
melamine+ammeline+ammelide+cyanuric acid04/24/2007
Melamine Ammeline
Ammelide
Cyanuric Acid
Building a conceptual model of the problem: Building a conceptual model of the problem: example based on “Melamine in example based on “Melamine in pet food” emergency eventpet food” emergency event
RefD
oc 15
RefD
oc 15
Exam
ple
Exam
ple
ECRT decides that:HypothesisIs accepted
Intervention A
InterventionB
InterventionC
ECRT COLLABORATESWITH EXPERTS TO DEFINE
OPTIONS OFINTERVENTION
What if scenario
…
ECRT COLLABORATESWITH EXPERTS,
ANALYSTS, AND DECISION MAKERS
TO APPLY DECISIONMAKING TOOLS
What if scenario
What if scenarioWhat if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
What if scenario
ECRT COLLABORATESWITH EXPERTS TO DISPLAY
SENSITIVITY ANALYSIS,GATHER INFORMATION
ECRT COLLABORATESWITH EXPERTS AND
RISK MANAGERSTO DEFINE
STAKEHOLDERS,CRITERIA
Aggregation of criteria in a single function in order to be able to “compare” and prioritize between alternative scenarios. This could be performed via a number of Decision Making tools.
Stakeholders – Criteria - Units of measure for criteria Stakeholders – Criteria - Units of measure for criteria
RefD
oc 16
RefD
oc 16
Defau
lt meth
od
to h
and
le info
rmatio
nD
efault m
etho
d to
han
dle in
form
ation
Decision makingmodels
Challenges in ameliorating risk assessments / risk analyses: Strategic versus tactical• Similarities
– Encompass uncertainties– Define problem in question precisely– Employ the best experts– Provide support tools for transparent, reproducible, fully
defensible assessments
• Further challenges when dealing with emergency events– All of the above become even more crucial when dealing
with emergency investigations!
Thank you!
• CFSAN / FDA colleagues– Sherri Dennis– Elisa Elliott– Jack Guzewich– David Hattan– Faye Fenstein– Karl Klonz– Sherri McGarry
QUESTIONS??
[email protected] JIFSAN / Fera annual meeting, June 2011