Embed Size (px)
Citation preview
2
Improving Scenario Analysis for HRAStudy: 2012 - 2014
3
Study overview
ObjectiveTo investigate how scenario analysis is performed by
HRA analysts in real world settings, and why it is sometimes considered difficult
GoalTo develop a practical handbook for analysts,
documenting a recommended approach, good practices and guidance on how to overcome typical challenges
4
Motivation for the study
Background International HRA Empirical study & US Domestic
study both identified that variability in human error predictions between analyst teams were, in part, due to deficiencies in qualitative scenario analysis methods and differences in approach
Follow-up interviews indicated that lack of guidance may be a primary cause of the differences in how HRA teams performed qualitative scenario analysis
5
Key study activities
Interviews 2 rounds of interviews
Workshops 2 workshops
Initial interviews, 2013 Participants from the international HRA
empirical study or the US domestic study 9 people interviewed About half had minimal experience of HRA,
half perform HRA regularly
Detailed interviews, 2014 Additional participants from HRA community
in US & Europe 8 people interviewed Almost all perform HRA regularly as part of
their job
Halden workshop, May 2013 Attended by HRA practitioners, researchers
and regulators Presented a generic approach to HRA Agreed on what constitutes a qualitative
“scenario analysis”
EPRI Charlotte workshop, Oct 2014 Attended by 15 HRA practitioners from the US Presented the first draft practical handbook Collected additional data regarding
challenges, good practices & guidance needs
6
Main findings from the study
7
Similar approaches to HRA Most people who perform
HRA regularly adopt a similar approach to qualitative analysis…
… despite the fact that there is very little guidance available on how to perform qualitative analysis
… regardless of which HRA quantification method they use
Scenario definition
Qualitative data collection
Qualitative data analysis
Human error quantification
9
Focus of this study The Improving Scenario
Analysis for HRA study focused on the first two steps:• Scenario definition• Qualitative data collection
Qualitative data analysis is investigated in a separate study by the Halden Reactor Project from 2015 – 2017.
Scenario definition
Qualitative data collection
Qualitative data analysis
10
Use of the simulator
There is a difference of opinion amongst practitioners about the value/reality of collecting data from a training simulator
Cannot get a real understanding of how the scenario might unfold without seeing it in the simulator
However, observation of a training exercise in the simulator does not allow the analyst to see how the operator might fail
Simulator schedules are often fully booked and so the analyst might not be able to observe relevant scenarios within the timescales of the analysis
11
Lack of practical guidance
Existing HRA guidance does not tell the analyst how to collect the necessary information for scenario analysis
The guidance on how to do scenario analysis is improving – for example, NUREG-1921 (Fire PRA)
But this still does not the more practical aspects of how to collect qualitative data
12
Importance of operator input
Analysts must go to the site whenever possible to talk to operations personnel
This is essential to ensure that the HRA reflects the “as operated” plant, rather than “as built”
It is also essential to understand the contextual factors that will influence how the operators will response in that scenario
HRA cannot be a desktop exercise
13
What type of information should analysts collect for HRA?
What it is really like to perform tasks• How difficult or complex the tasks might be, why, and what
impact this has on task performance• What kind of limitations or constraints (physical,
organizational, cultural) might exist and how these influence task performance
How operators would know that something is wrong• Whether the scenario event would be obvious to them• Whether and how they would be able to diagnose the event• Whether and how they would know what to do
14
Types of information…
Which plant systems and interfaces operators would use and how they would use them• Whether they are easy to use and interpret• Whether they are reliable and trustworthy
The challenges that they could experience• Inadequate procedures• Lack of training• Poor quality of HMIs• The likelihood of incorrect mindset or group think
15
Types of information…
The presence and effect of other Performance Shaping Factors (PSFs)• Time pressure• Stress• Workload
Operators can also• Confirm assumptions made by the analyst• Provide information on areas of uncertainty
16
Challenges with collecting information
17
Key challenges
Availability of knowledgeable and experienced operators
Access to where the operators are located
Availability of simulator time to run analysis scenarios
Unsurprisingly, the most knowledgeable and experienced operators are often the busiest
Analysts may have to go to the plant in the evening or at weekends to secure interviews
For control-room based analyses, it can be difficult (or impossible) to gain access to the control room for interviews or observations
This is particularly the case for contractors
Equally, simulator training schedules are defined well in advance and are extremely busy
If the analyst is allowed to observe an exercise, it will probably not be the analysis scenario
18
Key challenges
Risk of obtaining incorrect or contradictory information
Reluctance to talk about potential human errors
Perceived credibility of the scenario being analyzed
Highly unlikely that operators will deliberately give incorrect information, but the might not know that they are wrong
Operators may contradict one another
Human error can be a sensitive subject Operators may be reluctant to admit they could
do something wrong Mindset of “that would never happen here”
If no previous experience of the analysis scenario, it might be difficult for operators to perceive that it is credible – e.g. beyond design basis events, Fukushima-type events, etc.
19
Challenges from the analyst
Risk of being biased about how operators work
The International HRA Empirical study identified that analysts have a tendency to consider that operators perform a single “initial diagnosis” at the start of the event only
Subsequent observations of operating crews in the simulator showed that operators continuously make decisions whilst working through procedures and executing tasks
Some of the analysts failed to identify important PSFs, resulting in underestimations of HEPs.
(It should be noted that the analysts did not have an opportunity to interview operators or simulator trainers in this study importance of getting operator input!)
Detect Diagnose Decide
Act
Event
Basic behavioral model
20
Challenges from the analyst
Risk of over-reliance on a single information source
??
??
??
In the US HRA Empirical study, the analysts were allowed to interview simulator trainers as part of the HRA, but there were differences in how the analysts used the information from the trainers in their analyses
Two teams appeared optimistic in their timing analysis because the trainer underestimated the time needed to respond to an HFE, and they did not validate this estimation with anyone else
A third team was also told by the trainer that the operators would succeed within the time, but they then asked the trainer to talk through the action in detail, and identified some hidden error traps that would prevent the crew from succeeding
The subsequent empirical results showed that the crew did not manage to perform the action in time
21
The handbook of good practices
22
Aim of the handbook
To provide universal, method-independent guidance that can be tailored to fit the specific context of the scenario being analyzed
Difficult to write a handbook to fit every situation and HRA method
Handbook cannot (and does not intend to) replace the role of HRA training and mentoring
Can be seen as a knowledge transfer exercise – this is the first step towards documenting what experienced analysts know about HRA – tips & tricks, etc.
23
Recommended approach
24
Practical guidance
25
Good practice - be prepared
Contact the site early in the HRA to secure time with knowledgeable and experienced operators• Be realistic about how much time you will need
with each person or group• Be flexible in case the person or group is not
available on the day
Do your homework before you go to the site to avoid wasting time with basic questions
26
Good practice – interview technique
Ask probing questions to ensure enough detailed information is collected for analysis later in the HRA
Use examples to stimulate discussion of human error and to reinforce the credibility of the analysis scenario / HFEs
Use non-accusatory language when talking about human error E.g. Rather than asking what the operator could do wrong themselves,
ask what could go wrong that would prevent the operator from doing their job
Don’t use PRA-speak – put the “human” back into HRA!
27
Good practice – talk to lots of people
Interview multiple operators and/or groups of operators (crews) to get consensus on the information received and to minimize bias
Don’t interview only operating personnel – supervisors, shift managers, simulator trainers, process experts, systems experts etc. also have valuable insights and information and often have an operations background
28
Good practice – follow up
Be prepared to do a follow up visit to collect additional information if needed, to check details of the analysis, to confirm assumptions, etc.• If it is not possible to do a follow-up visit, then be
prepared to follow up by telephone or email
Send a follow up email after your visit thanking the participants for their time – this will benefit you in the long run!
29
Improving HRA PracticesStudy: 2015 - 2017
30
Motivation HRA analysts also report difficulties when performing
qualitative data analysis, e.g.
How far to decompose tasks in a task analysis How to identify credible human errors / Human Failure Events How to link task analysis and human error identification to fault
or event tree analysis (and ultimately to the Probabilistic Safety/Reliability Analysis)
The 2015-2017 Improving HRA Practices topic continues investigating the HRA process, with a focus on qualitative data analysis
1. Problem definition
2. Qualitative data collection
3. Task analysis
4. Human error analysis
5. Representation
7. Quantification
8. Impact assessment
10. Quality assurance
11. Documentation
Is screening required?
HEP acceptable?
Factors influencing
performance and error causes or
mechanisms
9. Error reduction
6. ScreeningInsignificant errors not studied further
Yes
No
Yes
No
Improving performance
Error avoidance
Generic HRA process Improving Scenario Analysis project
(2012 – 2014)
Improving HRA practices project
(2015 – 2017)
HWR-1120
Objectives To identify the different qualitative data analysis techniques
that are used in HRA, and the difficulties or challenges that analysts experience when using these
To identify qualitative data analysis techniques that are used in simulator experiments (e.g. at HAMMLAB) and investigate whether these could be used for HRA
To further develop the practical handbook to include a recommended approach and guidance for HRA analysts performing qualitative data analysis
