U.S. Department of Transportation Federal Railroad Administration

Human Reliability Analysis in Support of Risk Assessment for Positive Train Control

Office of Research and Development Washington, DC 20590

U.S. Department of Transportation Research and Special Programs Administration John A. Volpe National Transportation Systems Center Cambridge, MA 02142

Human Factors in Railroad Operations

DOT/FRA/ORD-03/15 Final Report June 2003

This document is available to the public through the National Technical Information Service, Springfield, VA 22161 This document is also available on the FRA website at www.fra.dot.gov

Notice

This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for its contents or use thereof.

Notice

The United States Government does not endorse products or manufacturers. Trade or manufacturers names appear herein solely because they are considered essential to the objective of this report.

i

REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503. 1. AGENCY USE ONLY (Leave blank)

2. REPORT DATE June 2003

3. REPORT TYPE AND DATES COVERED Final Report

March 2001-December 2001 4. TITLE AND SUBTITLE Human Reliability Analysis in Support of Risk Assessment for Positive Train Control

6. AUTHOR(S) John Wreathall, Emilie Roth, Dennis Bley, and Jordan Multer

5. FUNDING NUMBERS

R3103/RR304

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) U.S. Department of Transportation Research and Special Programs Administration John A. Volpe National Transportation Systems Center Cambridge, MA 02142-1093

8. PERFORMING ORGANIZATION REPORT NUMBER

DOT-VNTSC-FRA-03-03

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) U.S. Department of Transportation Federal Railroad Administration Office of Research and Development 1120 Vermont Avenue, NW Mail Stop 20 Washington, DC. 20590

10. SPONSORING/MONITORING AGENCY REPORT NUMBER

DOT/FRA/ORD-03/15

11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION/AVAILABILITY STATEMENT

This document is available to the public through the National Technical Information Service, Springfield, Virginia 22161. This document is also available on the FRA web site at www.fra.dot.gov.

12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words) This report describes an approach to evaluating the reliability of human actions that are modeled in a probabilistic risk assessment (PRA) of train control operations. This approach to human reliability analysis (HRA) has been applied in the case of a safety evaluation of the Communications-Based Train Management (CBTM) System being tested by CSXT Transportation, Inc. (CSXT). This report describes the overall approach to the HRA and its trial application to the CBTM evaluation.

15. NUMBER OF PAGES 140

14. SUBJECT TERMS cognitive task analysis, communications, decision-making, human factors, human reliability analysis (HRA), positive train control (PTC), probabilistic risk assessment (PRA), railroad operations 16. PRICE CODE

17. SECURITY CLASSIFICATION OF REPORT Unclassified

18. SECURITY CLASSIFICATION OF THIS PAGE Unclassified

19. SECURITY CLASSIFICATION OF ABSTRACT Unclassified

20. LIMITATION OF ABSTRACT

NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) Prescribed by ANSI Std. 239-18298-102

METRIC/ENGLISH CONVERSION FACTORS

ENGLISH TO METRIC METRIC TO ENGLISHLENGTH (APPROXIMATE) LENGTH (APPROXIMATE)

1 inch (in) = 2.5 centimeters (cm) 1 millimeter (mm) = 0.04 inch (in) 1 foot (ft) = 30 centimeters (cm) 1 centimeter (cm) = 0.4 inch (in)

1 yard (yd) = 0.9 meter (m) 1 meter (m) = 3.3 feet (ft) 1 mile (mi) = 1.6 kilometers (km) 1 meter (m) = 1.1 yards (yd)

1 kilometer (km) = 0.6 mile (mi)

AREA (APPROXIMATE) AREA (APPROXIMATE) 1 square inch (sq in, in2) = 6.5 square centimeters

(cm2) 1 square centimeter (cm2) = 0.16 square inch (sq in, in2)

1 square foot (sq ft, ft2) = 0.09 square meter (m2) 1 square meter (m2) = 1.2 square yards (sq yd, yd2)

1 square yard (sq yd, yd2) = 0.8 square meter (m2) 1 square kilometer (km2) = 0.4 square mile (sq mi, mi2) 1 square mile (sq mi, mi2) = 2.6 square kilometers

(km2) 10,000 square meters (m2) = 1 hectare (ha) = 2.5 acres

1 acre = 0.4 hectare (he) = 4,000 square meters (m2)

MASS - WEIGHT (APPROXIMATE) MASS - WEIGHT (APPROXIMATE) 1 ounce (oz) = 28 grams (gm) 1 gram (gm) = 0.036 ounce (oz) 1 pound (lb) = 0.45 kilogram (kg) 1 kilogram (kg) = 2.2 pounds (lb)

1 short ton = 2,000 pounds (lb)

= 0.9 tonne (t) 1 tonne (t) ==

1,000 kilograms (kg) 1.1 short tons

VOLUME (APPROXIMATE) VOLUME (APPROXIMATE) 1 teaspoon (tsp) = 5 milliliters (ml) 1 milliliter (ml) = 0.03 fluid ounce (fl oz)

1 tablespoon (tbsp) = 15 milliliters (ml) 1 liter (l) = 2.1 pints (pt) 1 fluid ounce (fl oz) = 30 milliliters (ml) 1 liter (l) = 1.06 quarts (qt)

1 cup (c) = 0.24 liter (l) 1 liter (l) = 0.26 gallon (gal) 1 pint (pt) = 0.47 liter (l)

1 quart (qt) = 0.96 liter (l) 1 gallon (gal) = 3.8 liters (l)

1 cubic foot (cu ft, ft3) = 0.03 cubic meter (m3) 1 cubic meter (m3) = 36 cubic feet (cu ft, ft3) 1 cubic yard (cu yd, yd3) = 0.76 cubic meter (m3) 1 cubic meter (m3) = 1.3 cubic yards (cu yd, yd3)

TEMPERATURE (EXACT) TEMPERATURE (EXACT) [(x-32)(5/9)] F = y C [(9/5) y + 32] C = x F

QUICK INCH - CENTIMETER LENGTH CONVERSION10 2 3 4

InchesCentimeters 0 1 3 4 52 6 1110987 112

5

3 QUICK FAHRENHEIT - CELSIUS TEMPERATURE CONVERSION -40 -22 -4 14 32 50 68 86 104 122 140 158 176 194 212

F

C -40 -30 -20 -10 0 10 20 30 40 50 60 70 80 90 100

For more exact and or other conversion factors, see NIST Miscellaneous Publication 286, Units of Weights and Measures. Price $2.50 SD Catalog No. C13 10286 Updated 6/17/98

ii

ACKNOWLEDGMENTS This report describes an approach to evaluating the reliability of human actions that are modeled in a probabilistic risk assessment (PRA) of train control operations. It describes an approach to human reliability analysis (HRA) and its application to a test case called Communications Based Train Management (CBTM). The Federal Railroad Administrations Office of Research and Development funded this research effort.

Many people and organizations made the completion of this task possible. We would like to thank the members of the RSAC PTC task force for the guidance and support they provided us throughout the HRA quantification project. We would particularly like to thank Ms Denise Lyle of CSXT for all her support. Denise facilitated our visits to the yard in Spartanburg, South Carolina to conduct interviews and observations of CSXT Locomotive engineers and conductors, and the CSXT Dispatch Center in Jacksonville, Florida, to interview and observe CSXT railroad dispatchers. She also responded kindly and promptly to our many requests for information. We would also like to thank Mr. Tim DePaepe, of the Brotherhood of Railway Signalmen (BRS), Mr. Bob Harvey of the Brotherhood of Locomotive engineers (BLE) and Dr. Fred Gamst of the University of Massachusetts for their thoughtful inputs and their careful review and comment on early drafts of the results of the two site visits and quantification materials.

We would also like to acknowledge the support we obtained from the local CSXT management and labor representatives in Spartanburg, South Carolina and Jacksonville, Florida. In particular we would like to acknowledge the support of Mr. Franky Allan, CSXT train master in Spartanburg, South Carolina and Mr. Terry Davis, local BLE representative in Augusta, Georgia, who facilitated our site visit and helped to recruit CSXT locomotive engineers and conductors to participate in the focus groups in Spartanburg, South Carolina. We would also like to acknowledge the support of Mr. Crawford Boggs, local labor representative for the dispatchers in Jacksonville, Florida, who helped to recruit CSXT dispatchers to interview and observe. We also owe a debt of gratitude to Mr. C. Brock Lucas and Mr. John Campbell of CSXT facility in Jacksonville, Florida for sharing their facility with us and enabling us to observe dispatcher operations over several days and talk with their training staff. Most importantly, we would like to thank all the individuals who participated in the interviews and focus groups held in Spartanburg, South Carolina and Jacksonville, Florida. They provided invaluable knowledge about local operations and CBTM.

We are also grateful for all the support we received in organizing the Human Factors Quantification Workshop. We would particularly like to thank the labor representatives on the PTC RSAC task force for their active support in identifying and recruiting local and national labor representatives to participate in the workshop. Their active participation was critical to the conduct of the workshop.

Most importantly we would like to thank all the individuals who participated in the two-day Human Factors Quantification workshop in Greenville, South Carolina: Crawford Boggs, Sherry Borener, Calvin Cummins, Terry Davis, Tim DePaepe, Manuel Galdo, Fred Gamst, Louise Gunderson, Bob Harvey, Jerry Hicks, Tim Hudson, Rick Inclima, Ron Lindsay, Denise Lyle, Robert McCown, Richard McCord, Ashleigh Merritt, Wayne Minger, Howard Moody, Bob Ralph, Jack Ramsey, Thomas Raslear, James Stem, Tom Sullivan, William Thompson, and Terry Tse.

iii

We would like to thank Dr. Ted Giras, Dr. Lori Kaufman, and Mr. Marc Monfalcone for their help in explaining how the ASCAP model works and indicating how they could incorporate the results of our human reliability analysis.

We are also indebted to Sherry Borener, of the Volpe Center, for helping us understand the contents and limitations of the FRA databases.

We would also like to offer Wayne Minger special thanks for the advice he offered for moving the study forward in a positive and productive way. He helped translate the language of engineering into human factors terms.

Finally, we would like to thank Dr. Thomas Raslear, Mr. Robert McCown, Mr. William Goodman, and Mr. Grady Cothen of the FRA for their support in this effort.

iv

CONTENTS Section Page

Executive Summary....................................................................................................................ix

1. Introduction..........................................................................................................................1

1.1 Use of Risk Assessment for FRA ........................................................................................1

1.2 Role of HRA ........................................................................................................................2

1.3 Purpose of Report ................................................................................................................3

2. Approach to Estimation of Human Reliability in Train Control System Studies .........5

2.1 Overall Approach.................................................................................................................5

2.2 Relationship with Risk Assessment Activities ....................................................................7

2.3 HRA Process........................................................................................................................8

2.3.1 Identify the Human Failure Events and Unsafe Actions to be Estimated .................9

2.3.2 Perform Qualitative Evaluation of Human Factors Issues.........................................9

2.3.3 Identify Sources of Relevant Data ...........................................................................14

2.3.4 Identify Limitations and Gaps in Data Sources .......................................................15

2.3.5 Elicit Expert Opinion to Adjust for the Gaps and Limitations in the Databases .....16

2.3.6 Synthesize and Document Results ...........................................................................18

2.3.7 Review .....................................................................................................................18

3. Example Analysis for CBTM Study.................................................................................19

3.1 What is Communication Based Train Management (CBTM)? .........................................19

3.2 Human Failure Events to be Estimated..............................................................................20

3.3 Qualitative Human Factors Analysis .................................................................................20

3.3.1 Interviews and observations of CSXT Locomotive Engineers and Conductors .....22

3.3.2 Interviews and observations of dispatchers .............................................................25

3.4 Quantitative Analysis.........................................................................................................26

3.4.1 Overall Analytical Process.......................................................................................26

3.4.2 Sources of Data ........................................................................................................28

3.4.3 Limitations and Gaps in the Databases....................................................................33

3.4.4 Expert Elicitation Process: Quantification workshop..............................................34

3.5 Results of CBTM Analyses ...............................................................................................36

3.5.1 Worked Example .....................................................................................................36

3.5.2 Summary of other CBTM related Analyses.............................................................43

v

4. Lessons Learned.................................................................................................................51

4.1 Establish Appropriate Level of Decomposition for Modeling Human Reliability............51

4.2 Insure Broad Participation by Stakeholders ......................................................................53

4.3 Select Data Sources and Their Uses ..................................................................................54

4.4 Use Qualitative Information to Guide Quantitative Analysis............................................56

4.5 Project Impact of New Technology Given Limited Experience........................................57

4.6 Expertise Needed for a Human Factors Quantification Team...........................................60

4.7 Integrate Results into PRA ................................................................................................62

5. Recommendations & Conclusions ....................................................................................65

5.1 Findings from CBTM Analysis .........................................................................................65

5.1.1 General Findings......................................................................................................65

5.1.2 Potential Limitations and Concerns.........................................................................65

5.2 Recommendations for Future HRA Studies ......................................................................66

5.3 Conclusions........................................................................................................................67

Appendix A. Summary of Locomotive Engineer and Conductor Interviews.................69

Appendix B. Summary of Dispatcher Interviews..............................................................87

Appendix C. Details of CBTM Human Factors Quantification Analyses.......................93

Appendix D. General Introduction to PRA .....................................................................109

Appendix E. Guidelines for Human factors and Human Reliability Analyses ............113

References.................................................................................................................................117

vi

LIST OF FIGURES Figure Page

E-1. Relationship of Safety, Human Errors, and Their Influences ................................................x

E-2. Comparison of Results for CSXT and Territory Experience ..............................................xiv

1. Relationship of Safety, Human Errors, and Their Influences....................................................6

2. Overall Analytical Process.......................................................................................................28

3. Analytical Process for Crew Exceedance Event......................................................................38

4. Distribution of Number of Blocks Issued per Authority .........................................................41

5. Exceedance Rate per Block for all CSXT Territory and CBTM only Territory .....................42

6. Histogram and Best-Fit Distribution for Dispatcher-Caused Exceedances.............................44

7. Histogram and Best-Fit Distribution for Speeding Events ......................................................45

8. Histogram and Best-Fit Distribution for Mis-Positioned Switches .........................................46

9. The Language of PRA from Bley et al. (1992)......................................................................109

LIST OF TABLES Table Page

1. Summary of Data Associated with Each Human Failure Event. .............................................31

2. CSXT Train Miles (from FRA database).................................................................................39

3. Track Lengths for Different Operating Modes (FRA data) .....................................................39

vii

viii

EXECUTIVE SUMMARY Overview

The railroad industry is developing a new generation of processor-based signal and train control systems to improve safety and enhance operations. To meet the challenge of enabling railroads to adopt new signal processor based technology while reducing risk, the Federal Railroad Administration published the Notice of Proposed Rule Making (NPRM), Standards for Development and Use of Processor-Based Signal and Train Control Systems; Proposed Rule, (Department of Transportation, 2001). The NPRM proposes that probability-based risk analyses (PRAs) be used as part of a performance-based standard to evaluate the risk associated with the introduction of new systems.

Humans play a very important role in ensuring safety with the current train control systems. Actions include stopping trains when reaching the ends of approved track occupancy (either signal- or block authority-based), keeping train speeds within approved limits, maintaining separation from roadway workers and work locations, and taking actions when things generally go wrong. The NPRM specifically identifies the need to consider human actions, including their ability to provide coverage (i.e., to correct or overcome failures) for the automatic systems.

Any meaningful PRA needs to examine human actions (errors, decisions, work-arounds, circumventions, etc.) in a way that accounts for what is known about human performance in technological environments and how human errors can result. This report describes a general human reliability analysis (HRA) methodology for analyzing human performance and estimating the reliability of human actions that can be used in support of PRAs being performed as part of the Product Safety Plan (PSP) submissions to the FRA. In order to exercise and illustrate the HRA approach, it was applied to the safety evaluation of the Communications-Based Train Management (CBTM) System being tested by CSX Transportation, Inc. (CSXT). The report describes the overall approach to the HRA and its trial application to the CBTM evaluation.

The report includes a set of guidelines and recommendations for performing a human reliability analysis to insure that the results will be credible, acceptable to the broad set of stakeholders, meet accepted standards for human reliability analysis, and able to be integrated into probabilistic risk assessments.

It is intended to provide guidance for both organizations that are trying to develop an HRA plan as well as regulatory agencies such as the FRA charged with evaluating an HRA analysis that may be submitted as part of a product safety plan.

Approach for Human Reliability Analysis

The purpose of human reliability analyses is to estimate the likelihood of particular human actions (that may prevent hazardous events) not being taken when needed, or other human actions that may cause hazardous events (by themselves or in combination with other conditions) occurring. Failures to take action to prevent hazardous events, and actions that cause hazardous events, are commonly called human errors in HRA. This term does not imply that people are necessarily personally responsible or culpable in some way, just that an action was omitted (or taken) that adversely influenced safety.

ix

Defenses

Individual/Team actions Individual/Team actions

Task/Environmental conditions Task/Environmental conditions

Organizational factors Organizational factors

Losses AccidentDANGER

Hazards

Causes

(Adapted from Managing the Risks of Organizational Accidents, Reason, 1997)

Figure E-1. Relationship of Safety, Human Errors, and Their Influences

Figure E-1 shows a top-level representation of human performance, how human errors can create weaknesses in safety defenses, and how those human errors are conditioned by the environment in which people work. At the very top level, potentially hazardous situations (such as train collisions with other trains or roadway workers and derailments due to overspeeding) are prevented from becoming accidents through defenses being in place. The defenses include the train crew complying with the rulebook of operations, the use of the computer-aided dispatch system (CADS), adhering to speed limits, and the application of fail-safe design principles. Fail-safe design seeks to eliminate the hazardous effects of a failure by having the failure result in non-hazardous consequences.

It is the purpose of the HRA task to estimate the probabilities of human errors that can potentially fail the defenses. However, this estimation needs to take into account the work environment and task conditions under which the work is done, since these can provide an important influence on the likelihood of error. For example, bad weather, long shift times, and high workload all can increase significantly the likelihood of human errors. In turn, work environment and task conditions are often influenced by organizational factors like work rules, duty times, and so on. Therefore, the error estimation process needs to account for these contributing factors.

Human reliability analysis employs a set of tools to estimate the likelihood of required human actions being performed when needed. These likelihoods can then be incorporated into the overall risk assessment, so they can be combined with other probabilities, such as those of

x

equipment faults and other hazardous states, to estimate the overall likelihood of hazardous events.

There are four main tasks that need to be performed as part of an HRA. These tasks represent the general process by which human reliability analysis supports probabilistic risk assessment tailored to railroad operations. The details of these steps may vary in each application.

1. Qualitative Evaluation of Human Factors Issues. Analyze the impact of the current work environment and new technology on human performance. This task requires study of operating rules, procedures, available data, as well direct observation of the work environment and interviews of individuals involved in the work. The goal is to identify the major sources of human risk and reliability with and without the new system as well as to understand the factors in the current environment that enable errors to be caught and recovered.

2. Survey of Databases for HRA Sources. Identify collections of data that may be relevant to the quantification of errors, problems associated with direct application of that data, and ways in which experts in operations can evaluate and adjust that data to the case at hand.

3. Quantification. Develop quantitative estimates of the likelihood of the human actions in question. The process for quantification always begins with an evaluation of the relevance of available data to the actions under analysis. The data often provide a broad base for estimation, but almost all databases have limitations and gaps (such as the criteria for events to be recorded) compared with the modeling requirements of the PRA. In many cases an expert estimation process is used to make adjustments for these limitations and gaps.

One approach is to conduct an expert elicitation workshop that brings together experts in human factors, HRA and PRA and people with extensive experience in railroad operations to examine the available data and agree on plausible quantifications. The operations experts examine the models and assumptions to ensure that they represent the system as it is (or will be) operated. Experts in analysis and operations then jointly examine the available data and agree on adjustments to compensate for known limitations. For many events there will be no relevant tabulated data. In such cases, the workshop facilitators elicit the best available evidence from the experience of the experts in operations, which is then used as a basis for direct estimation of the error probabilities of interest.

The error probabilities are represented by distributions rather than a single-point estimates so as to explicitly represent the range of uncertainty in the estimate.

4. Documentation. To permit review and later understanding of the details of the quantification, all results and processes must be well documented, providing the bases for all estimates.

Section 2 of the report provides a detailed description of the steps involved in these four main tasks. The steps in the HRA process include:

Identify the specific unsafe actions to be estimated, as defined by the context of the PRA.

xi

Perform a qualitative human factors analysis to identify the major factors contributing to human risk and reliability.

Identify the relevant data sources for each action to be modeled.

Identify the limitations and gaps in each data source as related to the actions being modeled.

Implement an expert elicitation process to overcome the limitations and gaps in the data sources.

Synthesize and document the results.

Perform a review of the results by people familiar with train control operations to make sure the analyses and results are compatible with their experience.

Example Analysis for CBTM Study

In developing the NPRM, the FRA and members of the Railroad Safety Advisory Committee (RSAC) task force charged with developing the rule were concerned with how to assess safety of railroad operations using the new systems; i.e., what is the impact on operating risk. While the proposed rule allows for use of both qualitative and quantitative risk assessment methods, the FRA has supported the development of a quantitative simulation approach called the Axiomatic Safety-Critical Assessment Process (ASCAP) developed by the University of Virginia (Kaufman & Giras, 2000; Monfalcone, Kaufman, & Giras, 2001).

A major objective of the HRA project was to provide a demonstration of the HRA quantification process as input to risk quantification models such as ASCAP. The CSXT CBTM safety case was used to illustrate the methodology. CBTM is a form of train control that provides a warning to the locomotive crew when the train is predicted to exceed the limits of its authority and stops the train if the operator fails to act in time.

The HRA process outlined above was used to estimate human reliability values for input to ASCAP. This involved:

Estimating human reliability values for the base case: current railroad operations in the territory where CBTM was tested. This CSXT territory was located between Spartanburg, South Carolina and Augusta, Georgia. It was largely dark territory, with direct train control (DTC) as the method of operation. The operations analyzed in this study were exclusively DTC.

Examining the potential impact of CBTM on human performance and human

reliability when added to the current DTC operations in the above territory.

The study analyzed the probabilities of specific human errors representing potential contributors to the risks being modeled in the ASCAP study of the CBTM system:

1. Train enters a block without authorization

2. Train exceeds the track speed limit

3. Train enters a preplanned work zone (published in the train bulletin) without authorization

xii

4. Train crosses a misaligned switch

The CBTM system can potentially reduce the likelihood of occurrence of these events; they fall within the set of functions PTC was intended to address. Therefore, the analysis was performed for the base case (current operations without CBTM) and the case when CBTM is operational. Other accident scenarios, such as those involving grade crossings or collisions with Hi-rail vehicles used by inspectors were not modeled because they were not affected by the planned use of CBTM and therefore are not part of the ASCAP study. These represent important risks and would be analyzed for new systems that could affect them.

Qualitative Analysis

The qualitative human factors analysis involved two aspects: (1) an analysis of the current work environment to understand the types of errors that can arise and the factors that contribute to those errors; and (2) an examination of the proposed CBTM system, its user interface and proposed human-system interaction, to assess its potential impact on human performance and human reliability.

An early prototype of the CBTM system was being tested on the CSXT territory between Spartanburg, South Carolina and Augusta, Georgia. This provided us an opportunity to (1) directly examine its user interface features and observe its operation, and (2) get input from CSXT locomotive engineers and trainers who had familiarity with the prototype CBTM system.

As part of qualitative analysis, two site visits were conducted: a visit to the yard in Spartanburg, South Carolina to interview and observe CSXT locomotive engineers and conductors, as well as to ride a locomotive equipped with the CBTM system; a visit to the CSXT Dispatch Center in Jacksonville, Florida, to interview and observe dispatchers to understand CSXT dispatch operations and the factors that could contribute to dispatcher errors.

The results of the interviews and observations provided the background necessary for structuring the topics covered in the elicitation of expert evidence and estimation of probability distributions that occurred during a Human Factors Quantification Workshop that was conducted as part of the HRA quantification process.

Quantitative Analysis

The primary tasks in the quantitative analysis were the identification of relevant sources of data, specification of their limitations and gaps, and application of an expert elicitation process to compensate for these limitations and gaps.

Two kinds of data are required in HRA studies: information about the numbers of events similar to those being modeled, and information about the number of opportunities for such events so that a probability or frequency of the events can be estimated. Two major sources of data were identified in this study: databases maintained by the FRA, and databases maintained by CSXT. Both sources contain information about the frequencies of events and the opportunities for such events.

While these databases contained relevant information, they exhibited certain limitations and gaps with regard to the events being analyzed. In order to compensate for these limitations, the data needed to be filtered and scaled. To perform these adjustments, a two-day expert elicitation workshop was held on October 29 and 30, 2001, in Greenville, South Carolina. Thirty attendees participated in the workshop including: four railroad representatives and associated consultants;

xiii

thirteen workers, union representatives and associated consultants; six FRA representatives and associated consultants; one University of Virginia (ASCAP contractor) representative; and six Volpe Center and associated consultants (including the HRA team).

The formal process for elicitation of expert evidence and estimation of probability distributions is discussed fully in the main report. The final probability estimates for the human error events were computed based on the combination of the databases and expert judgments and generally took the form of probability distributions.

Results

Train-caused Block Boundary Exceedances

This event involved a train entering a block for which it does not have authority because of an error by the train crew. Based upon the available data sources, two paths potentially existed to analyze the likelihood of train-caused block boundary exceedances. One was to use the CSXT disciplinary data that were associated with all CSXT operations, and the second was to focus on the experience within the trial territory (between Spartanburg, South Carolina and Augusta, Georgia).

The CSXT-wide analysis led to an estimate of the exceedance rate to be a distribution with a mean of 3.14 x 10 7 events per train-mile. The territory-specific estimate was a distribution having a mean of 5.26 x 10-7 per train mile. This difference of a factor of two was considered not significant, given the number of assumptions used to generate them. The ASCAP analysis modeled the exceedance rate per block, not per train-mile, in its estimates. Given that there were 19 blocks along the test territory of 120.5 miles, the average block length was 6.3 miles. Therefore, the mean exceedance rate per block using the CSXT experience was 1.99 x10-6 events per block, and using the territory experience was 3.34 x 10-6 per block. To select between these two results, their distributions were compared. The comparison is shown in Figure E-2.

All CSXT Territory

CBTM Territory

0.000.020.040.060.080.100.120.140.160.18

1.0E-06 2.0E-06 3.0E-06 4.0E-06 5.0E-06 6.0E-06

Exceedance Rate/Block

Prob

abili

ty

Figure E-2. Comparison of Results for CSXT and Territory Experience

The two distributions overlap, with the territory specific distribution (labeled CBTM territory) extending past the CSXT wide distribution (labeled All CSXT Territory). Based on this comparison, the workshop participants agreed that the CBTM territory result should be used

xiv

since its mean was slightly more conservative, and its distribution enclosed that of the CSXT-wide analysis. Therefore, the distribution for use in ASCAP for the probability of a train crew to exceed its limit of authority can be approximated by a normal distribution having a mean value of 3.3 x 10-6 per block boundary and a standard deviation of 6.8 x 10-7.

A similar quantification process was used to provide probability estimate distribution for each of the other human error events analyzed for the base case. The following results were obtained:

Dispatcher-caused Boundary Exceedances: The mean rate for dispatcher-caused exceedances was 3.5 x 10-6 exceedances/block boundary.

Overspeeding Events: The calculated rate for exceedances per restriction was a distribution with a mean of 4.6 x 10-6 exceedances per speed restriction.

Switches: Two switching errors were considered: the likelihood of a manual switch being left in the wrong position, and the likelihood of a train running over a mis-positioned switch. The distribution of the likelihood of a switch being in the wrong position at the time a train approaches had a mean value of 1.3 x 10-4 per train. Of the 10 manually positioned switches along the length of the route, crews stated that (because of the visibility of the specific switch targets) they would not be able to observe the state of 7 switches when traveling southbound and 6 switches when northbound in sufficient time to stop before running over the switches when traveling at track speed. Of the 3 southbound and 4 northbound switches where the potential existed for stopping, the distribution of the probability of being able to stop in time when traveling at track speed had a mean value of 0.22. If traveling at slow speed (less than 10 mph, as if expecting to enter the siding), the likelihood of failing to stop was considered very low (1 in 10,000).

Work Zones: Data necessary for this event were not available. Workshop attendees suggested using the same fraction as for exceeding DTC block authority

Three conditions were analyzed for the use of the CBTM system:

1. The crew fails to gain control of the locomotive/train following indication of a warning before the penalty brake is applied

2. Train crew over-relies on CBTM (a complacency effect)

3. The train crew enters incorrect consist information into the CBTM system

These three events were selected for quantification based on requirements defined by the PRA, as well as results of the qualitative analyses that were conducted prior to and during the quantification workshop that suggested that these events were situations of potential concern.

The workshop attendees agreed that there was insufficient experience with the CBTM system to confidently project its potential impact on human performance. The local CSX locomotive engineers and conductors indicated that while they had the most experience with CBTM, they have only had the opportunity to operate CBTM equipped trains a couple of times each. Further, the field-tested version of the CBTM prototype was expected to improve substantially prior to actual implementation. Consequently, experience with the CBTM prototype was not expected to be representative of performance of the final production system.

xv

Given the level uncertainty with respect to the likely impact of CBTM on human performance, participants recommended performing sensitivity studies to explore how different assumptions about the impact of CBTM on human reliability would affect the results of the CBTM case.

The results for each of the three individual CBTM issues discussed at the workshop are summarized in the main body of the report. In some cases numeric probability estimates were elicited from the workshop participants. These estimates are presented along with the assumptions that served as a basis for the probability estimates. These probability estimates are recommended as starting points for sensitivity analyses.

Conclusions

The HRA methodology was able to generate reasonable results (i.e., acceptable to the workshop participants) despite the fact that there was no directly applicable database.

The workshop format permitted experts from many different organizations and backgrounds to work together and reach consensus. Uncertainty was expressed through probability distributions that were accepted by the group. The HRA and PRA/ASCAP teams reached agreement that the HRA results were appropriate for use in the PRA.

The approach taken in this study provides one viable way for others to perform HRA studies in support of the FRAs proposed Standards for Development and Use of Processor-Based Signal and Train Control Systems. The lessons learned from performing this example analysis of the CBTM system were documented and provide guidance on avoiding potential pitfalls in future human reliability analyses studies.

Although participants thought the approach worked well, there were several areas of concern:

Biases in data. Data from operational exposure databases or from the experts opinions has the potential to contain biases that lead to incorrect estimates of probabilities. The approach taken in this study has been to review these databases for potential limitations and biases in the reporting requirements for the databases, review these limitations and biases with the workshop attendees, and make filtering and scaling adjustments based on the inputs of the participants. We recognize that these adjustments represent opinions and the adjusted values may still contain biases. As discussed in the main report, we took steps to limit the potential for significant biases in these opinions, but there is no guarantee that the results are entirely free from bias.

Level of modeling of human error events. The HRA task estimated the likelihood range for the human actions of concern, such as entering a block for which the train has no authority. In contrast the ASCAP simulation modeled human error events at a smaller level of decomposition, explicitly modeling errors in perception and action, and failures to recover (coverage) from these errors. The rationale for the level of modeling adopted in the HRA study and recommendations for ways to deal with the potential mismatch between the ASCAP and HRA modeling are provided in the main body of the document.

Modeling of future CBTM operations. When the current HRA study took place, the CBTM system was still undergoing field trials, its design was not finalized, and only a limited number of engineers, conductors, and dispatchers had experience with the system. These factors limited our ability to predict the likelihood of errors with confidence. Nevertheless, interviews with engineers and conductors who had experienced the trials of the CBTM system, and discussions held during the expert elicitation workshop enabled identification of potential areas of design

xvi

and operation that might result in errors or other operational problems. Sensitivity analyses were recommended as a strategy for dealing with the high level of uncertainty associated with the potential impact of CBTM on human performance.

Recommendations for Future Analyses of Rail HRA Studies

The analytical situation that arose in the present study, having some relevant data but with a variety of limitations (not a perfect match for what we want to estimate, with sources that may lead to both under- and over-estimates of frequency) are far from unique to our case. They happen often both in the railroad industry and other industries, and must be addressed explicitly.

The approach we took for combining hard data with expert judgment is a good approach that could be used in other applications. It uses hard data to ground the experts judgments, while using expert judgment to compensate for the known limitations of the existing data.

Guidelines for human factors and human reliability analyses were generated based on the results of this project and are included in Appendix E of this document. The guidelines are intended for organizations developing an HRA plan as well as regulatory agencies such as the FRA charged with evaluating an HRA analysis submitted as part of a product safety plan. Recommendations include:

1. Use an HRA team that includes members experienced in performing human factors studies, human reliability analyses, probabilistic risk assessments, and group facilitation.

2. Model human errors at compatible levels in the PRA and HRA tasks, preferably at the level of available data and experience.

3. Verify that the data sources (databases, expert judgment or a combination) are suitable for the tasks and associated errors being analyzed. Identify gaps or mismatches and utilize expert judgment to leverage the available data while compensating for the known limitations of the data.

4. Conduct qualitative task analyses with people experienced in using the existing systems. Activities should include interviews with workers using the existing systems or the target users of the system (in the case of technologies under development), their trainers and supervisors, so that all levels of experience are included.

5. Utilize expert elicitation methods that take into account known biases and other limitations of expert judgment. Experts should express their opinions in terms of ranges rather than single point values.

6. Solicit input from as broad a range of stakeholders as possible so that the analysis takes into account a wide range of perspectives. Accept quantitative inputs only during the elicitation process, from people with relevant operating experience.

7. Ask the broadest range of stakeholders possible to review the results of the analyses to foster support for the results.

xvii

xviii

1. INTRODUCTION This report describes an approach to evaluating the reliability of human actions that are modeled in a probabilistic risk assessment (PRA) of train control operations. This approach to human reliability analysis (HRA) has been applied in the case of a safety evaluation of the Communications-Based Train Management (CBTM) System being tested by CSXT Transportation, Inc. (CSXT). This report describes the overall approach to the HRA and its trial application to the CBTM evaluation.

1.1 Use of Risk Assessment for FRA Historically, the evaluation of train control systems has been design-based. That is, components of a train control system were evaluated based on engineering performance criteria taking into account operability, reliability, and maintainability criteria. With the advent of recent changes in electronic technology, FRA and the railroad industry felt that new and better train control systems might be adopted more quickly using a performance-based approach, assuming that safety could still be assured.

FRA and the industry agreed that performance standards should be based on accident risk assessment and that a quantitative assessment of safety risk associated with any new system should favorably compare against the existing system. Safety or accident risk is defined as the product of the probability of an accident and a measure of the severity or consequences of that accident.

The requirements for performing a quantitative risk assessment are contained in the Federal Railroad Administrations (FRAs) proposed Standard for Development and Use of Processor-Based Signal and Train Control Systems (Department of Transportation, 2001). The proposed rule addressed the development of positive train control (PTC) systems made possible by the introduction of emerging technology in processor-based signal and train control systems. Positive train control systems address three core functions:

Preventing train-to-train collisions;

Enforcing speed restrictions and temporary slow orders;

Providing protection for roadway workers and their equipment.

The complexity of these technologies (communication and information technology) requires additional safety considerations that current safety evaluation methods do not address.

The proposed rule adopted a performance-based approach to enable flexibility in the design and implementation of PTC systems while providing a mechanism to achieve safety goals. The performance standard adopted in the rule requires that the new product or system must not degrade safety below the level of the existing system. To evaluate whether this condition is met requires a risk assessment comparing the new system to the system it will replace.

This proposed rule would require that any railroad wishing to use a processor-based control system (such as a PTC system) to provide more effective or efficient control of train movements must submit a Product Safety Plan (PSP) that includes a quantitative risk assessment that compares the Mean Time to Hazardous Events (MTTHE) for related railroad operations with and without use of the processor-based control system to show that there would be no reduction in

1

safety from implementing the system. The proposed rule also requires that MTTHE values must incorporate the impact of all elements of the system. These elements include human factors as well as the hardware and software components.

While this rule is not final, it is considered very likely that the final rule will contain the same conceptual requirements for performing a quantitative risk assessment as part of the PSP.

In developing the proposed rule, the FRA and members of the Railroad Safety Advisory Committee (RSAC) task force charged with developing the rule were concerned with how to assess risk. Methods for estimating risk vary in complexity from parametric extrapolation of accumulated experience to quantitative modeling (Hollnagel, 1998). While the proposed rule allows for use of both qualitative and quantitative risk assessment methods, the FRA has supported the development of a quantitative modeling approach called the Axiomatic Safety-Critical Assessment Process (ASCAP) developed by the University of Virginia (Kaufman & Giras, 2000; Monfalcone et al., 2001). The ASCAP model considers all types of failures (including human) and is intended to estimate the overall riskboth the probabilities of accidents and the measures of their consequences. ASCAP may be used to determine the comparative risk of the base case vs. an alternative, in this case CBTM.

Train control systems have associated accident risks from non-human failures (i.e., mechanical, electrical, and electronic, materials) as well as human failures. This study focused on:

1) The development of an approach to assess only the human failures in train control systems;

2) The use of that approach to estimate probabilities of human failures on the

Spartanburg subdivision of the CSXT railroad under its current train control system (base case);

3) Estimation of likely human failure probabilities under a new and different type of

system (CBTM) that overlays on the existing one; and

4) Formatting and defining those human failure probabilities for use in the ASCAP model.

1.2 Role of HRA Within the scope of the PRA, it is necessary to include human actions and errors that can lead to (or prevent) the hazardous events whose frequencies are to be estimated. Humans play a very important role in ensuring safety with the current train control systems. Actions include stopping trains when reaching the ends of approved track occupancy (either signal- or block authority-based), keeping train speeds within approved limits, maintaining separation from roadway workers and work locations, and taking control when things generally go wrong. The draft rule specifically identifies the need to consider human actions, including their ability to provide coverage (i.e., to correct or overcome failures) for the automatic systems.

Human reliability analysis employs a set of tools to estimate the likelihood of these human actions being performed when needed. These likelihoods can then be incorporated into the overall risk assessment, so they can be combined with other probabilities, such as those of

2

equipment faults and other hazardous states, to estimate the overall likelihood of hazardous events.

Unlike the generally well-documented and accepted methods for estimating hardware failure probabilities, methods for estimating human reliability parameters are not well matured. There exist a wide range of available methodssee the review by Gertman and Blackman (1994), which documents many different approaches. Many more have been developed since that review. However, there is a growing recognition that the most effective methods are those based on failure data for the actual operating experience of the system being modeled and gathered over the widest range of field conditions. However, as with CBTM and other PTC systems for which there is little or no actual operational experience, the preferred HRA methods are those that can combine operating data with modeling or judgment since the operating data by themselves are insufficient or not directly associated with the system being modeled. This is the approach taken in this study.

Since the data are necessarily incomplete (the system not yet being in operation) or only partly relevant to the system being modeled, it is necessary to consider that the sparseness of the data and the judgments needed to supplement the data may introduce uncertainties in the results. In addition, we often do not have complete knowledge of the effects of all the factors that influence the human performance being modeled, another source of uncertainty in the predictions. In order to represent these different sources of uncertainty, we have taken the approach of explicitly representing these uncertainties by calculating distributions rather than providing single-point estimates for the probabilities of human error. More details, together with the overall approach to managing the different sources of uncertainty are presented in Section 2.3.3.

1.3 Purpose of Report The purpose of this report is two-fold: first, to describe a general HRA estimation process that can be used in support of PRAs being performed as part of the PSP submissions to FRA under the proposed standard described; and second, to present the steps in, and results of, the application of this process in the PRA of the CBTM system being tested by CSXT.

Section 2 of the report describes the principal steps in the HRA process that can be used in other applications. Section 3 presents the principal results of applying the method in the analysis of the CBTM system, with detailed results of the qualitative analysis being presented in Appendix A and B and those of the quantitative analysis in Appendix C. Section 4 of the report presents the lessons learned for future applications of the HRA modeling in future studies, and Section 5 summarizes the recommendations for future studies and the conclusions of this work.

3

4

2. APPROACH TO ESTIMATION OF HUMAN RELIABILITY IN TRAIN CONTROL SYSTEM STUDIES

2.1 Overall Approach The purpose of human reliability analyses is to estimate the likelihood of particular human actions (that may prevent hazardous events) not being taken when needed, or other human actions that may cause hazardous events (by themselves or in combination with other conditions) occurring.

Failures to take action to prevent hazardous events, and actions that cause hazardous events, are commonly called human errors in quantitative risk assessments. This term does not imply that people are necessarily personally responsible or culpable in some way, just that an action was omitted (or taken) that adversely influenced safety.

In the context of HRA a human error is simply an action taken (or omitted) by a person that leads to an unwanted outcomeit makes the situation less safe.1 Note that there is no attribution of blame or fault embedded in this view. People can be placed in situations where an error is almost inevitable. Often we can say It was not his fault in regard to some error where we can see almost anyone could make the same error in the same situation. Increasingly, the term unsafe action (or unsafe act), rather than human error, is being used in HRA, to emphasize that it is the action (or the failure to act) that is of concern, not whether the action would be considered an error. For example, if a person were led into taking an unsafe action by their training and procedures, many people would say that that was not an error in the normal sense of the term, yet the action had unsafe consequences.

Figure 1 shows a top-level representation of human performance, how human errors can create weaknesses in safety defenses, and how those human errors are conditioned by the environment in which people work. At the very top level, potentially hazardous situations (such as train collisions with other trains and roadway workers and derailments due to overspeeding) are prevented from becoming accidents through defenses being in place. The defenses include the train crew complying with the rulebook of operations, the use of the computer-aided dispatch system (CADS), adhering to speed limits, and the application of fail-safe design principles2. For the most part, these defenses prevent accidents. However, these defenses presently rely almost exclusively on human performancefor example, there are very few automated defenses other than the checking effects of CADS in dark territory.

1 The concept of human error has diverse interpretations in the different disciplines of engineering, psychology, and the law. See Human Error: Cause, Prediction, and Reduction (Senders & Moray, 1991) for the results of a workshop intended to characterize the different facets of the term. 2 Fail-safe design seeks to eliminate the hazardous effects of a failure by having the failure result in non-hazardous consequences.

5

Defenses

Individual/Team actions Individual/Team actions

Task/Environmental conditions Task/Environmental conditions

Organizational factors Organizational factors

Losses AccidentDANGER

Hazards

Causes

(Adapted from Managing the Risks of Organizational Accidents, Reason, 1997) Figure 1. Relationship of Safety, Human Errors, and Their Influences

Unsafe actions by individuals or teams (such as the train crew) can reduce the effectiveness of the defenses, thereby making the likelihood of an accident higher. It is the purpose of the probabilistic risk assessment (PRA) to estimate the frequencies of such accidents by estimating the probabilities of failure for each of the different defenses. It is the purpose of the HRA task to estimate the probabilities of the human errors that can potentially fail the defenses. However, this estimation needs to take into account the work environment and task conditions under which the work is done, since these can provide an important influence on the likelihood of error. For example, bad weather, long shift times, and high workload all can increase the likelihood of human errors. In turn, work environment and task conditions are often influenced by organizational factors like work rules, duty times, and so on. Therefore, the error estimation process needs account for these contributing factors, either explicitly (by the modeling process making adjustments) or implicitly (through using data that already incorporate the practical influence of these factors).

An important aspect of the human reliability analysis process is to identify the contributing factors that may cause an unsafe action to be made. Contributing factors can be external (to the person) conditions like poor radio equipment or signals, or a train that is difficult to control, or internal (to the person) conditions like fatigue or boredom, which we know lead to paying reduced attention to the track ahead. While conceptually separate, in practice these often interact. For example, fatigue is unlikely to cause an unsafe action in a simple routine task, but is very likely to cause an unsafe action in a very challenging situation where concentration or detailed memory recall is required. In practice, we think it makes little difference whether a contributing factor is classified as external or internal. What matters is whether it is a problematic situation.

6

Therefore, when this report discusses contributing factors, we generally do not concern ourselves with whether they are external or internal.

Further, things can become more complicated when one unsafe action becomes a contributing factor for another. For example, an engineer may mishandle the train, and the resulting behavior of the mishandled train creates the conditions that lead to further errors. In the current example, unsafe actions in train control may lead to an overspeeding train. The engineer, in trying to control the overspeeding train, may make braking errors that cause a derail. In other industries, most accidents involve multiple unsafe actions3.

2.2 Relationship with Risk Assessment Activities Human reliability analysis is just one component, though a very important one, of an overall PRA such as the type required under the proposed FRA rule. In terms of the relationship between HRA and the PRA, perhaps the most important is that the PRA defines the scope of human errors for HRA required for estimation. The PRA lays out the basic events that can (singly or in combination) result in the hazardous events of concern to the end-user of the studyhere, the FRA. An event, refers to a significant occurrence that has the potential to be an accident in the wrong circumstances. For example, a train being in a block for which it has no authority is an event. If another train happened to be in the same block traveling in a location and at a speed where it would not see the intruder in time to stop, then a collision would occur. The train being in the block may be the result of an unsafe action, such as the engineer failing to recognize the limit of his authority or the dispatcher incorrectly giving the engineer verbal authority to proceed. However, the train could enter the unauthorized block for other reasons, such as mechanical failure of the braking system. Therefore, an event can occur for several or many reasons, some of which are unsafe actions. A human failure event refers to an event that occurs as a result (either in part or entirely) of one or more unsafe actions.

The PRA usually specifies what human failure events are to be quantified, such as train enters block without authority. Once the PRA has established the overall framework of actions that need to be modeled, the HRA can develop its own internal set of representations of human actions that are consistent with the type of analysis to be performed for the individual human errors. The HRA examines the set of contexts and unsafe actions that can produce that human failure event. In most cases there are multiple different contexts and unsafe actions that can produce the same human failure event. For example, a train can enter a block without authority because of a dispatcher error (e.g., the dispatcher gave the train crew verbal authority to enter the block, but failed to enter the information into the Computer-aided dispatch system). Alternatively the train could enter the block without authority because the train crew was distracted and failed to stop. Yet another alternative is that the train crew intended to stop but underestimated the braking distance required. The function of the HRA analysis and quantification process is to uncover the various contexts and unsafe actions that can result in a

3 For example, a review of major aviation accidents by the U.S. National Transportation Safety Board (NTSB) showed that in the 37 major accidents reviewed from 1978 to 1990, the range for the number of unsafe actions per accident was from 3 to 19, with a median of 7. [A Review of Flight Crew-Involved, Major Accidents of U.S. Air Carriers, 1978 Through 1990 - Safety Study (NTSB/SS-94/01 (PB94-917001)). Washington, DC: U.S. National Transportation Safety Board. 1994].

7

given human error event, and to quantify the probability of the human failure event, given the variety of contexts and unsafe actions that can lead to it.

2.3 HRA Process The estimation of the probabilities of human failure events and their contributing unsafe actions can be performed in several different ways. First, various different kinds of models exist to estimate these probabilities and are based on such parameters as the time available for people to take necessary actions, or the quality of indications and instructions for various tasks. Many of these models are summarized by Gertman & Blackman (1994), although more recent developments, such as the ATHEANA method (NRC, 2000) that focuses on cognitive processes and problems, are not included. Almost all of the these developments have taken place in the context of the nuclear power industry and the need to model human actions under extremely rare and challenging conditions, as during a nuclear reactor accident for which few relevant data exist.

A second approach is to recognize that data exist that are related to the kinds of failure events and unsafe actions being modeled. Unlike the actions associated with extremely rare events, these data are usually associated with everyday, or at least frequent, activities like routine train operations, maintenance actions, and so on. Depending on the kinds of data that are gathered, these data sources can be used to identify ranges of probabilities for specific types of unsafe actions.

A third way of estimating human error probabilities is to use the experience of domain experts as a basis for estimation. In particular, the experts need to be experienced in the performance of the tasks being modeled, and the different kinds of errors that can occur under actual working conditions. While, these experts (such as locomotive engineers and dispatchers) may not have expertise to express their opinions in formal statistical terms, techniques have been developed to help elicit their knowledge and convert that knowledge into probabilities, as described below.

These general approaches are not necessarily mutually exclusive. Wreathall, in an evaluation of these different approaches (Wreathall, 2001), recommended that the most useful results for those cases where relevant data exist, is to combine the use of data and expert estimation. The data often provide a broad base for estimation, but almost all databases have limitations and gaps (such as the criteria for events to be recorded) compared with the modeling requirements of the PRA. The expert estimation process provides a way to make adjustments for these limitations and gaps. This overall approach is recommended for studies such as this where an agency like FRA must evaluate base cases and the effects of change.

This section describes the basic steps involved in performing a human reliability analysis. The objective is to describe a general process that can be used to perform an HRA as part of a PRA. The goal is to generate HRA results that are credible, acceptable to the broad set of stakeholders, meet accepted standards for human reliability analysis, and are able to be integrated into probabilistic risk assessments.

The general steps that need to be performed as part of an HRA are:

Identify the specific unsafe actions to be estimated, as defined by the context of the PRA

8

Perform a qualitative human factors analysis to identify the major factors contributing to human risk and reliability.

Identify the relevant data sources for each action to be modeled

Identify the limitations and gaps in each data source as related to the actions being modeled

Implement an expert elicitation process to overcome the limitations and gaps in the data sources

Synthesize and document the results

Perform a review of the results by people familiar with train control operations to make sure the analyses and results are compatible with their experience.

Each of these steps is described in more detail below.

2.3.1 Identify the Human Failure Events and Unsafe Actions to be Estimated

In most cases, the PRA will have developed a list of human failure events that it considers as potential contributions to the hazardous events it is modeling. Sometimes these will be identified to the level of unsafe actions. It is recommended that the HRA modeling team and the PRA team jointly review this list and agree to a scope of the HRA modeling that will satisfy the requirements of the PRA.

This list should be developed to identify the particular unsafe actions relevant to the human failure events being analyzed and the level at which they will be modeled. For example, will the modeling separately represent basic errors and recoveries or will they be modeled such that only the final outcome state will be represented? Will the model, for example, separately identify the failure of the engineer to recognize the end of their authority and failure of the conductor to correct the engineers unsafe action, or will the analysis just model failure of the crew to stop at the appropriate limit? As observed by Wreathall (2001), the recommended practice is to model at the level of the events in the database if possible, since this results in fewer opportunities for mismatches between the data and the modeling. Any lower level or subdivision of modeling should be undertaken only if necessary to generate results that must be used at different places in the PRA. For example, unsafe actions and their recoveries should be separated only if the recovery mechanisms in the situations being analyzed as part of the PRA are substantially different from those represented in the database.

The product of this activity will be an agreed scope of unsafe actions to be modeled in the HRA task, and for which results will be provided to the PRA at the end of the HRA task.

2.3.2 Perform Qualitative Evaluation of Human Factors Issues

Once the scope of failure events and unsafe actions to be modeled in the HRA is defined, the next step is to develop a qualitative understanding of the major factors contributing to human risk and reliability. This involves a human factors analysis of the current work environment, and its impact on human performance.

A qualitative analysis also serves to identify the possible impact of a new technology on human performance and the potential for unsafe actions.

9

The FRA risk-based evaluation process requires a risk analysis to determine whether the introduction of a new technology, such as positive train control, will result in a level of safety that is equal or higher than the level of safety given current technology. This requires first performing an analysis to quantify the risk associated with the base case (with existing technology) and then comparing this risk to the estimated risk once the new technology is introduced.

A qualitative analysis can identify the major sources of human risk and reliability in the base case. It can also be used to identify the possible impact of the new system on human performance and potential for unsafe actions. The qualitative results can feed into the HRA quantification process and provide additional qualitative information to support evaluation of the proposed new technology (Product Safety Plan).

Evaluating factors influencing human reliability in the current environment

While documents such as operating rules and procedures, and human performance databases, can serve as a starting point for a human factors analysis, these sources often provide an incomplete picture of the actual demands of the work environment and work practice.

A more comprehensive understanding can be obtained through direct observation of the work domain and interviews with the people who are involved in the work (Mumaw, Roth, Vicente, & Burns, 2000). In the context of railroad operations, this means conducting visits to the work sites in question (e.g., dispatch centers, rail yards) to observe the work context directly, and interviewing the people who have direct experience with the job (e.g., locomotive engineers, dispatchers, roadway workers). Useful sources of information include: the workers themselves, labor representatives, first-line supervisors and managers, and training staff.

Observation and interview methods may draw on a variety of methods that include ethnographic approaches (Gamst, 1990; Heath & Luff, 2000; Jordan & Henderson, 1995; Nardi, 1997), cognitive field studies (Roth and Patterson, in press), one-on-one structured interview techniques, or focus group techniques that elicit information from multiple people at once (Krueger & Casey, 2000).

Observations and interviews enable human reliability analysts to uncover and document physical, cognitive, and collaborative demands imposed by the work domain and the strategies that workers have developed to cope with those demands. In many cases these factors and the strategies that domain practitioners have developed to cope with them are not documented or well understood and can only be uncovered by observing and interviewing the individuals directly engaged in the work.

Observations and interviews provide an important source of information about the nature of the work, the factors in the environment that add complexity and create opportunities for error, and the kinds of errors that can occur. This includes an understanding of the broad range of worker duties and practices, the characteristics of the physical environment that can contribute to error (e.g., lighting, temperature, noise), the characteristics of the tools and systems that people interact with that can contribute to error (e.g., characteristics of computer systems, radios), the mental and physical demands of the work itself (e.g., the cognitive demands, the distractions that can arise, the need to time-share tasks), the need for communication and coordination with others within and outside the immediate work environment, as well as the characteristics of the

10

organizational environment (e.g., attitudes, policies, procedures) that can influence performance and contribute to error.

Observations and interviews provide an opportunity to learn about the kinds of errors that have occurred, and the factors that contributed to those errors. It allows the analyst to learn about near misses that were never documented since they didnt lead to a reportable accident.

People not only contribute to increased risk through errors, they also contribute to increased reliability by catching and correcting problems before they lead to an accident. An important objective of the qualitative analysis is to identify the individual, team, organizational, and system factors that enable problems to be caught and corrected before they lead to serious negative consequences.

Recent research across a variety of domains (e.g., aviation, medicine) have shown that highly trained professionals make errors with relatively high frequencies (Amalberti & Wioland, 1997; Reason, 1998). For example, Amalberti reports error rates of up to 3 per hour for aviation cockpit crews are not unusual. The mark of a high reliability system is not that errors are rarely made, but that there are mechanisms in place that enable error detection and recovery. The best pilots and surgeons anticipate the likelihood of errors and develop effective compensatory and error recovery strategies. Similarly, the mark of a high-reliability team is that they are able to catch and recover from each others errors. For example, in the studies of the cockpit crews cited by Amalberti, the overwhelming majority of the errors are detected and recovered by the crews in less than 10 seconds.

One of the important aims of a qualitative analysis is to understand the factors in the current environment that enable errors to be caught and recovered. Understanding the factors that make the current system robust to errors in evaluating the potential impact of proposed changes. Changes in technology can have unintended negative consequences. For example, they may eliminate a feature of the current environment that on the surface appears to be of no consequence, but in fact supports robust performance and reduces the potential for error.

Evaluating the potential safety consequences of new technology

One of the proposed uses of human reliability analysis is to support the risk-based evaluation of new technology. When a new technology is introduced that requires human interaction, you cannot evaluate the performance of the new technology in isolation. You need to consider the role that the human may play in either enhancing the overall performance or degrading it. This requires performing analyses to quantify risk in the base case (with existing technology) and comparing this risk to the estimated risk once the new technology is introduced.

In estimating the risk associated with the new technology, it is necessary to consider the impact of the new technology on human performance (Parasuraman & Riley, 1997; Parasuraman, Sheridan & Wickens, 2000). There are a number of human factors issues that need to be considered when evaluating the likely impact of a new system on human performance and potential for error.

One of the first questions to ask is what is the joint person-machine system design? This includes:

What functions will human and machine agents perform?

11

What information will be passed among them?

How good is the performance of the human and machine elements of the system expected to be (e.g., what is the expected accuracy; what is the expected reliability)?

Whether, and on what basis, one element of the system will be allowed to over-ride or take over from the other?

Other questions to consider include:

Does the new support system change how the human performs?

Does the new support system prevent and/or catch and help recover from the types of unsafe actions known to occur in the base system?

Does the new system introduce any new sources of risk?

o Does it contribute to any new types of unsafe action? o Does it place the human in situations that might encourage them to

circumvent it?

o Does it introduce any other new sources of risk? Are there mechanisms built into the new support system that allow the human to play

a supervisory control role that would mitigate the potential for any new sources of risk created by the introduction of the system (i.e., opportunities for humans to provide coverage for any new sources of risk)?

If designed well, the joint human-machine system can perform better than either human or machine on their own. If designed poorly, the joint human-machine system can actually perform worse than each of the individual elements. For example, if an automated system has a relatively high miss rate or a relatively high false alarm rate (e.g., in ambiguous or conflict situations), then the human may choose not to use it, or over-ride its decision even under conditions where the automated system is correct (Parasuraman & Riley, 1997). Conversely, the human may accept the recommendation of the automated system even in cases where the automated system is beyond its bounds of competence. The goal is to assign roles to the human and computer elements of a system, and provide them with the necessary information and displays to support these roles, so as to maximize the joint human-machine system performance.

Problems associated with poor joint human machine system design have included:

Loss of operator vigilance and situation awareness resulting in complacency and an increase in vigilance-associated human errors. As operator confidence in the automatic system increases, the operators tend to become more complacent and less vigilant. Thus, they may fail to detect indications of impending or existing automation problems which require human intervention (Sheridan, Gamst, & Harvey, 1999).

New opportunities for unsafe actions related to configuring the automation (e.g., inputting wrong values into the automated system such as a wrong ID or destination code)

12

Skill loss. With increased supervisory train control technology, the opportunity for the operators to perform the task themselves decreases. The lack of opportunity for practice contributes to skill loss. Skill loss is a problem where the automated system becomes inoperable or is beyond its bounds of competence and the human must take over.

An increase in workload demands during high tempo high-risk conditions where workload is already very high. One of the common pitfalls of automated systems is that they automate the easy elements of a task, reducing workload during periods where workload is already low, but require extensive human intervention for the difficult cases (e.g., aircraft landings) where workload is high.

There are a number of qualitative methods that can be used to evaluate the potential impact of a new technology on human performance. Approaches include:

A review of the relevant research base both within the railroad industry and in related industries (e.g., aviation, process control). Examples include a review of experiences within the railroad industry with respect to the introduction of new train control technologies such as the Automatic Train Control Systems that was evaluated as part of the Swedish TRAIN-project (Kecklund and the project group, 2001), as well as review of experiences with new automation in the aviation industry (e.g., Woods, Sarter, and Billings, 1997).

A human factors evaluation of the proposed design or of an early prototype implementation of the design can be performed to assess how well the proposed design adheres to established human factors design principles (Billings, 1997). This can be performed by a human factors specialist with knowledge of problems associated with poor joint person-machine designs and human-centered design principles for effective joint person machine systems (e.g., Christoffersen and Woods, in press; Roth, Malin, and Schreckenghost, 1997).

Interviews of domain practitioners who have had an opportunity to review and/or use early prototypes of the proposed system. Domain practitioners have operational knowledge and experience that allow them to recognize factors that may limit the usefulness or usability of the system that the designers may not be aware of. Examples include complex cases that the system will not be able to handle, environmental issues such as lighting or noise level that may make the user interface difficult to use, or high workload or multiple attention demands that may make it difficult to use the system as envisioned by the designers (e.g., a locomotive engineer may need to focus his or her visual attention out the window and may be unable to continuously monitor a display for messages and warnings.)

More formal person-in-the-loop evaluations of the system. These person-in-the-loop tests involve evaluation of the joint person-machine system. The tests examine the ability of domain practitioners to utilize the system effectively in a range of realistic conditions. For example, if there is a new automated system about to be implemented in a locomotive cab, then a person-in-the-loop test would involve having locomotive engineers run a train equipped with the system. Objective measures (e.g., time to detect a system message, time to take necessary action) can then be obtained to assess

13

the impact of the new technology on performance. The tests could be done either in a high fidelity simulator or in the field.

Generally the qualitative analysis incorporates several of these methods in order to gain a fuller understanding of the issues involved in the introduction of the new technology and the potential impact on human performance.

Summary

Qualitative analyses enable human reliability analysts to more realistically model the types of unsafe actions that occur and the factors that contribute to those errors. Specifically, qualitative analyses enable human reliability analysts to:

Identify the major sources of human risk and reliability in the base case:

1. What are the most likely forms of unsafe action in the base case?

2. What are the factors that are most likely to contribute to those errors?

3. What recovery mechanisms do humans provide that contributes to a robust, high-reliability system?

Identify the likely impact of the new system on human performance:

1. Does the new system prevent and/or catch and recover from the types of unsafe actions that are known to occur in the base system?

2. Does the new system change how the human performs?

3. Does it contribute to any new types of unsafe action (e.g., foster complacency, create a source of distraction)?

4. Does the new system introduce any new sources of risk? Does the system design allow the human catch and recover from the system errors?

Results feed into the HRA quantification process and provide additional information to support evaluation of the proposed system (Product Safety Plan)

2.3.3 Identify Sources of Relevant Data

The process of quantification begins with an evaluation of the relevance of available data to the human actions under analysis. For each of the human actions identified in the list created jointly with the PRA task, it is necessary to identify potentially relevant data sources that can be used to estimate the frequencies with which these errors may occur, and what the number of opportunities may be for such events. Dividing the numbers of errors by the corresponding numbers of opportunities will yield the needed probability of error per occurrence.

It is unlikely that one data source will provide all the needed information. Further, if possible it is helpful to obtain multiple data sources so that several estimates can be created for cross-comparison and selection of a suitable probability range can be made for each unsafe action to be analyzed.

Examples of potentially useful data include the following. Specific additional sources may exist, depending on the particular unsafe actions being analyzed.

FRA incident databases

14

FRA operating experience databases

Railroad incident databases

Railroad disciplinary actions databases

Railroad operating experience databases

The data in the FRA databases are generally available for specific railroads. The railroad-specific data may be available for specific sections of track or territories, or only for the whole system. Issues associated with using these data are discussed next.

2.3.4 Identify Limitations and Gaps in Data Sources

It will be almost certain that the databases identified in the previous step will not match exactly the unsafe actions and events being analyzed in the HRA. Typically, there are two kinds of gaps

1. The database includes events that are not relevant to the kinds of unsafe actions being analyzed

2. The database does not include all events of the type being analyzed in the HRA

An example of the first gap would be the reports of all incidents within a railroad system when the analysis is only concerned with (for example) overspeeding or authority exceedance within one particular type of train control system. In this case, the database must be filtered to identify only the events that match the scope of the analysis. Other examples of events requiring filterin