Predictors of Threat and Error Management: Identification ... · PDF filebeen reconceptualized explicitly as the development of threat and error counter-measures (Helmreich, Merritt,

Predictors of Threat and ErrorManagement: Identification of Core

Nontechnical Skills and Implications forTraining Systems Design

Matthew J. W. ThomasUniversity of South Australia

Adelaide, Australia

In normal flight operations, crews are faced with a variety of external threats andcommit a range of errors that have the potential to impact negatively on the safety ofairline operations. The effective management of these threats and errors thereforeforms an essential element of enhancing performance and minimizing risk. Recentresearch has reinforced the need to examine a range of nontechnical or crew resourcemanagement skills that form threat and error countermeasures. This article providesan analysis of the predictors of threat and error management in normal flight opera-tions within the context of a Southeast Asian airline. Through the structured observa-tion of crews’performance during normal flight operations, data were collected in re-lation to a set of contextual factors and nontechnical skills. Crews’ threat and errormanagement actions were then analyzed in relation to these factors, and predictivemodels of threat and error management at various phases of flight were developed.The results of this study demonstrate the ways in which this type of data analysis canhighlight the strengths and weaknesses of operational performance and suggest thatthis type of performance evaluation can offer individual organizations invaluable in-formation for enhanced training system design through the further development ofscenario-based training.

For any organization involved in high-risk operations, the adequate performanceof personnel is a crucial aspect of maintaining safety. It is now clearly understoodthat any failures of safety stem not simply from isolated incidences of human error

THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 14(2), 207–231Copyright © 2004, Lawrence Erlbaum Associates, Inc.

Requests for reprints should be sent to Matthew J. W. Thomas, Flexible Learning Centre, Univer-sity of South Australia, City West Campus, Y1–35 Yungondi Building (GPO Box 2471) North Terrace,Adelaide SA 5000. E-mail: [email protected]

but rather from wide-ranging organizational factors. Encapsulating considerableresearch into organizational safety, models of accident trajectories have been de-veloped that include both active failures of personnel and systems as well as latentconditions that may lie dormant in an organization’s operational system for consid-erable time (Reason, 1990, 1997). Closely aligned to the concept of active failuresand latent conditions are the terms error and threat, respectively, concepts thathave recently been the focus of considerable research in the commercial aviationsetting.

Defined as situations, events, or errors that occur outside the flight-deck, threatsare conditions that have the potential to impact negatively on the safety of a flight.In turn, defined as crew action or inaction that leads to a deviation from crew or or-ganizational intentions or expectations, errors are taken to be an unavoidable andubiquitous aspect of normal operations. As these two factors form fundamentalcausal components of incidents and accidents, it is argued that the management ofthreat and error must form the focus of any organization’s attempts to effectivelymaintain safety in high-risk operations (Klinect, Wilhelm, & Helmreich, 1999).

THREAT AND ERROR MANAGEMENT ANDNONTECHNICAL SKILL DEVELOPMENT

Threat and error management involves the effective detection and response to in-ternal or external factors that have the potential to degrade the safety of operations(Helmreich, Klinect, & Wilhelm, 1999). From this new approach to safety hasemerged a redefined emphasis on training that extends beyond merely the develop-ment of technical proficiency in areas such as system operation and specificpsychomotor skills. Central to the emergent focus on threat and error managementis the position that effective operational performance is dependent on the inte-grated use of specific technical skills and generic nontechnical skills such as cog-nitive and interpersonal skills.

The specific development of crews’ nontechnical skills is not a recent additionto aviation training. In direct response to the realization of the contribution of hu-man factors to incidents and accidents, training programs were developed to focuson nontechnical skills. Crew resource management (CRM), defined as the crews’effective use of all available resources to achieve safe and efficient flight opera-tions, has historically been an important focus toward the reduction of human errorand the enhancement of safety (Lauber, 1987; Wiener, Kanki, & Helmreich, 1993).The primary focus of CRM was the development of discrete nontechnical skillssuch as communication, leadership, decision making, and conflict resolution aswell as stress and fatigue management (Helmreich & Wilhelm, 1991). Evolvingthrough a number of successive generations, CRM training has in recent times

208 THOMAS

been reconceptualized explicitly as the development of threat and error counter-measures (Helmreich, Merritt, & Wilhelm, 1999).

The effective training of personnel in threat and error management will not beable to provide a panacea that eliminates the human contribution to incidents andaccidents (Helmreich & Merritt, 2000). However, specifically tailored trainingprograms can certainly make a critical contribution to the minimization of risk. It isnow well understood that flight crew act as the last line of defense in what is often aflawed sociotechnical system within which aircraft operate (Reason, 1997). There-fore, by equipping crew members with skills in threat and error management, anorganization can provide additional defenses against both active failures as well aslatent conditions that may lie undetected within the organization or the broader op-erating environment.

CURRENT DEFICIENCIES IN THREAT ANDERROR MANAGEMENT TRAINING

The development of crews’ nontechnical skills through such mechanisms as Hu-man Factors courses and CRM training is now a widespread regulatory require-ment. However, the development of nontechnical skills remains a problematic areaof aviation training. As Trollip (1995) argued, one of the major problems facingnontechnical skill development by flight crew is that the traditional approaches totraining are generally ineffective for the development of nontechnical skills.

It has been demonstrated recently that there is a lack of coherence in the ap-proaches to nontechnical skill development across airlines. Significant differenceexists in relation to whether the training adopts a focus on crews’ attitudes or spe-cific behaviors as well as in relation to the specific labels, descriptions, and repre-sentations of the attitudes or skills that are the focus of training (Salas, Rhodenizer,& Bowers, 2000). Furthermore, it is evident that the instructional techniques em-ployed in the development of nontechnical skills require further development. Forinstance, recent studies have indicated that nontechnical skills frequently remainneglected in the evaluation and debriefing of pilots in regular check situations(e.g., Hörmann, 2001).

Current approaches to flight crew training have also been criticized for not suf-ficiently integrating crews’ technical and nontechnical skill development. Fre-quently, flight crew training involves a preliminary focus on aircraft technicalknowledge and operational procedures, with the development of nontechnicalknowledge and skills occurring in latter stages of training and in isolation fromreal-world operational contexts (Johnston, 1997). This lack of integrated trainingpresents a major barrier to the development of effective threat and error manage-ment skills. In short, there is a lack of clarity about the relative importance of awide range of attitudes and behaviors that contribute to effective threat and error

PREDICTORS OF THREAT AND ERROR MANAGEMENT 209

management and a deficit in the industry’s understanding of the most effectivemechanisms for their training.

TOWARD A MORE DETAILED UNDERSTANDINGOF THREAT AND ERROR MANAGEMENT

There is little doubt that understanding how threats, errors, and their managementinteract to determine the quality of operational performance is critical to safety inhigh-risk industries (Helmreich, 2000). Although the link between nontechnicalskills and threat and error management is obvious, there remains a significant needto develop a more detailed understanding of the factors that contribute to effectivethreat and error management. Moreover, to design effective training curricula inthis area, it is necessary to increase the depth of one’s understanding of threat anderror management in the environment of normal operations. New tools for opera-tional performance evaluation provide unprecedented opportunities for the analy-sis of threat and error management during normal flight operations. In particular,the Line Operations Safety Audit (LOSA) methodology developed by the HumanFactors Research Project at the University of Texas enables the collection of de-tailed data in relation to the occurrence of threats and errors during normal opera-tions and details of the types of behaviors crews employ in response to these threatsand errors (Helmreich, Klinect, et al., 1999; Klinect et al., 1999).

The study I present in this article involves the analysis of the types of contextualfactors and nontechnical skills that contribute to threat and error management byflight crews during normal flight operations. The aim of this research is to providea more detailed understanding of threat and error management behaviors with theobjective of better informing training system design.

METHOD

Participants

The study was undertaken within a Southeast Asian airline operating both domes-tic and international routes as part of a broader project involving a structured evalu-ation of both normal operations and flight crew training (Thomas, 2003). Normalline operations from two fleets of the airline were examined, one comprised ofBoeing 737–300 and 737–400 aircraft flying domestic short-haul operations andthe other comprised of Airbus A330–300 aircraft flying medium-haul interna-tional routes. The primary focus of behavioral analysis was that of the flight crew,which was consistently comprised of a two person active crew of a Captain and a

210 THOMAS

First Officer. However, some additional specific data was collected in relation tothe actions of individual crew members.

Design and Procedure

In the study, I adopted an observational design and employed a highly structuredobservational performance evaluation methodology for data collection and analy-sis. Data were collected by a group of 25 senior flight crew from the airline whowere trained in using the LOSA methodology (Helmreich, Klinect, et al., 1999;Klinect et al., 1999). The observers collected data from the jump seat during nor-mal line operations. A 2-day training session was held for the observers and cov-ered all logistic and technical aspects of data collection. The standardization of ob-servers was a major focus of the training sessions, and interrater reliability wasestablished through a process of reflective analysis of videotaped examples ofcrew performance. In total, 323 sectors (individual airport-to-airport flights) ofnormal operations were observed (approximately 200 for the B737 fleet and 100for the A330 fleet).

Measures

Dependent variables. The major focus of this study was the threat and errormanagement of flight crew during normal line operations. The outcomes of crews’threat management actions were simply recorded as a single dichotomous vari-able, namely, whether the crew had or had not effectively managed the threat. Forinstance, in relation to an encounter with adverse weather en route, effective detec-tion of the weather, and appropriate actions undertaken to avoid weather penetra-tion would be coded as an “effectively managed” threat. Conversely, if the crewfailed to detect the threat or if their management actions lead to an error being com-mitted, the threat would be coded as “not effectively managed.” The crews’ errormanagement actions were initially coded under three categories according to thethreat and error management model: (a) the error was trapped, which means that itwas detected and managed before it became consequential; (b) the error was exac-erbated, which means the error was detected, but the crew’s action or inaction leadto a negative outcome; and (c) the crew could fail to respond to the error, whichmeans the crew either failed to detect or ignored the error. Due to the limited num-bers of errors that were exacerbated, data were recoded to form two distinct dichot-omous dependent variables: first, whether the crew had or had not trapped the errorand second, whether the crew had or had not failed to respond to an error.

Independent variables. A set of contextual factors and crews’ nontechni-cal skills were examined as possible predictors of threat and error management.


Although not specifically manipulated within the observational design, both thecontextual factors and nontechnical skill ratings were considered as importantindependent variables, as they are frequently identified as essential mediators ofoperational performance and safety. First, during each flight, data were collectedon a series of seven contextual factors that might impact on the crews’ manage-ment of threats and errors. These seven contextual factors are listed in Table 1.

Crews’ nontechnical performances were evaluated using set of behavioralmarkers adapted from the existing NOTECHS and LOSA methodologies. Abroad four-category structure of communication, situation awareness, task man-agement and decision making was adapted from the NOTECHS system, whichhas been developed under European regulations (Flin, Goeters, Hörmann, &Martin, 1998). Under these four categories, 16 behavioral markers were devel-oped to provide performance data on a wide range of competencies that havebeen specifically identified as threat and error avoidance, detection, and manage-ment actions (Helmreich, Wilhelm, Klinect, & Merritt, 2001). Observers re-corded scores against each of the four categories of nontechnical skill for eachflight observed to gain an overall impression of crew performance at the macrolevel. At a higher level of resolution, observers recorded scores against the 16 in-dividual behavioral markers for each of five phases of flight for every flight ob-served. The four categories and 16 behavioral markers are found in Table 2.

The observational methodology also involved the collection of qualitativedata by the observers in the form of a written narrative. In relation to each quan-titative measure embedded within the methodology, observers were required towrite a descriptive narrative of the events they observed, which described crewactions and justified the observer’s coding and rating of threat and error manage-ment and nontechnical skills. This information was used to develop a richer un-derstanding of the crews’ actions and was used in the overall analysis and inter-pretation of data.

212 THOMAS

TABLE 1Independent Variables Identified as Contextual Factors

for Threat and Error Management

Variable

Length of flight (min)Departure timeLate departureNumber of threatsNumber of errorsExperience of the Captain (years)Experience of the First Officer (years)

Statistical Analysis

Once the data were collected, they were collated and subjected to both descriptiveand multivariate statistical analysis. As the dependent variables describing threatand error management outcomes were recorded as dichotomous nominal values,logistic regression was utilized as the most appropriate multivariate technique. Lo-gistic regression is used as an alternative to standard multiple regression tech-niques in describing and testing hypotheses about relations between a categoricaloutcome variable and one or more predictor variables (Cizek & Fitzgerald, 1999).Further, logistic regression was chosen as an alternative to discriminant analysisthat involves assumptions of multivariate normality. According to standard proce-dures, independent variables that were not recorded as continuous values wererecoded and analyzed as categorical variables (Peng, Lee, & Ingersoll, 2002). Thelogistic regression models were subjected to overall model evaluation and tests ofgoodness of fit to ensure the models were sound.

One possible implication of the study design was the potential for commonmethod variance (CMV) to result in a circularity of the relations between observ-ers’ perceptions of threat and error management and their ratings of crews’ non-technical skills. As Kline, Sulsky, and Rever-Moriyama (2000) suggested, CMV isa common problem facing self-report measures in which relations between a rangeof variables are detected in data collected using the same instrument. In these situa-tions, the relations that emerge might be the result of a common relation with a


TABLE 2Behavioral Markers for Core Nontechnical Skills

Category Behavioral Marker

Communication CommunicationEnvironmentLeadership–followershipInquiryAssertivenessCooperationStatement of plans and changes

Situation Awareness VigilanceMonitoring and cross-check

Task Management Briefing and planningWorkload managementWorkload prioritizationAutomation managementManagement of fatigue and stress

Decision Making Contingency planningProblem identificationEvaluation of plans

third spurious and unmeasured variable rather than an independent relation be-tween the two or more variables as measured. Within this study, it was suggestedthat CMV is unlikely to effect the statistical analysis and findings for two reasons:(a) significant independence between the variables measured by the observers waswell defined within the measurement protocols, and (b) observers were required tojustify their ratings with clear evidence of observed crew behaviors that were sub-jected to independent review by a data-cleaning roundtable. These research designstrategies effectively minimized possible observer bias and reduced the likelihoodof CMV effects.

RESULTS

Descriptive Analysis of Threat and Error Management

From the 323 observations of line operations, 451 individual threats were observedand recorded. This was an average of 1.4 threats per observed flight operation.Klinect et al. (1999) reported a similar average number of threats per flight, withdata from 314 sectors indicating an average of 1.9 threats per flight. Markedinterairline variability was evident in the baseline study, with the average numberof threats ranging from 0.3 to 3.3 per flight. This considerable range highlights theutility of threat analysis in identifying the unique operational characteristics of anairline or fleet’s operating environment.

On average, in the line operation environment, 57.4% of threats in this studywere well managed by the crews. However, the remaining 42.6% of threats werepoorly managed by the crews. As a threat, by definition, has the potential to be det-rimental to the safety of the flight, it was hoped that a much larger proportion ofthreats would have been effectively managed by the crews. There were 22 differenttypes of threats faced by flight crews during normal operations. Table 3 shows thedistribution of the 10 most common threats, which accounted for 90% of all threatsencountered.

From the 323 line observations, a total of 508 errors were committed, represent-ing an average of 1.57 errors per flight. Captains were responsible for the majorityof the errors, with the First Officer responsible for less than one fourth of the er-rors. Although procedural errors were the most common type of errors committedby flight crew, a significant proportion of intentional noncompliance errors werecommitted, demonstrating a propensity for flight crew to violate standard operat-ing procedures (SOPs) and regulations. In coding an intentional noncomplianceerror, observers were required to demonstrate demonstrable behaviors that indi-cated that the noncompliance was intentional. Although not suggesting that crewattitudes, values, or behavior are at fault, this error type highlights the need for fur-ther investigation in relation to the possible root causes of intentional noncompli-

214 THOMAS

ance including procedure design and operational necessity. A summary of the er-rors and their management by flight crew is provided in Table 4.

Nearly half of all errors remained undetected by flight crew. First Officers per-formed better than Captains in error detection, and air traffic control also detecteda significant number of errors before flight crew were able to do so. Less than onefourth of all errors were effectively managed by flight crew, with the most commonerror management response being a failure to respond to errors.

Predictors of Effective Error Management

At a macro level, error management during all phases of flight was examined in re-lation to the seven contextual factors and the four major categories of nontechnicalperformance. Using logistic regression, a model was developed for effective errormanagement, which examined the predictors of errors being trapped by the flightcrew. As described in Table 5, it was found that three of the contextual factors andnontechnical skills were found to be significant predictors of errors being trappedby the flight crew. Of the nontechnical skills, it was found that, controlling for allother independent variables, an increase in the crews’ decision-making perfor-mance was linked to an increased likelihood of the error being trapped. Similarly,the more experienced the First Officer, the more likely errors were to be trapped.However, if the Captain committed the error, it was significantly less likely to betrapped by the flight crew. These findings demonstrate a delicate balance betweenthe benefits of experience and the negative effects of power on crew interaction andperformance.

When examining crews’ failure to respond to errors, it was found that the expe-rience of the First Officer and whether the Captain had committed the error were


TABLE 3The Most Common Threats Encountered by Flight

Crews and Their Management

Threat Type Frequency (% of All Threats)a % Managed % Not Managed

Weather 20.6 53.8 46.2Aircraft malfunctions 14.4 75.4 24.6Operational pressure 11.5 38.5 61.5Traffic 7.8 54.3 45.7ATC command 7.5 44.1 55.0Airport conditions 7.1 78.1 21.9Terrain 6.2 53.6 46.4Ground handling event 5.8 42.3 57.7Passenger event 5.1 52.2 47.8Communication event 4.0 72.2 27.8

aN = 451.

216

TABLE 4A Summary of Errors Committed by Flight Crews and Their Management

Error Origin

Frequency(% of AllErrors)a Error Type

Frequency(% of AllErrors)a Error Detection

Frequency(% of AllErrors)a Error Response

Frequency(% of AllErrors)a

Captain 60.8 Intentionalnoncompliance

38.4 Captain 15.9 Trap 23.6

First Officer 24.8 Procedural 41.3 First Officer 24.2 Exacerbate 13.2Both crew 14.4 Communication 6.7 Both crew 3.3 Fail to respond 63.2

Proficiency 3.3 Air trafficcontrol

7.7

Decision making 10.2 Nobody 46.9Other 2.0

aN = 508.

significant in the prediction of ineffective error management. Another importantpredictor of ineffective error management as shown in Table 6 was the late depar-ture of the flight. If the flight was late to depart, crews were 1.89 times more likelyto fail to respond to an error.

Although these results identify core contextual factors and nontechnical skillsat the macro level, by examining errors committed during specific phases of flight,a more finer resolution analysis was achieved. Of the errors committed by flightcrew, 85% occurred during the predeparture, takeoff, and descent-approach-land-


TABLE 5Logistic Regression Analysis—Contributions of Contextual Factors

and Nontechnical Skills to Errors Being Trapped by Flight Crew During AllPhases of Flight

Predictor β SE β Wald’s χ2 df p eβ

Experience of FirstOfficer

0.060 .026 5.440 1 .020 1.061

Decision making 0.809 .189 18.351 1 .000 2.247Captain as error

origin–0.966 .289 11.159 1 .001 0.381

Constant –2.773 .535 26.893 1 .000 0.062

Note. Model χ2 = 58.729, df = 3, p < .001. Nagelkerke R2 = .234; 79.4% correct classification. Inreporting logistic regression, β refers to the regression coefficient. The statistical significance of the re-gression coefficient is measured using Wald’s χ2 with p < .05 used as the test of significance. The eβ re-fers to the odds ratio that provides an indication of the change in the likelihood of the dependent vari-able associated with changes in the independent variable.

TABLE 6Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crews’ Failure to Respond to Errors During All

Phases of Flight


Late departure .637 .308 4.285 1 .038 1.891Experience of

First Officer–.049 .022 4.871 1 .027 0.952

Captain as errororigin

.968 .248 15.280 1 .000 2.633

Constant .153 .288 0.280 1 .597 1.165


ing phases of flight. Accordingly, crews’ error management behaviors were exam-ined against data from the seven contextual factors and 16 behavioral markers ofnontechnical skills for each of these three phases of flight.

During the predeparture phase, it was found that only one nontechnical skillemerged as a significant predictor of errors being trapped by flight crew. As themodel in Table 7 indicates, an increase in crew cooperation resulted in a marked in-crease in the odds of an error being trapped.

It was found that during the predeparture phase, higher levels of contingencyplanning by the flight crew significantly predicted a decrease in the likelihood thatthe crew would fail to respond to an error (Table 8). This finding suggests thatcrews who actively engage in predeparture contingency planning may have height-ened expectation that unexpected errors might occur and are thus less likely to failto detect or ignore them.

During the takeoff phase, contingency planning was also found to be a signifi-cant predictor of effective error management, thus confirming its core function as

218 THOMAS

TABLE 7Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Errors Being Trapped by Flight Crew During the

Predeparture Phase


Cooperation 2.517 1.080 5.435 1 .020 12.389Constant –7.436 3.090 5.791 1 .016 0.001


TABLE 8Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crews’ Failure to Respond to Errors During the

Predeparture Phase


Contingencyplanning

–1.484 .541 7.525 1 .006 0.227

Constant 3.933 1.390 8.003 1 .005 51.055


an error countermeasure. However, increased workload management, defined asthe effective management of workload through the definition of roles and responsi-bilities and the appropriate sharing of operational tasks, was found to decrease thelikelihood of errors being trapped and increase the likelihood of failure to respondto errors. A number of possibilities can account for this somewhat anti-intuitivefinding. First, it is possible that heightened levels of workload management resultin a diversion of cognitive resources that might be better focused on other areasthat are more effective threat and error countermeasures. A second suggestion isthat the often highly proceduralized tasks assigned to each crew member duringthis phase of flight might be working against error detection and management. Inshort, the script that each crew member must follow could leave little room for ef-fective threat and error management. The predictive models for error managementduring the takeoff phase can be found in Table 9 and Table 10.

The descent-approach-landing phase of flight is typically characterized by in-creased workload and an overall increase in risk. It was found that the nontechnicalskills of vigilance and problem identification were significantly related to in-creased likelihood of errors being trapped by crews during the descent. Both thesenontechnical skills are linked closely to the notion of situation awareness and referto the perception and comprehension of critical operational cues. As shown in Ta-ble 11, it was also found that an increase in the number of errors committed bycrews decreases the likelihood of errors being trapped.

If the Captain was responsible for the error, it was again found that the likelihoodof the crew failing to respond to the error increased. However, the demonstration ofhigh levels of assertiveness by the crew and one would suggest the First Officer waslinked to a decrease in the likelihood that the crew would fail to detect or ignore theerror. These findings reinforce the important contribution of the climate and power



Takeoff Phase


Workloadmanagement

–3.047 1.030 8.754 1 .003 0.048

Contingencyplanning

1.948 0.769 6.415 1 .011 7.014

Constant 2.150 1.723 1.558 1 .212 8.588


structure on the flight deck and emphasize the need for assertiveness duringhigh-risk situations. The model describing contributions of contextual factors andnontechnical skills to crews’ failure to respond to errors is provided in Table 12.

Predictors of Effective Threat Management

At the macro level, it was found that two contextual factors and two nontechnicalskill categories were significant predictors of effective threat management byflight crews during all phases of flight. First, the more experienced the First Offi-

220 THOMAS

TABLE 10Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crews’ Failure to Respond to an Error During the

Takeoff Phase


Workloadmanagement

2.297 0.864 7.066 1 .008 9.941

Contingencyplanning

–1.113 0.576 3.738 1 .053 0.329

Constant –2.804 1.513 3.434 1 .064 0.061



Descent-Approach-Landing Phase


Number oferrors

–0.751 .272 7.652 1 .006 0.472

Vigilance 1.093 .506 4.677 1 .031 2.984Problem

identification1.105 .558 3.927 1 .048 3.020

Constant –4.050 1.480 7.491 1 .006 0.017


cer, the more likely it was that crews would effectively manage threats. Conversely,an increase in the number of errors decreased the likelihood that threats were effec-tively managed. Both situation awareness and decision making were positivelylinked to effective threat management, highlighting the importance of these cate-gories of nontechnical skills. The overall model of predictors of effective threatmanagement can be found in Table 13.

It was found that the majority of threats were encountered by flight crews dur-ing the predeparture and the descent-approach-landing phase of flight. Accord-ingly, the threat management performance of crews was analyzed for these twophases of flight against the seven contextual factors and 16 behavioral markers of


TABLE 12Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crew’s Failure to Respond to an Error During the

Descent-Approach-Landing Phase


Captain as errororigin

1.555 .608 6.531 1 .011 4.735

Assertiveness –0.849 .339 6.253 1 .012 0.428Constant 0.856 .979 0.764 1 .382 2.354


TABLE 13Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crew’s Effective Threat Management During All

Phases of Flight


Experience of FirstOfficer

.084 .032 7.126 1 .008 1.088

Situationawareness

1.224 .338 13.098 1 .000 3.401

Decision making 0.943 .311 9.204 1 .002 2.568Number of errors –0.590 .138 18.314 1 .000 0.544Constant –4.859 .905 28.814 1 .000 0.008


nontechnical skills. During the predeparture phase, a model of effective threatmanagement emerged that included four factors. First, the nontechnical skills ofstatement of plans and changes and briefing and planning were found to contributesignificantly to increased likelihood of effective threat management. An increasein the number of errors committed by crews was found to decrease the likelihoodof effective threat management. However, if crews encountered an increased num-ber of threats, they were more likely to engage in effective threat management.This model is shown in Table 14.

A similar predictive model for effective threat management emerged in relationto threats encountered during the descent-approach-landing phase of flight. Theexperience of the First Officer was again found to be related to an increase in thelikelihood of effective threat management, whereas an increase in the number oferrors was related to a decrease in the likelihood of effective threat management asdescribed in Table 15.

Overall, the individual predictive models of threat and error management high-light a range of contextual factors and nontechnical skills that act as threat and er-ror countermeasures. The difference between the models for each threat and errormanagement task and each phase of flight have significant implications for train-ing system design.

DISCUSSION

Predictors of Threat and Error Management

In this study, I investigated the contribution of contextual factors and nontechnicalskills to threat and error management during normal flight operations. Each of the

222 THOMAS

TABLE 14Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crews’ Effective Threat Management During the

Predeparture Phase


Number of threats 1.353 0.608 4.949 1 .026 3.871Number of errors –0.894 0.409 4.783 1 .029 0.409Statement of plans

and changes3.009 1.104 7.427 1 .006 20.727

Briefing and planning 2.284 1.318 3.001 1 .083 9.812Constant –14.819 4.701 9.936 1 .002 0.000


four categories of nontechnical skills were represented as predictors of threat anderror management behaviors across the different phases of flight. Further, a varietyof contextual factors were also found to contribute to the predictive models ofthreat and error management. However, it was found that the contribution of thecontextual factors and nontechnical skills were quite different for effective threatmanagement when compared to effective error management. Similarly, differ-ences in the predictive models were also found between the successive phases offlight. These differences can be best demonstrated by examining each of the fourcategories of nontechnical skill and the contextual factors in turn.

In relation to the broad nontechnical skill category of crew communication, itwas found that cooperative and interactive functions were essential for high levelsof operational performance during the preflight phase. In this study, I found thatthe statement of plans of changes and cooperation were predictors of effectivethreat and error management, respectively. These aspects of nontechnical compe-tence can be described simply as forms of communication that involved the ex-change of information and the development of a shared understanding of situa-tions. In contrast, during the descent-approach-landing phase when actions weretime critical, assertiveness was found to be a major predictor of effective errormanagement. This highlights the use of a different communication strategy inachieving effective performance, one that serves more direct and action-orientedfunctions. These findings suggest that a close relation exists between interactivegroup processes and effective threat and error management on the flight deck. Anumber of recent research studies have begun to explore the role of communica-tion in error management and have focused on the identification of effective com-munication strategies (Bowers, Jentsch, Salas, & Braun, 1998; Fischer & Orasanu,1999). However, the results of this study suggest that it is unlikely that any singlecommunication strategy will be found to be most effective in relation to threat anderror management, and stress that effective threat and error management involves


TABLE 15Logistic Regression Analysis—Contributions of Contextual Factors andNontechnical Skills to Crews’ Effective Threat Management During the

Descent-ASpproach-landing Phase


Experience ofFirst Officer

.149 .069 4.702 1 .030 1.160

Number of errors –.695 .251 7.652 1 .006 0.499Constant –.073 .991 0.005 1 .942 0.930


skills in the adaptation of communication strategies to suit situational require-ments.

The nontechnical skill category of task management was the least representedin the models of effective threat and error management. This suggests that consid-erable further research is required to unpack the individual nontechnical skillsfound in this category. One specific skill, that of briefing and planning, was foundto be a predictor of effective threat management in the preflight phase, again rein-forcing the importance of communication strategies that assist in the developmentof a shared understanding of situations between crew members as a crucial aspectof threat and error management.

The nontechnical skill category of situation awareness was found to be relatedto effective threat management across all phases of flight. This finding reinforcesthe perspective that maintaining an accurate mental model of the status of aircraftsystems and environmental factors is essential to effective performance (Endsley,1993). At a finer level of resolution, vigilance was found to be an important predic-tor of effective error management during the descent-approach-landing phase. Aserror detection is the primary action in effective error management, the benefits ofincreased vigilance during time-critical phases of flight are clearly supported bythis research.

The final category of nontechnical skill, that of decision making, was foundto be related to effective threat and error management across all phases of flight.Specifically, in relation to effective error management, contingency planning andproblem identification were found to be critical. This finding can be interpretedas highlighting the importance of remaining open to and being prepared for theoccurrence of error. Crews that planned for a variety of eventualities and wereable to identify problems as they occurred in normal operations were more likelyto effectively manage error when it occurred. This finding reinforces the needfor training to emphasis the ubiquitous nature of human error in normal opera-tions and prepare crews to manage that error as it occurs. As Reason (1997) ar-gued, at a broad level, both risk perception and hazard awareness are essentialaspects of the organizational management of error. The findings of this studysuggest that these approaches can easily be extended to the level of the individ-ual operator in the commercial aviation environment, and therefore, an importantstep toward effective threat and error management is increasing pilot awarenessof the potential for error.

In this research study, I also found that a variety of contextual factors werelinked to both effective and ineffective threat and error management during normaloperations. First, a number of operational factors were found to be important pre-dictors of threat and error management. Whereas the length of the flight and thetime of day were not found to be linked to threat and error management, opera-tional pressure through a late departure was clearly linked to poor error manage-ment. An increased number of threats during normal operations was actually

224 THOMAS

linked to improved threat management, a finding that might suggest the positiveinfluence of increased cognitive arousal and an external frame of reference createdby a number of external factors impinging on normal operations. In contrast, an in-creased number of errors impacted negatively on both threat and error manage-ment of crews. Together, these findings suggest that the quality of threat and errormanagement by crews is particularly sensitive to the influence of both external andintracrew pressures.

Second, a number of factors related to crew dynamics were found to be predic-tors of threat and error management. The experience levels of pilots was found tobe a particular influence on threat and error management. Specifically, the moreexperienced the First Officer, the more likely crews were to effectively manageboth threats and errors. Although this finding may initially indicate a simple rela-tion between experience and operational expertise, the links to other influencessuch as assertiveness suggest experience is as much an influence on the ability of aFirst Officer to speak up and identify threats and errors as it is on technical exper-tise. Furthermore, the origin of the error was a significant predictor of error man-agement. If the Captain committed the error, it was systematically less likely to bewell managed across all phases of flight. These findings highlight the importanceof crew dynamics and the influence of the flight deck authority gradient on effec-tive threat and error management.

Together, these findings provide significant empirical support for the impor-tance of nontechnical skills in minimization of risk and the overall enhancement ofoperational performance. Further, the findings of this research stress the impor-tance of taking into consideration contextual factors such as crew experience, op-erational pressures, and team structure in our understanding of effective threat anderror management.

Applications: Implications for Training System Design

An explicit aim of this research study was to further develop the understanding ofcrews’ threat and error management during normal line operations to inform train-ing systems design. Although this study has provided a robust analysis of the pre-dictors of threat and error management during normal operations, there is an inher-ent danger in attempting to generalize these results to any airline or indeed anyorganization operating in high-risk industries. It is likely that the unique character-istics of an organization will produce significantly different models of effectivethreat and error management. In particular, organizational and national culture hasrecently been highlighted as a critical factor in accepting the validity of generaliza-tions about models and concepts in the field of industrial and organizational psy-chology (Gelfand, 2000). Therefore, whereas this research does provide consider-able insights into the types of contextual factors and nontechnical skills essential toeffective threat and error management, the conclusions drawn from this work are


more powerful in terms of the processes that an organization can undertake to ef-fectively inform their own training systems design. In essence, this study demon-strates that an organization should utilize new forms of performance evaluationdata in the creation of effective threat and error management training.

The use of performance evaluation data is an essential component of trainingsystems design, and it is paramount that training decisions are based on high-qual-ity data (Goldsmith & Johnson, 2002). However, as Helmreich and Merritt (2000)suggested, training curricula developed in one organization is less effective whenexported to another organization. Especially in relation to threat and error manage-ment, factors such as organizational and national culture, aircraft type, existingtraining regimes, SOPs, and the unique operating environments of individual air-lines must be taken into account when designing effective training programs.Therefore, whereas it is desirable to develop detailed task analyses and a general-ized understanding of job performance, an organization must utilize its own per-formance evaluation data to ensure that its training systems are effective in prepar-ing crews for work in what is likely to be a unique operating environment.

In the United States, the Federal Aviation Administration has explicitly ac-knowledged the need for airlines to design training systems that respond directly totheir individual operational requirements. Through the Advanced QualificationProgram (AQP), airlines are able to take advantage of flexibility in their expositionof regulatory requirements such that they can customize their flight crew trainingprograms to reflect specific operational needs (Neumeister & Mangold, 1997).One of the major components of the AQP is the requirement for an airline to under-take a formal data collection and validation program to ensure that their training ismeeting operational needs (Taggart, 1994). Accordingly, although this researchdoes not profess to provide predictive models of threat and error management thatcan be generalized across any number of airlines, the research does highlight theutility of a new approach to training needs analysis that has the potential to over-come some of the limitations of traditional approaches. The forms of data collec-tion and analyses discussed in this article highlight potentially new and more pow-erful forms of operational performance evaluation that have the potential to addconsiderably to the AQP process.

Traditional approaches to aviation instructional design predominantly employCognitive Task Analysis (CTA) as the data source for development of training in-terventions (Oser, Cannon-Bowers, Salas, & Dwyer, 1999). Typically, CTA pro-vides those responsible for training design with detailed analysis of both theknowledge and skills required for expert performance through the decompositionof work elements. However, it has been acknowledged that CTA has the propensityto provide decontextualized forms of information about effective performance thatcan be rather abstracted from the operational environment (Seamster, Redding, &Kaempf, 1997). Accordingly, although CTA is a robust empirical method for thedetailed analysis of individual job performance, it does not provide sufficient in-

226 THOMAS

formation to an organization about the complex contextual factors that drive andinfluence performance. Through its reductionistic approach to the analysis of op-erator performance, it is doubtful that CTA alone can provide a sufficiently com-prehensive analysis of performance to inform the development of threat and errormanagement training. As effective threat and error management is highly contextspecific, training toward the development of these skills must require the rich inter-pretation of effective performance within complex environmental and organiza-tional frameworks.

The new tools for operational performance evaluation and in particular theLOSA methodology discussed in this article provide an organization with richforms of operational performance evaluation data that can complement traditionalCTA approaches to training systems design. By integrating detailed informationabout individual job performance with rich information about contextual factorsand nontechnical performance that underpin safety in the real-world operationalenvironment, significant benefits can be achieved in training systems design.

Applications: Integrating Technical and Nontechnical SkillDevelopment

The current lack of true integration between the development of technical and non-technical skills has been highlighted as a major deficiency in current approaches toflight-crew training (Hörmann, 2001; Johnston, 1997). Typically, an initial typetraining program within an airline context progresses through linear phases, eachphase making use of increasing technological complexity to support instruction.Training will typically commence with classroom based ground school to intro-duce technical systems knowledge, then progress to the use of computer-basedtraining (CBT) and flight-training devices (FTDs) to simulate the operation andfunctions of individual aircraft systems, and finally culminate in the use of fullflight simulators to integrate system operation. During this program some non-technical skill development might be “integrated” into this predominantly techni-cal training program during the latter stages of full flight simulation, yet the major-ity of nontechnical skill training occurs during discrete classroom sessions. Thistraditional approach to training ensures adequate knowledge and skills in aircraftoperation yet does not do so in a manner that promotes effective development ofthreat and error management skills that are operationally relevant to the aircrafttype and to the operational environment in which the pilots will be required to per-form.

Given the results of this study, one possible mechanism for improving these tra-ditional approaches to training would be through the further development of sce-nario-based training, which explicitly integrates technical and nontechnical train-ing. Currently, scenario-based training occurs through the process of line-orientedflight training (LOFT) in which crews practice both technical and nontechnical


skills in realistic and challenging flight situations (Dismukes, McDonnell, & Jobe,2000). Typically, LOFT involves a “full-mission” operational scenario presentedduring full flight simulation that presents crews with a variety of problems includ-ing multiple aircraft systems malfunctions and other in-flight events as they com-plete a simulated flight sector. The scenario-based training contained in LOFT ses-sions is typically limited to the final stages of initial type training and then periodicsessions as parts of recurrent training.

The further development of such scenario-based training in threat and errormanagement might take advantage of discrete events throughout the trainingprogram in which a crew must utilize both technical and nontechnical skills tosolve operational problems. To this end, the integrated technical and nontechni-cal skill development would filter backward through the training program andutilize less expensive technologies than full flight simulation in the deploymentof scenario-based training. Rather than maintaining the current emphasis offull-mission scenarios in LOFT, threat and error management training could in-troduce operational scenarios into discrete training events built using either CBTor FTDs.

In many respects, the scenarios captured by the use of LOSA-type evaluationsprovide essential data to inform such integrated training design, as they containconcrete examples of both effective and ineffective threat and error countermea-sures used in practice. For instance, during training in the use of the flight manage-ment computer (FMC), a scenario for use in a fixed-base simulator might be devel-oped in relation to adverse weather conditions on descent. Working as a crew, thetrainees would be required to engage in appropriate action utilizing knowledge ofthe airline’s SOPs and technical knowledge of the aircraft systems along with non-technical skills such as decision making and communication. Rather than being anexercise purely in reprogramming the FMC, such scenario-based training wouldachieve effective integration of technical and nontechnical skill toward enhancedoperational performance.

Full training programs could therefore be developed as a series of sce-nario-based training events in which crews progressively work through the use ofaircraft systems in a manner that is both operationally relevant and extends beyondmerely technical knowledge and skills. Possible benefits from this approach wouldinclude enhanced training effectiveness through the immediacy and improved fo-cus of instructor feedback as well as the obvious benefits in relation to increasingthe operational relevance of nontechnical skills.

Although this does not represent a radical departure from the broad linear struc-ture of aviation training, this scenario-based approach can achieve for the first timeeffective integration of technical and nontechnical skills training. Accordingly, itdoes offer a significantly different approach to nontechnical skill development.Ideally, through the development of such scenario-based approaches to training, a

228 THOMAS

more coherent concept of operational performance can be developed in which theartificial divide between the perceived technical and nontechnical aspects of per-formance is eliminated.

CONCLUSIONS

The purpose of this study was to examine the influence of contextual factors andnontechnical skills on threat and error management during normal flight opera-tions. In the study, I employed new operational performance evaluation methodol-ogies to gather a range of data on the strengths and weaknesses of crew perfor-mance within two fleets of a single airline. The predictive models of threat anderror management provide significant empirical support for the importance of non-technical skills in minimization of risk and the overall enhancement of operationalperformance. Further, the findings of this research stress the importance of takinginto consideration contextual factors such as crew experience, operational pres-sures, and team structure in the industry’s understanding of effective threat and er-ror management. The results of this study demonstrate the ways in which this typeof data analysis can highlight the strengths and weaknesses of operational perfor-mance and suggest that this type of performance evaluation can offer individual or-ganizations invaluable information for enhanced training system design throughthe development of scenario-based training.

ACKNOWLEDGMENTS

I acknowledge and sincerely thank the members of the Human Factors ResearchProject at the University of Texas and the LOSA Collaborative. In particular,thanks go to Robert Helmreich, James Klinect, John Wilhelm, and Patrick Murraywho offered considerable support for the use of their LOSA methodology. Furtherthanks and acknowledgment go to both the Line Operations and Training divisionsof the airline involved in this study. The airline deserves commendation for itscommitment to operational improvement and enhancement of its safety throughthe trial of this new methodology. The airline approached the project with enthusi-asm and the project could never have been realized without the airline’s willing-ness to open its line operations to analysis and evaluation. The preliminary datacollection for this study was undertaken within the School of Aviation at MasseyUniversity. Sincere thanks are extended to Graham Hunt for his support for the ini-tial project.


REFERENCES

Bowers, C. A., Jentsch, F., Salas, E., & Braun, C. C. (1998). Analyzing communication sequences forteam training needs assessment. Human Factors, 40, 672–679.

Cizek, G. J., & Fitzgerald, S. M. (1999). An introduction to logistic regression. Measurement and Eval-uation in Counseling and Development, 31, 223–240.

Dismukes, R. K., McDonnell, L. K., & Jobe, K. K. (2000). Facilitating LOFT debriefings: Instructortechniques and crew participation. International Journal of Aviation Psychology, 10, 35–57.

Endsley, M. R. (1993). A survey of situation awareness requirements in air-to-air combat fighters. In-ternational Journal of Aviation Psychology, 3, 157–168.

Fischer, U., & Orasanu, J. (1999). Say it again, Sam! Effective communication strategies to mitigate pi-lot error. In R. S. Jensen (Ed.), Proceedings of the Tenth International Symposium on Aviation Psy-chology (CD-Rom). Columbus: Ohio State University.

Flin, R., Goeters, K.-M., Hörmann, H.-J., & Martin, L. (1998, September). A generic structure ofnon-technical skills for training and assessment. Paper presented at the 23rd Conference of the Euro-pean Association for Aviation Psychology, Vienna, Australia.

Gelfand, M. J. (2000). Cross-cultural industrial and organisational psychology: Introduction to the spe-cial issue. Applied Psychology, 49, 29–31.

Goldsmith, T. E., & Johnson, P. J. (2002). Assessing and improving evaluation of aircrew performance.International Journal of Aviation Psychology, 12, 223–240.

Helmreich, R. L. (2000). On error management: Lessons from aviation. British Medical Journal, 320,781–785.

Helmreich, R. L., Klinect, J. R., & Wilhelm, J. A. (1999). Models of threat, error and CRM in flight op-erations. In R. S. Jensen (Ed.), Proceedings of the Tenth International Symposium on Aviation Psy-chology (pp. 677–682). Columbus: Ohio State University.

Helmreich, R. L., & Merritt, A. C. (2000). Safety and error management: The role of crew resourcemanagement. In B. J. Hayward & A. R. Lowe (Eds.), Aviation resource management (pp. 107–119).Aldershot, England: Ashgate.

Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The evolution of crew resource management.International Journal of Aviation Psychology, 9, 19–32.

Helmreich, R. L., & Wilhelm, J. A. (1991). Outcomes of crew resource management training. Interna-tional Journal of Aviation Psychology, 1, 287–300.

Helmreich, R. L., Wilhelm, J. A., Klinect, J. R., & Merritt, A. C. (2001). Culture, error and crew re-source management. In E. Salas, C. A. Bowers, & E. Edens (Eds.), Improving teamwork in organiza-tions (pp. 305–331). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Hörmann, H.-J. (2001). Cultural variation of perceptions of crew behaviour in multi-pilot aircraft. LeTravail Humain, 64, 247–268.

Johnston, N. (1997). Teaching human factors for airline operations. In G. J. F. Hunt (Ed.), Designing in-struction for human factors training in aviation (pp. 127–160). Aldershot, England: Ashgate.

Kline, T. J. B., Sulsky, L. M., & Rever-Moriyama, S. D. (2000). Common method variance and specifi-cation errors: A practical approach to detection. The Journal of Psychology, 134, 401–421.

Klinect, J. R., Wilhelm, J. A., & Helmreich, R. L. (1999). Threat and error management: Data from lineoperations safety audits. In R. S. Jensen (Ed.), Proceedings of the Tenth International Symposium onAviation Psychology (pp. 683–688). Columbus: Ohio State University.

Lauber, J. K. (1987). Cockpit resource management: Background and overview. In H. W. Orlady & H.C. Foushee (Eds.), Cockpit resource management training: Proceedings of the NASA/MAC work-shop (pp. 5–14). Moffett Field, CA: NASA-Ames Research Center.

Neumeister, D. M., & Mangold, S. M. (1997). Implementing the model AQP database: Lessonslearned. In R. S. Jensen & L. Rakovan (Eds.), Proceedings of the Ninth International Symposium onAviation Psychology (pp. 70–75). Columbus: Ohio State University.

230 THOMAS

Oser, R. L., Cannon-Bowers, J. A., Salas, E., & Dwyer, D. J. (1999). Enhancing human performance intechnology-rich environments: Guidelines for scenario-based training. In E. Salas (Ed.), Hu-man/Technology interaction in complex systems (Vol. 9, pp. 175–202). Stamford, CT: JAI.

Peng, C. J., Lee, K. L., & Ingersoll, G. M. (2002). An introduction to logistic regression analysis and re-porting. Journal of Educational Research, 96, 3–14.

Reason, J. (1990). Human error. Cambridge, England: Cambridge University Press.Reason, J. (1997). Managing the risks of organizational accidents. Aldershot, England: Ashgate.Salas, E., Rhodenizer, L., & Bowers, C. A. (2000). The design and delivery of crew resource manage-

ment training: Exploiting available resources. Human Factors, 42, 490–511.Seamster, T. L., Redding, R. E., & Kaempf, G. L. (1997). Applied cognitive task analysis in aviation.

Aldershot, England: Ashgate.Taggart, W. R. (1994). Crew resource management: Achieving enhanced flight operations. In N.

Johnston, N. McDonald, & R. Fuller (Eds.), Aviation psychology in practice (pp. 309–339).Aldershot, England: Ashgate.

Thomas, M. J. W. (2003). Improving organisational safety through the integrated evaluation of opera-tional and training performance: An adaptation of the line operations safety audit (LOSA) methodol-ogy. Human Factors and Aerospace Safety, 3, 25–45.

Trollip, S. R. (1995). Issues in teaching human factors efficiently. In N. Johnston, R. Fuller, & N. Mc-Donald (Eds.), Aviation psychology: Training and selection (pp. 252–257). Aldershot, England:Ashgate.

Wiener, E. L., Kanki, B. G., & Helmreich, R. L. (Eds.). (1993). Cockpit resource management. SanDiego, CA: Academic.

Manuscript first received February 2003


Documents

Predictors of Threat and Error Management: Identification ... · PDF filebeen reconceptualized explicitly as the development of threat and error counter-measures (Helmreich, Merritt,