June 2002

NASA/CR—2002–211406

Activity Tracking for Pilot Error Detection fromFlight DataTodd J. CallantineSan Jose State University, San Jose, California

The NASA STI Program Office ... in Profile

Since its founding, NASA has been dedicated tothe advancement of aeronautics and spacescience. The NASA Scientific and TechnicalInformation (STI) Program Office plays a keypart in helping NASA maintain this importantrole.

The NASA STI Program Office is operated byLangley Research Center, the lead center forNASA’s scientific and technical information.The NASA STI Program Office provides accessto the NASA STI Database, the largestcollection of aeronautical and space science STIin the world. The Program Office is alsoNASA’s institutional mechanism fordisseminating the results of its research anddevelopment activities. These results arepublished by NASA in the NASA STI ReportSeries, which includes the following reporttypes:

• TECHNICAL PUBLICATION. Reports ofcompleted research or a major significantphase of research that present the resultsof NASA programs and include extensivedata or theoretical analysis. Includescompilations of significant scientific andtechnical data and information deemed tobe of continuing reference value. NASAcounterpart of peer-reviewed formalprofessional papers, but having lessstringent limitations on manuscript lengthand extent of graphic presentations.

• TECHNICAL MEMORANDUM.Scientific and technical findings that arepreliminary or of specialized interest, e.g.,quick release reports, working papers, andbibliographies that contain minimalannotation. Does not contain extensiveanalysis.

• CONTRACTOR REPORT. Scientific andtechnical findings by NASA-sponsoredcontractors and grantees.

• CONFERENCE PUBLICATION.Collected papers from scientific andtechnical conferences, symposia,seminars, or other meetings sponsoredor co-sponsored by NASA.

• SPECIAL PUBLICATION. Scientific,technical, or historical information fromNASA programs, projects, and missions,often concerned with subjects havingsubstantial public interest.

• TECHNICAL TRANSLATION.English-language translations of foreignscientific and technical materialpertinent to NASA’s mission.

Specialized services that complement the STIProgram Office’s diverse offerings includecreating custom thesauri, building customizeddatabases, organizing and publishing researchresults ... even providing videos.

For more information about the NASA STIProgram Office, see the following:

• Access the NASA STI Program HomePage at http://www.sti.nasa.gov

• E-mail your question via the Internet tohelp@sti.nasa.gov

• Fax your question to the NASA STIHelp Desk at (301) 621-0134

• Telephone the NASA STI Help Desk at(301) 621-0390

• Write to: NASA STI Help Desk NASA Center for AeroSpace Information 7121 Standard Drive Hanover, MD 21076-1320

NASA/CR—2002–211406

Activity Tracking for Pilot Error Detection fromFlight DataTodd J. CallantineSan Jose State University, San Jose, California

National Aeronautics andSpace Administration

Ames Research CenterMoffett Field, California 94035

Acknowledgments

This work was funded under the System Wide Accident Prevention element of the FAA/NASAAviation Safety Program. Thanks to the NASA Langley B757 Flight Test team for their assistancewith data collection.

Available from:

NASA Center for AeroSpace Information National Technical Information Service7121 Standard Drive 5285 Port Royal RoadHanover, MD 21076-1320 Springfield, VA 22161301-621-0390 703-605-6000

IntroductionProblems associated with human error have longbeen recognized (e.g., Babbage, 1961). Morerecently, Perrow (1984) characterized how highsystem complexity contributes to accidents and,together with the introduction of ‘glass cockpit’aircraft (e.g., Wiener and Curry, 1980),invigorated interest in human error. Reason’s(1990) theoretical treatment marks the beginningof human error research on several fronts. Onesort is devoted to the collection and analysis ofdata on human error and its effects. For example,Johnson (1998) investigates methods foranalyzing temporal features of incidents, as wellas new ways to report incidents (Johnson, 2000).Accessing information in a large database ofincident reports has in turn led to research onadvanced search tools (McGreevy, 2001).

Another research area focuses on formal methodsthat can help reveal potential error-relatedproblems during the design process. For example,Degani and Heymann (2000) use formalspecifications of system behavior to identifyunsafe interface abstractions. Sherry et al. (2001)use a formal system model to explain howoperators misunderstand a system, and how itmight be redesigned. Formal task representationsalso enable scrutiny of human-error tolerance(Wright, Fields, and Harrison, 1994) and temporalaspects of operator, system, and environmentalbehavior (Fields, Wright, and Harrison, 1996). Amethodology for analyzing the potential forhuman error during the design process alsoincorporates some of these ideas (Fields, Harrison,and Wright, 1997). Johnson (2001) examineshow error reporting can be used to supportsystem refinement in the initial stages ofimplementation when the design is still in flux.

Models of human operators anchor twoadditional areas of research. One usesengineering-oriented computational models asthe basis for preventing error and improvingerror recovery by training and later aiding theoperator (e.g., Mitchell, 2000). Another researcharea seeks to develop models, either theoretical(e.g., Busse and Johnson, 1998) or computational

(e.g., Byrne and Bovair, 1997), that can elucidatethe cognitive bases of human error.

This report describes an application of the CrewActivity Tracking System (CATS) that couldcontribute to future efforts to reduce flight crewerrors. It demonstrates how CATS tracks crewactivities to detect errors, given flight data and airtraffic control (ATC) clearances (alreadyprovided, in some cases, by digital data linkcommunication technology, e.g., Smith, Brown,Polson, and Moses, 2001). CATS implements aso-called ‘intent inference’ technology, calledactivity tracking, in which it uses a computational‘engineering’ model of the operator’s task,together with a representation of the currentoperational context, to predict nominallypreferred operator activities and interpret actualoperator actions.

CATS, too, has its roots in glass cockpit aircraftautomation research. It was originallyimplemented to track the activities of Boeing 757pilots, with a focus on automation mode errors(Callantine and Mitchell, 1994). The CATSactivity tracking methodology was validated as asource of real-time knowledge to support a pilottraining/aiding system (Callantine, Mitchell, andPalmer, 1999). CATS is useful as an analysis toolfor assessing how operators use proceduresdeveloped to support new operational concepts(Callantine, 2000a, 2000b). It also serves as aframework for developing agents to representhuman operators in incident analyses anddistributed simulations of new operationalconcepts (Callantine, 2001a).

The research described here draws in large partfrom these earlier efforts. In particular, the CATSmodel of B757 flight crew activities has beenexpanded and refined. The representation ofoperational context used to reference the modelto predict nominally preferred activities hassimilarly undergone progressive refinement. And,while the idea of using CATS to detect flight crewerrors from flight data is not new, this reportpresents an example of CATS detecting agenuine, in-flight crew error from actual aircraftflight data.

2

Using CATS to detect errors from flight data hasseveral potential benefits (Callantine, 2001b).First, CATS provides information aboutprocedural errors that do not necessarily result indeviations, and therefore would not otherwise bereported. Second, CATS enables airline safetymanagers to ‘automatically’ incorporateinformation about a detected error into a CATS-based training curriculum. Other pilots could‘relive’ a high-fidelity version of the context inwhich another crew erred. Increasing theefficiency and fidelity of information transferabout errors to the pilot workforce in this waywould likely yield safety benefits.

It is important to note that flight crews need notview such an application as punitive. It isincumbent on airline safety and trainingmanagers to ensure that the CATS model used todetect errors exactly matches the trainingprovided to flight crews. Research indicates thatmuch of what pilots know about some autopilotfunctionality currently is not formally trained(Mitchell, 2000). Thus, a safety-enhancementprogram that uses CATS to detect errors wouldimprove training by requiring safety and trainingmanagers to explicate policies about how anaircraft should preferably be flown.

The report is organized as follows. It firstdescribes the CATS activity trackingmethodology, and information flow in CATS.The report then describes the CATSimplementation for detecting pilot errors. It firstdescribes flight data obtained for thisdemonstration from the NASA Langley Boeing757 (B757) Airborne Research IntegratedExperiment System (ARIES) aircraft. It nextdescribes two key representations. The first is aportion of a CATS model of B757 flightoperations. The second is a representation of theconstraints conveyed by ATC clearances thatplays a key role in representing the currentoperational context (Callantine, 2002). Anexample from the available flight data thenillustrates CATS detecting pilot errors. The reportconcludes with a discussion of future researchchallenges.

Activity TrackingActivity tracking is not merely the detection ofoperational ‘deviations’. The activity trackingmethodology involves first predicting the set ofexpected nominal operator activities for thecurrent operational context, then comparingactual operator actions to these expectations toensure operators performed correct activities. Insome situations, various methods or techniquesmay be acceptable; therefore the methodologyalso includes a mechanism for determining that,although operator actions do not matchexpectations exactly, the actions are nonethelesscorrect. In this sense, CATS is designed to ‘track’flight crew activities in real time and ‘understand’that they are error-free. As the example belowillustrates, ‘errors’ CATS detects include thosethat operators themselves detect and rapidlycorrect; such errors may nonetheless be useful toexamine.

In addition to parameters that define the state ofthe controlled system, activity tracking alsorequires data about the dynamic set of constraintson controlled system behavior, as well as dataabout actual operator actions. For flight deckapplications, constraint data in the form of datalinked ATC clearance information will likely bewidely available in the near future, as notedabove, but a number of legal issues impede therelease of pilot action data (U.S. GAO, 1998).This report takes the view that the promise ofsignificant safety benefits, together withanonymity provisions similar to those of theAviation Safety Reporting System (ASRS), canhelp overcome these issues in the future. Activitytracking also requires a valid model of nominallycorrect operator activities suitable for deriving theset of ‘preferred’ operator actions predicted(expected according to the nominal model) for agiven operational context. For the flight deck,such models may be adapted from extantAdvanced Qualification Program (AQP) models(U.S. FAA, 1995) and validated in high fidelitysimulations. (The original CATS B757 model,however, was initially derived from a trainingprogram at a major airline, together with expertinput from line pilots.)

3

CATS identifies two types of errors: errors ofomission, and errors of commission. It furtheridentifies errors of commission that result whenthe ‘right action’ is performed with the ‘wrongvalue.’ CATS does not base these determinationson a ‘formulaic’ representation of how sucherrors would appear in a trace of operatoractivities, nor attempt to further classify errors(e.g., ‘reversals’) as in some research on formalmethods for identifying potential errors (Wright,Fields, and Harrison, 1994). Indeed, this would bedifficult, given that the CATS model does notrepresent the ‘steps’ of procedures explicitly as‘step A follows step B;’ instead it representsprocedures implicitly by explicitly specifying theconditions under which operators shouldpreferably perform each action. CATS predictsconcurrent actions whenever the current contextsatisfies conditions for performing two or moreactivities. CATS interprets concurrent actionswhenever the granularity of action data identifiesthem as such.

Like analysis techniques that rely on a‘reflection’ of the task specification in a formalmodel of a system (Degani and Heymann, 2000,

Sherry et al., 2001), CATS relies on a correctlyfunctioning system to reflect the results of actions(or inaction) in its state. CATS identifies errors byusing information in the CATS model thatenables it to assess actions (or the lack thereof, inthe case of omissions) in light of the currentoperational context and the future contextformed as a result of operator action (orinaction). Thus, one might view the CATS errordetection scheme as ‘closing the loop’ between arepresentation of correct task performance andthe controlled system, and evaluating feedbackfrom the controlled system to ensure it ‘jibes’with correct operator activities. Given that thesystem is operating normally and providing‘good data,’ this is a powerful concept.

Crew Activity Tracking System (CATS)CATS implements a methodology for activitytracking in a computer-based system that hasbeen validated to work in real time (Callantine,Mitchell, and Palmer, 1999). Figure 1 genericallydepicts information flow in CATS, between acontrolled system and CATS, and between CATSand applications based on it. As described above,CATS uses representations of the current state ofthe controlled system and constraints imposed by

StateControlled System

Context

Activity Model

Constraints

Actions

Predictions

Interpretations

Integrated Aid/

Training System

CATS

Analysis Tool

Operator(s)

Environment

Figure 1. Information flow within and between CATS and a generic human-machine system, and applications to error analysis, aiding, and training.

4

the environment (including performance limitson the controlled system) to derive the currentoperational context. CATS then uses thisrepresentation to generate predictions from itsmodel of operator activities. CATS comparesdetected operator actions to its predicted activities,and it assesses actions that it cannot immediatelyinterpret as matching a prediction by periodicallyreferencing the activity model until it receivesenough new context information to disambiguatepossible interpretations.

Thus, two threads comprise the activity trackingmethodology as implemented in CATS: a‘prediction thread’ responsible for generating thecontext information necessary to predict nominalactivities, and an ‘interpretation thread’ thatinterprets operator actions. Displays of theresulting interpretations—together with displaysfor visualizing the input data, current operationalcontext, and activity model—comprise a CATS-based analysis tool (Callantine, 2000a, 200b).Predictions and interpretations supply theinformation necessary for an aid that is integratedinto the displays of the controlled system(Callantine, 1999) or, in the case of a tutoringsystem, a high-fidelity simulation of thecontrolled system.

CATS Implementation for FlightData Error DetectionThe following subsections specifically describethe implementation of CATS for detecting piloterrors from flight data. The first is devoted to theflight data itself. The second illustrates a portionof the CATS model, and the third describes howCATS generates the current operational contextusing a representation of ATC clearanceconstraints. The CATS model fragment is relevantto an example of CATS detecting pilot errorspresented in the fourth subsection.

The following subsections all assume someknowledge of commercial aviation and a B757-style autoflight system. The basic scheme is thatpilots first program the flight plan into the FMSvia the CDU. After engaging the autopilot (orflight director) and the autothrottles, they interact

with aircraft’s Mode Control Panel (MCP), settingtactical targets and engaging pitch, roll, and thrustmodes as required to comply with air trafficcontrol clearances. High-level modes such asLateral Navigation (LNAV) and VerticalNavigation (VNAV) track the FMS-programmedplan; other modes, such Flight Level Change (FLCH), achieve a tactical target state (the MCP targetaltitude, in the case of FL CH). A detaileddescription of the Boeing 757 autoflight systemmode usage is provided in Callantine, Mitchell,and Palmer (1999); see Billings (1997), Sarterand Woods (1995), and Wiener (1989) fordiscussions of mode errors and automation issues.

B757 ARIES Flight DataThe NASA Langley B757 ARIES aircraft, with itsonboard Data Acquisition System (DAS),provided the flight data for this research (Figure2). The DAS collects data at rates in excess of 5Hz, using onboard computers that perform sensordata fusion and integrity checking. In futureapplications such functionality may be requiredwithin CATS. Table 1 shows the collection ofvalues that comprise the data set. The data includeinformation from important cockpit systems. Therightmost column of Table 1 shows data CATSderives from the sampled values using filteringtechniques. Included are crew action events CATSderives from the values of control states. Targetvalue settings on the MCP are derived with‘begin’ and ‘end’ values, as in formal actionspecification schemes (Wright, Fields, andHarrison, 1996). Like the initial CATS research(Callantine and Mitchell, 1994), this applicationfocuses on interactions with the autoflight systemMCP, so it only uses some of the available data.

Absent from data in Table 1 are important flightmanagement system (FMS) data, includingactions pilots perform using the flightmanagement computer (FMC) control anddisplay units (CDUs). This is a shortcoming ofthe B757 ARIES DAS that future research seeksto rectify. In the interim, tracking CDUinteractions with CATS is feasible with the NASAAmes Advanced Concepts Flight Simulator(ACFS), a full-motion, high-fidelity glass cockpitsimulator (Callantine, 2000), and its desktop

5

counterpart, the ‘miniACFS.’ To detect entriesthat a pilot types into the CDU scratchpad, CATSuses a parsing mechanism. It detects CDUkeystrokes and ‘releases’ a fully-formed action(e.g., ‘crossing restriction entered’) when thecharacter string created exactly matches a valuethat CATS expects, or when it can be determinednot to match any related value. Thus, unlikeCATS in general, this parsing process doesincorporate a rudimentary a priori model of whatsorts of errors a pilot might make. This is an areaof further research. Also absent from Table 1 aredata concerning ATC clearances. For the presentapplication, cockpit observations provide requiredclearance information.

CATS Model of B757 Navigation ActivitiesFigure 3 depicts a fragment of the CATS modelused to detect errors from B757 ARIES data. Themodel decomposes the highest level activity, ‘flyglass cockpit aircraft,’ into sub-activities as

necessary down to the level of pilot actions.Figure 3 illustrates eight actions. All actionsderivable from the data are included in the fullmodel. Each activity in the model is representedwith conditions that express the context underwhich the activity is nominally preferred, givenpolicies and procedures governing operation ofthe controlled system. The parenthesizednumbers in Figure 3 refer to Table 2, which liststhe ‘and-or trees’ that comprise these rules.

For comparison to other work that considershuman errors involved with CDU manipulations(e.g., Fields, Harrison, and Wright, 1997), themodel fragment in Figure 3 shows just one ofnumerous FMS configuration tasks. Note that aCATS model can also include cognitive, verbal,and perceptual ‘activities,’ but CATS can onlypredict, not interpret, activities for which noconfirmatory data exists. Thus, such activities arenot relevant to the present application.

Figure 2. Data Acquisition System (DAS) onboard the NASA B757 ARIES aircraft (inset).

6

Table 1. B757 ARIES data used in the present research, including derived states and action events (rightmost column). Some variablesappear multiple times, because the B757 ARIES DAS collects them from multiple sources.

Time variablestimetime1time2time3Environmental informationtotal_air_temptrue_wind_dirwind_speedAC position/attitudebaro_altbaro_corrflight_path_angleground_speedcomputed_airspeedcalibrated_airspeedmachmagnetic_headingmagnetic_track_anglepitch_angleradio_altituderoll_angletrue_track_angleiru_potential_vert_speedhybrid_lathybrid_lonAC configuration/controlsleft_engine_eprright_engine_eprflap_posspeed_brake_handleleft_throttle_posright_throttle_posgross_weightMCP target valuessel_mcp_altitudesel_mcp_headingsel_mcp_speedsel_mcp_vert_speedmcp_flare_retard_ratesel_mcp_machMCP bank angle settingsbank_angle_lim_flaps_25bank_angle_lim_flaps_15bank_angle_lim_auto

NAV/COMM datadme_rangeleft_dme_freqright_dme_freqleft_dme_distright_dme_distleft_vhf_freqright_vhf_freqFMC datafmc_target_airspeedfmc_selected_altitudefmc_selected_airspeedfmc_selected_machfmc_crz_altitudefmc_etafmc_desired_trackfmc_wpt_bearingfmc_cross_track_distfmc_vert_devfmc_range_to_altfmc_wide_vert_devAFDS statesap_cmd_ctr_engdap_cmd_cen_gc_huhap_cmd_cen_gr_huhleft_ap_cmd_engdap_cmd_left_engdright_ap_cmd_engdap_cmd_right_engdap_cmd_center_engdap_cws_center_engdap_cws_left_engdap_cws_right_engdap_in_controlfd_c_onfd_fo_onfd_on_cfd_on_foAFDS switchesap_cmd_center_reqdap_cmd_right_reqdap_cws_center_reqdap_cws_left_reqdap_cws_right_reqdap_cmd_left_reqd

AFDS modesfl_ch_engdhdg_hold_engdhdg_sel_engdland_2_greenland_3_greenalt_hold_engdvnav_armed_engdlnav_armed_engdspeed_mode_engdthrust_mode_engdloc_engdvert_spd_engdapprch_armed_engdloc_armed_engdback_course_armed_engdglideslope_engdMCP Speed display statusmcp_speed_display_blankAutothrottleat_armedMCP switcheshdg_sel_reqdhdg_hold_reqdlnav_reqdvnav_reqdspd_reqdapprch_reqdloc_reqdalt_hold_reqdvs_mode_reqdfl_ch_reqdthrust_mod_reqdIAS/Mach togglemach_toggledCrew Alert levelscrew_alert_level_acrew_alert_level_bcrew_alert_level_cStatus dataeec_validengine_not_out

FMC-A/T internal datafmc_at_mach_mode_reqdfmc_at_airspeed_mode_reqdfmc_active_climbfmc_climb_mode_reqdfmc_active_cruisefmc_con_mode_reqdfmc_crz_mode_reqdfmc_active_descentfmc_display_annunc_onfmc_eng_ident_1fmc_eng_ident_2fmc_eng_ident_3fmc_eng_ident_4fmc_eng_ident_5fmc_eng_ident_6fmc_eng_ident_7fmc_eng_ident_8fmc_eng_ident_9fmc_eng_ident_10fmc_ga_mode_reqdfmc_idle_thr_reqdfmc_msg_annunciatedthrottle_retard_reqdpitch_speed_control_engdvnav_operationallnav_operationaltmc_validVNAV submodesfmc_vnav_speed_operationalfmc_vnav_path_operationalfmc_vnav_alt_operationalThrust ratingsfmc_rating_1_reqdfmc_rating_2_reqdfmc_offset_annunciatedfmc_throttle_dormant_reqdfmc_thr_mode_reqdfmc_to_mode_reqdreq_1_valid_resvreq_2_valid_resv

Derived statesvert_speedalt_cap_engagedspd_win_auto_chngap_cmd_engdDerived MCP actionsset MCP hdgset MCP altset MCP spdset MCP machset MCP vshdg sel presshdg hold presslnav pressvnav pressspd pressapprch pressloc pressalt hold pressvs mode pressfl ch pressthrust mode pressmach toggledc ap cmd switch pressl ap cmd switch pressr ap cmd switch pressarm autothrottlesOther derived actionstune left VHFtune right VHFset flapsset spoilers

7

Fly

gla

ssco

ckp

ita

ircra

ft (

1)

Pe

rfo

rmta

keo

ff r

oll

(2)

Co

nfig

ure

airc

raft

(3

)

Co

nfig

ure

au

toth

rott

les

(4)

Co

nfig

ure

flig

ht

dir

ect

or

gu

ida

nce

(5

)

Co

nfig

ure

au

top

ilot (

6)

Co

nfig

ure

flig

ht

ma

na

ge

me

nt

syst

em

(7)

Na

vig

ate

with

AP

or

FD

gu

ida

nce

(8

)

Exe

cute

ro

ute

mo

difi

catio

n(s

) (1

2)

Pu

sh E

XE

C k

ey

(3

3)

Pe

rfo

rma

pp

roa

ch (

10

)

Co

nfig

ure

cro

ssin

gre

stri

ctio

n[D

ime

nsi

on

=V

ER

T]

(11

)

Acc

ess

CD

UL

EG

S p

ag

e (

15

)P

ush

CD

UL

EG

S k

ey

(30

)

Ad

d t

o c

ross

ing

fix (

17)

En

ter

cro

ssin

gre

stric

tion

(1

6)

Typ

e c

ross

ing

rest

rict

ion

va

lue

insc

ratc

hp

ad

(3

1)

Lin

e s

ele

ctcr

oss

ing

fix

(3

2)

Na

vig

ate

La

tera

lly[D

ime

nsi

on

=L

AT

] (1

3)

Na

vig

ate

Ve

rtic

ally

[Dim

en

sio

n =

VE

RT

] (

14

)

Co

nfig

ure

com

mu

nic

atio

ns

(9)

Se

t ta

rge

ta

ltitu

de

(1

8)

Dia

l MC

P a

ltitu

de

kno

b (

34

)

Ach

ieve

/m

ain

tain

alti

tud

e (

19

)

Fly

usi

ng

FL

CH

(2

2)

En

ga

ge

FL

CH

[m

od

e =

FL

CH

] (2

7)

Pu

sh M

CP

FL

CH

sw

itch

(35

)

Fly

usi

ng

V/S

(23

)

Ma

na

ge

sp

ee

d[D

ime

nsi

on

=S

PD

] (2

4)

Ad

just

spe

ed

(28

)

Dia

l MC

Psp

ee

d k

no

b(3

6)

Ho

ld a

ltitu

de

(2

0)

Fly

pro

file

(2

1)

Fly

usi

ng

VN

AV

(2

5)

En

ga

ge

VN

AV

[mo

de

=V

NA

V]

(29

)

Pu

sh M

CP

VN

AV

sw

itch

(37

)

Ma

na

ge

sp

ee

d[D

ime

nsi

on

=S

PD

] (2

6)

Figu

re 3

. Fra

gmen

t of

CA

TS

mod

el f

or B

757

oper

atio

ns.

8

Table 2. ‘And-or’ trees of conditions under which the CATS model in Figure 3 represents activities as‘nominally preferred.’ CATS predicts an activity when its conditions, plus all the conditions of its parentactivities, are satisfied by the current operational context.

(1) start-of-run

(2) (not above-runway-elevation)

(3) (and (not above-clean-speed) (not flight-surfaces-within-limits) (not gear-within-limits) )

(4) (not autothrottle-armed)

(5) (not flight-director-on)

(6) [ (and (not autopilot-cmd-mode-engaged) above-1000-feet-AGL ) ]

(7) (or (not programmed-route-within-limits) route-uplink-received )

(8) (and above-1000-feet-AGL (or autopilot-cmd-mode-engaged flight-director-on) )

(9) (not comm-frequency-within-limits)

(10) (or approaching-glideslope-intercept-point approach-localizer-intercept-point)

(11) (not crossing-restriction-within-limits)

(12) route-modifications-within-limits

(13) (or autopilot-cmd-mode-engaged flight-director-on)

(14) (or autopilot-cmd-mode-engaged flight-director-on)

(15) (not cdu-page-LEGS)

(16) (and cdu-page-LEGS (not crossing-restriction-built) )

(17) (and cdu-page-LEGS crossing-restriction-built)

(18) (not mcp-altitude-within-limits)

(19) (or (and (not current-altitude-within-limits) (not profile-within-limits-for-now) ) expedite-needed )

(20) (and current-altitude-within-limits (not profile-within-limits-for-now ) )

(21) profile-within-limits-for-now

(22) (or (not altitude-close-to-target) expedite-needed)

(23) altitude-close-to-target

(24) (or fl-ch-engaged vs-engaged)

(25) profile-within-limits-for-now

(26) vnav-engaged

(27) (not fl-ch-engaged)

(28) (not target-speed-within-limits)

(29) (and (not vnav-engaged) (not capturing-required-altitude) )

(30) (not cdu-page-LEGS)

(31) (not crossing-restriction-built)

(32) crossing-restriction-built

(33) route-modifications-within-limits

(34) (not mcp-altitude-within-limits)

(35) mcp-altitude-within-limits

(36) (not target-speed-within-limits)

(37) mcp-altitude-within-limits

9

Several important features of the CATS modeldeserve mention. First, when using the model topredict the currently preferred set of activities,CATS searches the model top-down, so thathigher level activities ‘subsume’ their children(i.e., the conditions on an activity must be metbefore CATS can predict any of its children).Thus, CATS makes predictions andinterpretations at every level of abstraction themodel represents. Second, the model itself is‘memoryless.’ Given some context, CATS canpredict what the operators need to do (asdiscussed below, however, historic information iscontained in the context, and in some cases doesimpact the CATS predictions). Third, the modelcan be structured to represent the activities of anindividual operator, or a team of operators (cf.Fields, Harrison, and Wright, 1997); either way,CATS is capable of detecting errors that relate toassigned operator roles and responsibilities.Fourth, the model contains information—beyondthat provided by its structure and preferenceconditions—to support error detection. One typeof information concerns the automation modethat should be operational if a mode-engagementaction was successfully invoked. Anotherconcerns the ‘dimension’ of the operationalcontext that an activity addresses.

Representation of ATC ClearanceConstraints for Context GenerationEnvironmental constraints play a key role indefining the goals that shape worker behavior incomplex sociotechnical systems (Vicente, 1999).CATS also relies on a representation ofenvironmental constraints to construct arepresentation of the current operational context(Figure 1). These factors motivated recentresearch on a symbolic representation of theconstraints ATC clearances impose on flightoperations (Callantine, 2002). Figure 4 shows therepresentation, which represents three keydimensions of constraints: vertical, lateral, andspeed. CATS employs a rule base that enables itmodify this constraint representation to reflect theconstraints imposed (or removed) by each newATC clearance.

As discussed in Callantine (2002), CATS definescontext from a human operator’s perspective tobe the situation plus any activities the operator isengaged in performing. The situation is definedas the system’s state, together with environmentalconstraints and all salient relationships betweenthe state and constraints. Each of these elements isadditionally considered to have historic, current,and planned (or predicted) future components.States and constraints are also decomposedhierarchically at multiple levels of abstraction asnecessary.

CATS uses a representation of context of thisform to generate a summary of the currentoperational context suitable for evaluating theconditions under which activities are preferred, inorder to predict activities, and for determiningwhether an operator action it did not expect is inerror. Whenever the state or constraints change,CATS examines the salient relationships togenerate a set of ‘context specifiers’ thatsummarizes the current operational context; theseare the descriptive clauses that appear in theconditions listed in Table 2. CATS also uses thesymbolic constraint representation to maintain arecord of compliance with constraints. This isimportant not only for context generation, butalso for logging flight path deviations.

Error Detection ExampleThe report now presents an example of CATSdetecting errors from B757 ARIES flight datacollected during recent flight test activities.Although the data are real, in the flight testenvironment, strict procedures about how thepilots should preferably fly the airplane areunreasonable. Nonetheless, by imposing themodel depicted in part in Figure 3, CATS wasable to detect errors, and the errors were notcontrived. While the errors CATS detects areinsignificant, because they in no waycompromised safety, the exercise nonethelessdemonstrates the viability of CATS for errordetection. It should be noted that, in thisapplication, as the following ‘snapshots’ show,CATS runs at between twelve and twenty-twotimes real time.

10

Figure 5 shows the CATS interface at the start ofthe scenario (Scenario Frame 1). The crew hasjust received a clearance to “climb and maintain16,000 feet.” CATS modifies its representationof ATC clearance constraints accordingly, andusing the updated context, predicts that the crewshould set the new target altitude on the MCP bydialing the MCP altitude knob.

In Scenario Frame 2 (Figure 6), a pilot insteadpushes the VNAV switch. Because CATS has notpredicted this action, it cannot interpret the actioninitially. CATS instead continues processing data.

In Scenario Frame 3 (Figure 7), CATS hasreceived enough new data to interpret the VNAVswitch press action. Had the action been correct,the autoflight system state would have reflectedthis by engaging the VNAV mode andcommencing the climb. However, VNAV will not

engage until a new target altitude is set. To assessthe VNAV switch press with regard to the currentcontext, in which airplane is still in ALT HOLDmode at 12,000 feet, CATS searches its model todetermine if any parent activities of the VNAVswitch press contain information linking theaction to a specific context. CATS finds that the‘engage VNAV’ activity should reflect VNAVmode engagement in the current context (seeFigure 3). Because this is not the case, CATS flagsthe VNAV switch press as an error. Meanwhile,CATS still expects the crew to dial the MCPaltitude knob.

In Scenario Frame 4 (Figure 8), a pilot doesbegin setting the MCP altitude. CATS interpretsthis action as matching a current prediction, butwith an incorrect value, as the altitude setting hasnot yet reached 16,000.

Figure 4. Snapshot of a CATS representation of environmental constraints constructed from thefiled flight plan, and modified according to constraints conveyed by ATC clearances.

11

Figure 5 (Scenario Frame 1). In response to a clearance to climb, CATS predicts the crew should setthe new target altitude on the MCP by dialing the MCP altitude knob._

Figure 6 (Scenario Frame 2). CATS detects that a crew member pressed the VNAV switch instead.

12

CATS does not flag this action as a ‘wrong value’error, however, because it is only the start of thealtitude setting. CATS continues to predict ‘dialMCP altitude knob’ because the context specifier‘mcp-altitude-within-limits’ is not generatedwhen the current MCP target altitude is comparedto the value specified by the representation ofATC constraints (see Figure 3 and Table 2).

In Scenario Frame 5 (Figure 9), one pilot pushesthe VNAV switch a second time before thealtitude setting is complete. As the other pilotcompletes the altitude setting, CATS interprets theend of the altitude setting action as matching itsprediction.In Scenario Frame 6 (Figure 10), CATS detectsthat a pilot has pressed the FL CH switch (perhapsto begin the climb in FL CH mode, since VNAVdid not engage). Because the MCP target altitudeis now properly set, CATS predicts the crew

should engage VNAV, which is preferredaccording to the CATS model.

CATS detects a second FL CH switch press inScenario Frame 7 (Figure 11). Perhaps a pilotperformed this action as ‘insurance’ to engage amode to begin the climb. Because FL CH modeengages, and this is reflected in CATS’representation of the current context, CATSinterprets both FL CH switch presses as correctacceptable alternative actions. By this time, CATShas also flagged the second VNAV switch press asan error.

In the final frame of the scenario (ScenarioFrame 8, Figure 12), the aircraft has begunclimbing in FL CH mode. At this point the crewopts to engage VNAV mode. At last, CATSdetects the predicted VNAV switch press andinterprets it as correct.

Figure 7 (Scenario Frame 3). CATS cannot reconcile the VNAV switch press with the current context,and therefore flags it as an error. CATS is still expecting the crew to dial the MCP altitude knob.

13

Figure 8 (Scenario Frame 4). CATS detects a pilot starting to dial the MCP altitude, andinterprets it as matching its prediction, but with the wrong value (This is not an error,because the action is only the start of the altitude setting).

Figure 9 (Scenario Frame 5). CATS detects a second VNAV switch press, prior to thetime when the altitude setting is finished.

14Figure 11 (Scenario Frame 7). CATS detects a second ‘insurance’ FL CH switchpress, and interprets it as acceptable as it did the first FL CH switch press.

Figure 10 (Scenario Frame 6). CATS detects that the crew has now opted to engage FL CH modeby pressing the FL CH switch; but because the altitude is now properly set, CATS now predicts thecrew should push the VNAV switch to engage VNAV (the preferred mode according to the CATSmodel).

15

Conclusions and Future ResearchThe above example demonstrates that CATS candetect errors from flight data. Although the errorsCATS detects are inconsequential, this researchindicates CATS can provide contextualinformation useful for disambiguating the causesof deviations or unusual control actions that arisein incident or accidents. Discoveries made usingCATS can be incorporated into training curriculaby connecting a CATS-based training system to asimulator and allowing pilots to ‘fly’ underconditions that correspond the actual context ofan error-related event. Such capabilities are alsouseful outside the airline arena as they supportboth fine-grained cognitive engineering analysesand human performance modeling research.

Using CATS with flight data collected at‘continuous’ rates results in better performance.Event-based data, such as those available from theNASA ACFS, require more complicatedinterpolation methods to avoid temporal ‘gaps’in the CATS representation of context that canadversely affect CATS performance. Importantdirections for further research involve improvingthe coverage of flight data to include the FMSand CDUs, as well as work on methods toautomatically acquire ATC clearanceinformation. This research indicates that, if CATShas access to data with full, high-fidelity coverageof the controlled system displays and controls, itcan expose the contextual nuances that surrounderrors in considerable detail.

Figure 12 (Scenario Frame 8). The crew opts to engage VNAV; CATS detects the predicted VNAVswitch press and interprets it as correct (elapsed time from Scenario Frame 1 is ~42 secs).

16

ReferencesBabbage, C. (1961). On the principles of thecalculator and other seminal writings by CharlesBabbage [b.1792-d.1871] and others. In P.Morrison & E. Morrison (Eds.), New York, NY:Dover.

Billings, C. (1997). Human-centered aviationautomation. Mahwah, NJ: Lawrence ErlbaumAssociates.

Busse, D., and Johnson, C. (1998). Modelinghuman error within a cognitive theoreticalframework. In F. Ritter & R. Young (Eds.),Proceedings of the 2nd European Conference ofCognitive Modeling (ECCM-98), Nottingham,UK: Nottingham University Press, 90-97.

Byrne, M., and Bovair, S. (1997). A workingmemory model of a common procedural error.Cognitive Science, 21(1), 31-61.

Callantine, T. (1999). An intelligent aid for flightdeck procedure performance: The Crew ActivityTracking System "Task Caddy." Proceedings ofthe 1999 IEEE Conference on Systems, Man, andCybernetics, Tokyo, October, 1095-1100.

Callantine, T. (2000a). A glass cockpit crewactivity analysis tool. SAE Technical Paper 200-01-5522. Warrendale, PA: SAE International.

Callantine, T. (2000b). Online analysis of highfidelity simulation data. Proceedings of theInternational Conference on Cognitive SystemsEngineering for Process Control 2000, Taejon,Korea, November, 175-182.

Callantine, T. (2001a). Agents for analysis anddesign of complex systems. Proceedings of the2001 International Conference on Systems, Man,and Cybernetics, Tuscon, AZ, October, 567-573.

Callantine, T. (2001b). The crew activity trackingsystem: Leveraging flight data for aiding,training, and analysis. Proceedings of the 20thDigital Avionics Systems Conference, Daytona

Beach, FL, October, 5.C.3-1—5.C.3-12 (CD-ROM).

Callantine, T. (2002). A representation of airtraffic control clearance constraints for intelligentagents, Proceedings of the 2002 IEEE Conferenceon Systems, Man, and Cybernetics, Hammamet,Tunisia, October.

Callantine, T., and Mitchell, C. (1994). Amethodology and architecture for understandinghow operators select and use modes ofautomation in complex systems. Proceedings ofthe 1994 IEEE Conference on Systems, Man, andCybernetics, San Antonio, TX, 1751-1756.

Callantine, T., Mitchell, C., and Palmer, E. (1999).GT-CATS: Tracking operator activities incomplex systems. NASA Technical Memorandum208788, Moffett Field, CA, NASA Ames ResearchCenter.

Degani, A. and Heymann, M. (2000). Someformal aspects of human automation interaction.NASA Technical Memorandum 209600, MoffettField, CA: NASA Ames Research Center.

Fields, R., Harrison, M., and Wright, P. (1997).THEA: Human error analysis for requirementsdefinition. Technical Report 2941997, York, UK:University of York Computer ScienceDepartment.

Fields, R., Wright, P., and Harrison, M. (1996).Temporal aspects of usability: Time, tasks anderrors. SIGCHI Bulletin 28(2).

Johnson, C. (1998). Representing the impact oftime on human error and systems failure.Interacting with Computers, 11, 53-86.

Johnson, C. (2000). Novel computationaltechniques for incident reporting. In D. Aha & R.Weber (Eds.), Intelligent Lessons LearnedSystems: Papers from the 2000 Workshop(Technical Report WS-00-03), Menlo Park, CA:AAAI Press.

17

Johnson, C. (2000). Supporting the analysis ofhuman error in national and internationalincident reporting schemes. In P. Cacciabue(Ed.), Proceedings of the 19th European AnnualConference on Human Decision Making andManual Control, Ispra, Italy, June, 137-144.

Johnson, C. (2001). The London AmbulanceService, computer aided dispatch system: A casestudy in the integration of accident reports andthe constructive design of safety-critical computersystems. Reliability Engineering and SystemsSafety, 71(3), 311-326.

McGreevy, M. (2001). Searching the ASRSdatabase using QUORUM keyword search,phrase search, phrase generation, and phrasediscovery, NASA Technical Memorandum210913, Moffett Field, CA: NASA AmesResearch Center.

Mitchell, C. (2000). Horizons in pilot training:Desktop tutoring systems. In N. Sarter & R.Amalberti (Eds.), Cognitive Engineering in theAviation Domain. Mahwah, NJ: LawrenceErlbaum Associates, 211-251.

Perrow, C. (1984). Normal accidents: Living withhigh-risk technologies. New York, NY: BasicBooks.

Reason, J. (1990). Human error. Cambridge:Cambridge University Press.

Sarter, N., and Woods, D. (1995). How in theworld did we ever get into that mode? Mode errorand awareness in supervisory control. HumanFactors, 31(1), 5-19.

Sherry, L, Feary, M., Polson, P., Mumaw, R., andPalmer, E. (2001). A cognitive engineeringanalysis of the vertical navigation (VNAV)function. NASA Technical Memorandum210915, Moffett Field, CA: NASA AmesResearch Center.

Smith, N., Brown, J., Polson, P., and Moses, J.(2001). An assessment of flight crew experienceswith FANS-1 controller-pilot data linkcommunication in the South Pacific. Proceedingsof the 4th USA/Europe Air Traffic ManagementR&D Seminar, Santa Fe, NM, December.

U.S. Federal Aviation Administration (1995).Advanced qualification program, AdvisoryCircular AC 120-54.

U.S. General Accounting Office (1998). Aviationsafety: U.S. efforts to implement FlightOperational Quality Assurance programs, FlightSafety Digest, 17(7-9), 1-36.

Vicente, K. (1999). Cognitive work analysis:Toward safe, productive, and healthy computer-based work. Mahwah, NJ: Lawrence ErlbaumAssociates.

Wiener, E. (1989). The human factors ofadvanced technology ("glass cockpit") transportaircraft. NASA Contractor Report 177528,Moffett Field, CA: NASA Ames Research Center.

Wiener, E., and Curry, R. (1980). Flight deckautomation: Promises and problems. Ergonomics,23, 995-1011.

Wright, P., Fields, R., and Harrison, M. (1994).Deriving human-error tolerance requirementsfrom tasks. Proceedings of the 1st IEEEInternational Conference on RequirementsEngineering, Colorado Springs, CO, April.

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the dataneeded, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, toWashington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget,Paperwork Reduction Project (0704-0188), Washington, DC 20503.

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE

June 2002

3. REPORT TYPE AND DATES COVERED

Technical Memorandum4. TITLE AND SUBTITLE

Activity Tracking for Pilot Error Detection from Flight Data

5. FUNDING NUMBERS

728-20-10

6. AUTHOR(S)

Todd J. Callantine

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Ames Research CenterMoffett Field, California 94035-1000

8. PERFORMING ORGANIATION

REPORT NUMBER

IH-030

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

10. SPONSORING/MONITORING

AGENCY REPORT NUMBER

NASA/CR—2002–21140611. SUPPLEMENTARY NOTES

Point of Contact: Everett Palmer, M/S 262-4, Ames Research Center, Moffett Field, CA 94035 (650) 604-6073

12A. DISTRIBUTION/AVAILABILITY STATEMENT

Subject Category: 03-01, 63-02 Distribution: PublicAvailability: NASA CASI (301) 621-0390

12B. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words)

This report presents an application of activity tracking for pilot error detection from flight data, anddescribes issues surrounding such an application. It first describes the Crew Activity Tracking System(CATS), in-flight data collected from the NASA Langley Boeing 757 Airborne Research IntegratedExperiment System aircraft, and a model of B757 flight crew activities. It then presents an exampleof CATS detecting actual in-flight crew errors.

14. SUBJECT TERMS

Human error detection, Activity tracking, Glass cockpit aircraft15. NUMBER OF PAGES

18

16. PRICE CODE

17. SECURITY CLASSIFICATIONOF REPORT

Unclassified

18. SECURITY CLASSIFICATIONOF THIS PAGE

Unclassified

19. SECURITY CLASSIFICATIONOF ABSTRACT

Unclassified

20. LIMITATION OF ABSTRACT

UnlimitedNSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89)

Prescribed by ANSI Std. Z-39-18298-102