Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr....
If you can't read please download the document
Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr. Ute Fischer Sponsored by the FAA, Tom McCloy as Technical
Elizabeth Scottie-Beth Fleming Committee Members: Dr. Amy
Pritchett, Chair Dr. Karen Feigh Dr. Ute Fischer Sponsored by the
FAA, Tom McCloy as Technical Monitor DEVELOPING A TRAINING PROGRAM
FOR THE TRAFFIC ALERT AND COLLISION AVOIDANCE SYSTEM IN
CONTEXT
Slide 2
Overview +Introduction to TCAS and Background Information
+Design of Training Program +Evaluation of the Training Program
+Results +Conclusions 2 Develop a training program intended to
improve pilots understanding of TCAS use for collision avoidance in
a range of traffic situations
Slide 3
Traffic alert and Collision Avoidance System (TCAS) 3 +Pilot
always shown the Traffic Situation Display (TSD) +TCAS delivers a
two stage advisory and vertical avoidance maneuver Traffic Advisory
(TA) - Traffic Traffic Resolution Advisory (RA) - Climb Climb
+Pilot is to follow an RA, even if it conflicts with ATC
instructions, unless the pilot believes that safe flight would be
jeopardized Federal Aviation Administration (2000). Introduction to
TCAS II Version 7. Washington, D.C.
Slide 4
Collision Avoidance in a Broader Context Pilots dont only
interact with TCAS, they also receive information from ATC and the
environment 4 Pilots Response Advised Maneuvers for Collision
Avoidance Notification and Awareness Awareness of other aircraft
via TSD TCAS Traffic Advisory TCAS Resolution Advisory ATC Traffic
Callout Visual acquisition of other aircraft ATC Maneuver
Compliance to TCAS Compliance to ATC Personal Assessment and
Maneuver Decision Party-line Information
Slide 5
Current TCAS Training Requirements: Ground Training +Classroom
or computer based +>60 training requirements +Measures
performance through quizzes and/or activities 5 To comply to the
RA, you should Pull the stick back Push the stick forward Federal
Aviation Administration (2001). Advisory Circular No. 120-55B: Air
Carrier Operational Approval and Use of TCAS II. Washington,
D.C.
Slide 6
Current TCAS Training Requirements: Flight Training 6
Slide 7
Observed TCAS Use and Operation o The TCAS Operational
Performance Assessment (TOPA) monitored the terminal area of 8
major airports and examined pilot compliance to climb and descend
advisories 7 41% to Climb RAs 59% to Descend RAs TOPA observed
compliance rates of Misunderstanding/ Confusion Aggressiveness
Non-Compliance Olson, W. and J. Olszta (2010). TCAS Operational
Performance Assessment in the U.S. National Airspace. IEEE/AIAA
Digital Avionics Systems Conference. Olszta, J., & Olson, W.
(2011). Characterization and Analysis of Traffic Alert and
Collision Avoidance Resolution Advisories Resulting for 500' and
1,000' Vertical Separation. Paper presented at the Ninth USA/Europe
Air Traffic Management Research and Development Seminar (ATM 2011),
Berlin, Germany.
Slide 8
Observed TCAS Use and Operation 8 [My FO and I] incorrectly
interpreted the red 'above' target on the VSI and responded
improperly. We further reviewed the procedures, agreeing that a
person should fly 'away' from the red VSI indication, if instructed
via RA. ACN:785761, 2008 [My FO and I] incorrectly interpreted the
red 'above' target on the VSI and responded improperly. We further
reviewed the procedures, agreeing that a person should fly 'away'
from the red VSI indication, if instructed via RA. ACN:785761, 2008
Pilot reports flying into the red region on the VSI Non-Compliance
Aggressiveness Misunderstanding/ Confusion NASA. (2009). Aviation
Safety Reporting System. Retrieved August 1, 2010:
http://asrs.arc.nasa.gov/ Descending into an airplane that is
clearly descending? TCAS software clearly did not give appropriate
guidance, nor did it self-correct when the initial guidance was so
clearly wrong ACN: 854982, 2009 Descending into an airplane that is
clearly descending? TCAS software clearly did not give appropriate
guidance, nor did it self-correct when the initial guidance was so
clearly wrong ACN: 854982, 2009 Pilot disagrees with advised
descend RA
Slide 9
Observed TCAS Use and Operation 9 Non-Compliance
Misunderstanding/ Confusion Aggressiveness Far Eastern Air B757
Response to Descend RA (TCAS advised a descent rate of 1500 FPM)
Image copied from Lacagnina (2008). Easy Does It. Aero Safety World
: http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1
Slide 10
Research Statement and Objectives Develop a training program
intended to improve pilots understanding of TCAS use for collision
avoidance in a range of traffic situations 10 (1) Train pilots to
understand TCAS use for collision avoidance in the actual traffic
and operational traffic environment (2) Provide pilots with a
well-rounded knowledge of different traffic situations that may
result in TCAS advisories (1) Train pilots to understand TCAS use
for collision avoidance in the actual traffic and operational
traffic environment (2) Provide pilots with a well-rounded
knowledge of different traffic situations that may result in TCAS
advisories
Slide 11
11 Demonstration Based Training Event Based Training Approach
to Training Design Complement of Two Methods
Slide 12
Demonstration Based Training (DBT) +Computer-Based Training ~25
minutes Provides conceptual understanding of TCAS Outlines rules
for compliance +6 segments Introduction to TCAS Traffic Situation
Display TCAS Advising Logic Traffic Advisories Resolution
Advisories Example Timeline of RA Evolution +Mid-Training Quizzes
12
Slide 13
DBT: Demonstrations of RAs 13
Slide 14
Event Based Training (EBT) +Presents traffic events that create
the requirement to act +Builds context and complexity into each
scenario as the flight progresses +Uses a more realistic training
environment Fowlkes, J., Dwyer, D., Oser, R., & Salas, E.
(1998). Event-Based Approach to Training (EBAT). The International
Jounal of Aviation Psychology, 8(3), 209-221. 14
Slide 15
Air Traffic Transcripts Charts & Checklists Audio
Communications (Aviation Intercom) TCAS Alerts Simulation
Architecture Eyetracker PartyLine First Officer Captain
Experimenter/Instructor Coded Log of Flights B747-400 Simulator
(RFS) TSD PFD ND PFD ND Touch screen SideStick ATC->TSD VGA ATC
Air Traffic Simulator (TGF) TCAS Logic 15 Simulator Study in
Integrated Flightdeck ATC Environment Pritchett, A., Fleming, E.,
Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012).
Pilot Interaction with TCAS and Air Traffic Control. Paper
presented at the 2nd International Conference on Application and
Theory of Automation in Command and Control Systems (ATACCS),
London.
Slide 16
Structure of Events: Descend RA Example 16 Training Objective
Context of EventPerformance Measures Feedback Session Accurate
interpretation of and response to TCAS Descend RA. Instrument
Meteorological Conditions (daytime, clouds, no winds) ATC provides
no traffic information Conflict caused by IFR traffic enroute (most
likely on departure) RA maneuvering should not violate ATC
instructions No conflicting ATC or party-line information Pilot
disengages autopilot and flight directors If the pilot did not meet
any particular performance measure, review the correct response in
regards to that measure Pilot responds to advisory with appropriate
vertical speed Pilot ensures vertical speed is not excessive o
Pilot notifies ATC of response to TCAS advisory as the maneuver is
performed Pilot reengages autopilot and flight directors o Pilot
notifies ATC of TCAS advisory and response after clear of conflict
Pilot returns to original clearance (if needed)
Slide 17
Evaluating the training programs impact
Slide 18
+Baseline Study +Training Study +6 scenarios with 2 traffic
events per scenario +Traffic events defined by RA type ATC
information Traffic density Overview of Study 18 Comparison to
identify the impact of training
Slide 19
Overview of Study Pre-Training Data Collection (50 minutes)
Pre-Experiment Questionnaire Pre-Experiment Quiz TCAS Training
Program (50 minutes) Introduction to TCAS Demonstration Based
Training Short Section Quizzes Event Based Training Evaluating TCAS
Training Program (120 minutes) Flight Scenarios Post Scenario
Questionnaires Debrief (15 minutes) Post- Experiment Questionnaire
19
Slide 20
Skill-Based Behavior Rule-Based Behavior Knowledge-Based
Behavior 20 Skill-Based Behavior Rule-Based Behavior
Knowledge-Based Behavior Does the training program improve pilot
performance in response to TCAS advisories and increase pilot
understanding of TCAS?
Slide 21
21 Does the training program improve pilot performance in
response to TCAS advisories and increase pilot understanding of
TCAS? Decrease time to achieve compliance Decrease time to
disconnect autopilot Reduce aggressive response features Increase
percentage of RA duration in compliance Increase appropriate
response post-Clear of Conflict Increase understanding of TCAS
Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior
Knowledge- Based Behavior
Slide 22
Assessing the Pilots Response: Skill-Based Behaviors 22
Vertical Speed time 5 seconds RA Climb 2 sec TCAS weakens required
vertical rate TCAS assumed g pull-up Clear of Conflict TCAS assumed
constant vertical RA rate Autopilot Disconnect Time TCAS RA
Maneuver Pilots Response Time Pilot First Achieves Compliance
Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior
X
Slide 23
Mean Time to Comply: During Training 23 Skill-Based Behavior p
MM = Significance for the mixed model p 2 = Significance of the
variance p = Significance of the means Rule-Based Behavior
Knowledge- Based Behavior
Slide 24
+During training, no significant differences +Post training
Autopilot disconnect time decreased Pilot response was more
consistent for one event Autopilot Disconnect: Post Training 24
Skill-Based Behavior Rule-Based Behavior Knowledge- Based
Behavior
Slide 25
Impact of Training Program 25 Decrease time to achieve
compliance Decrease time to disconnect autopilot Reduce aggressive
response features Increase percentage of RA duration in compliance
Increase appropriate response post-Clear of Conflict Increase
understanding of TCAS Increase trust in TCAS Skill-Based Behavior
Rule-Based Behavior Knowledge- Based Behavior Skill-Based Behavior
Rule-Based Behavior
Slide 26
Assessing the Pilots Response: Aggressiveness 26 Vertical Speed
time 5 seconds RA Climb 2 sec TCAS weakens required vertical rate
TCAS assumed g pull-up Clear of Conflict TCAS assumed constant
vertical RA rate Maximum vertical rate Maximum vertical rate
difference Vertical rate difference TCAS RA Maneuver Pilots
Response Skill-Based Behavior Rule-Based Behavior Knowledge- Based
Behavior *Altitude Deviation
Slide 27
Assessing the Pilots Response: Compliance 27 Vertical Speed Not
in Compliance In Compliance TCAS RA Maneuver Pilots Response time 5
seconds RA Climb 2 sec TCAS weakens required vertical rate TCAS
assumed g pull-up Clear of Conflict TCAS assumed constant vertical
RA rate Skill-Based Behavior Rule-Based Behavior Knowledge- Based
Behavior
Slide 28
Assessing the Pilots Response: Return to Clearance 28 TCAS RA
Maneuver Pilots Response TCAS assumed constant vertical RA rate
TCAS weakens required vertical rate Clear of Conflict Altitude time
5 seconds RA Climb TCAS assumed g pull-up Cleared altitude at time
of RA Pilot holds new altitude Contacts ATC and request new
clearance or ask for further instructions Pilot begins descent back
to originally cleared altitude May inform ATC of response to RA
Skill-Based Behavior Rule-Based Behavior Knowledge- Based
Behavior
Slide 29
Before Training +89% of the pilots did not know that an RA
should cause less than 500 feet of altitude deviation +56% of the
pilots responded that they would hold current altitude achieved
after responding to an RA, as opposed to returning to their
clearance +When asked about airline procedures for following TCAS,
all 18 pilots noted the need for complying with an RA BUT 28% of
the pilots commented compliance wasnt necessary is there was a TCAS
malfunction or if the RA would cause an unsafe situation 29
[Pilots] must always comply with a TCAS RA unless [aircraft]
performance is hindered (i.e. operating single engine) or [there
is] an obvious TCAS malfunction (ie you can see traffic and it is
not a threat) Skill-Based Behavior Rule-Based Behavior Knowledge-
Based Behavior
Slide 30
+During training, aggressive response features decreased +Post
training, same decreasing trend observed Aggressiveness 30 Altitude
Deviation Average Vertical Rate Difference Maximum Vertical Rate
Difference All decreased Skill-Based Behavior Rule-Based Behavior
Knowledge- Based Behavior
Slide 31
+During training, percentage compliance decreased in training
event with Climb RA (93.1% compared to 99.6%) +Post training No
significant differences in means observed But, trained pilots had a
more consistent response Percentage Compliance 31 Skill-Based
Behavior Rule-Based Behavior Knowledge- Based Behavior
Slide 32
Return to Clearance 32 Skill-Based Behavior Rule-Based Behavior
Knowledge- Based Behavior Pilot returns to original clearance No
Yes
Slide 33
Impact of Training Program 33 Decrease time to achieve
compliance Decrease time to disconnect autopilot Reduce aggressive
response features Increase percentage of RA duration in compliance
Increase appropriate response post-Clear of Conflict Increase
understanding of TCAS Increase trust in TCAS Skill-Based Behavior
Rule-Based Behavior Knowledge- Based Behavior Rule-Based Behavior
Knowledge- Based Behavior
Slide 34
Before Training +94% pilots agreed with statement I understand
TCAS maneuvers when they are issued, but Only 50% of the pilots
correctly identified the assumptions made by TCAS advisory logic
When asked to interpret TSD symbols, only 50% of the pilots got all
parts of the associated questions correct 34 Skill-Based Behavior
Rule-Based Behavior Knowledge- Based Behavior
Slide 35
My understanding of TCAS has increased +50% pilots reported an
increase in understanding TCAS logic +27% pilots claimed to have
learned about different types of TCAS RAs (notably, Crossing RAs)
Post Training Understanding of TCAS 35 Types of RA's were not
previously taught. We were taught simply to comply Strongly
Disagreed 0 0 Neutral 6% Agreed 61% Strongly Agreed 33% Skill-Based
Behavior Rule-Based Behavior Knowledge- Based Behavior
Slide 36
Post Training Trust in TCAS I am more likely to trust TCAS
after completing todays training 36 My trust in TCAS was already at
a maximum so I wouldnt be more likely to trust it Strongly
Disagreed 11% Disagreed 0 Neutral 39% Agreed 39% Strongly Agreed
11% Skill-Based Behavior Rule-Based Behavior Knowledge- Based
Behavior
Slide 37
Post-Training 37 Decrease time to achieve compliance Decrease
time to disconnect autopilot Reduce aggressive response features
Increase percentage of RA duration in compliance Increase
appropriate response post-Clear of Conflict Increase understanding
of TCAS Increase trust in TCAS
Slide 38
Conclusions and Future Work Conclusions +Pilots may not need
more training, but instead need better training Integrating DBT and
EBT methods permits ground-based and flight training material to be
more cohesive EBT structure allows for the design of purposeful
training events +Current FAA mandated TCAS training objectives may
not fully reflect all training areas Language needed to address the
reduction of excessive responses to RAs Future Work +What
facilities and technologies would be required to implement this
type of training program? +What implications arise when considering
training design versus system design? Would incorporating human
factors considerations in initial design stages decrease the amount
of required training? 38
Slide 39
39
Slide 40
Conclusions and Future Work Conclusions +Pilots may not need
more training, but instead need better training Integrating DBT and
EBT methods permits ground-based and flight training material to be
more cohesive EBT structure allows for the design of purposeful
training events +Current FAA mandated TCAS training objectives may
not fully reflect all training areas Language needed to address the
reduction of excessive responses to RAs Future Work +What
facilities and technologies would be required to implement this
type of training program? +What implications arise when considering
training design versus system design? Would incorporating human
factors considerations in initial design stages decrease the amount
of required training? 40
Slide 41
Acknowledgements +Work sponsored by the FAA, Tom McCloy as
Technical Monitor +34 pilot participants +Dr. Amy Pritchett +Dr.
Karen Feigh +Dr. Ute Fischer +Dr. Wesley Olson, MIT Lincoln Labs
+Wayne Gallo, FAA +Roger Sultan, FAA +Kylie Garey +TCAS Team:
William Cleveland, Vlad Popescu, Justin Mullins, Anil Bozan, Henry
Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, Dhruv Thaakar,
Jonathan Zoetrum, Jelle Wissink +CEC Lab Members 41