The LOSA Archive: The data The LOSA Archive: The data and how it can be usedand how it can be used

Ashleigh Merritt, Ph.D.Ashleigh Merritt, Ph.D.The University of Texas Human The University of Texas Human Factors Research Project (UT)Factors Research Project (UT)

22ndnd ICAO Global Symposium on Threat and Error ICAO Global Symposium on Threat and Error Management (TEM) and Normal Operations Management (TEM) and Normal Operations Safety Survey (NOSS) in Air Traffic ControlSafety Survey (NOSS) in Air Traffic Control7-8 February 20077-8 February 2007

Purpose of PresentationPurpose of Presentation

Show how the LOSA Archive is used Show how the LOSA Archive is used in the airline worldin the airline world

Show how a similar NOSS Archive Show how a similar NOSS Archive could benefit the ATC worldcould benefit the ATC world

What is LOSA?What is LOSA?

Line Operations Safety AuditsLine Operations Safety Audits Forerunner/Prototype for NOSSForerunner/Prototype for NOSS Normal operations monitoring based on cockpit Normal operations monitoring based on cockpit

observations of regularly scheduled commercial observations of regularly scheduled commercial flightsflights

Trained expert observers using TEM taxonomy to Trained expert observers using TEM taxonomy to frame, describe, code the flightframe, describe, code the flight

ICAO: Industry best practice for normal operations ICAO: Industry best practice for normal operations monitoring (ICAO LOSA Manual, Doc 9803)monitoring (ICAO LOSA Manual, Doc 9803)

FAA: (FAA Advisory Circular 120-90) FAA: (FAA Advisory Circular 120-90)

TEM in the Airline WorldTEM in the Airline World

ThreatsThreats:: Adverse weather, traffic Adverse weather, traffic congestion, terrain, airport conditions, congestion, terrain, airport conditions, aircraft malfunctions, problems with aircraft malfunctions, problems with ground, maintenance, dispatch, cabin, and ground, maintenance, dispatch, cabin, and of course….. ATCof course….. ATC

ErrorsErrors:: Handling, automation, systems, Handling, automation, systems, procedures, briefings, communicationprocedures, briefings, communication

Undesired Aircraft StatesUndesired Aircraft States: Speed, lateral : Speed, lateral and vertical deviations, unstable and vertical deviations, unstable approaches, incorrect engine/system approaches, incorrect engine/system configurations, taxiway incursionsconfigurations, taxiway incursions

The LOSA Archive (Archie)The LOSA Archive (Archie)

A database maintained and updated by UT A database maintained and updated by UT Data from airlines who do a LOSA with TLCData from airlines who do a LOSA with TLC As of February 2007, data from:As of February 2007, data from:

30 airline LOSAs (including 4 repeat 30 airline LOSAs (including 4 repeat LOSAs)LOSAs)

~6000 regularly scheduled commercial ~6000 regularly scheduled commercial flightsflights

22,000+ threats22,000+ threats 15,000+ errors15,000+ errors 2,500+ undesired aircraft states2,500+ undesired aircraft states

LOSA AirlinesLOSA Airlines

Archive: Three Main UsesArchive: Three Main Uses

1.1. BenchmarkingBenchmarking

2.2. Industry TrendsIndustry Trends

3.3. Interface with other Interface with other safety information safety information sourcessources

Archive Use #1: BenchmarkingArchive Use #1: Benchmarking

Empirical TEM dataEmpirical TEM data New form of data - “One hand clapping” New form of data - “One hand clapping”

Example: Airline XExample: Airline X 36% of flights had a mismanaged threat36% of flights had a mismanaged threat 30% of flights had a mismanaged Handling 30% of flights had a mismanaged Handling

errorerror 26% of flights had some form of intentional 26% of flights had some form of intentional

noncompliance. noncompliance. Is that good, bad or average?Is that good, bad or average?

Airline X and the LOSA ArchiveAirline X and the LOSA Archive

36% of flights had a mismanaged threat36% of flights had a mismanaged threat Average across 30 LOSAs is 34%. Range 16-Average across 30 LOSAs is 34%. Range 16-

63%63% 30% of flights had a mismanaged Handling 30% of flights had a mismanaged Handling

errorerror Average is 37%. Range 13-88%Average is 37%. Range 13-88%

26% of flights had some form of 26% of flights had some form of intentional noncompliance. intentional noncompliance. Average is 46%. Range 22-90%Average is 46%. Range 22-90%

Archive Use #2: Industry Archive Use #2: Industry TrendsTrends

Normal Ops monitoring provides important Normal Ops monitoring provides important base rate informationbase rate information

Example: Unstable Approaches Example: Unstable Approaches We know how many incidents/accidents We know how many incidents/accidents

involve an unstable approach, but do we know involve an unstable approach, but do we know how many unstable approaches occur without how many unstable approaches occur without incident? incident?

Average is 5% of flights. Range 0-15% across Average is 5% of flights. Range 0-15% across 30 LOSAs30 LOSAs

10% of UASs are linked to a mismanaged ATC 10% of UASs are linked to a mismanaged ATC threatthreat

ATC Threats in the LOSA ArchiveATC Threats in the LOSA Archive

Data are based on ~2400 Flights (~9500 threats)

ATC the most common threat ¼ of all threats were ATC threats 10% of all ATC threats were

mismanaged by the crews In all, 30% of all mismanaged threats

were ATC

ATC threatsATC threats

2426 Flights

2349 ATC threats

Challenging clearances/ late change (46%)

Runway Changes (14%)

ATC error (13%)Other ATC

threats (20%)Language

difficulty (7%)

Mismanaged ATC threatsMismanaged ATC threats

2426 Flights

2349 ATC threats 236 mismanaged

Challenging clearances

(53%)

Runway Changes (14%)

Language difficulty (8%)

Similar call signs (5%)

ATC error (7%)

ATC threats -> ATC threats -> Crew ErrorsCrew Errors

2426 Flights

236 mismanaged ATC threats

280 errors

Automation errors (20%) – wrong

MCP/FCU altitude setting dialed

Communication errors (27%) -

misinterpret ATC instructions

Aircraft handling (24%) –

unintentional speed, lateral, vertical

deviations

Cross-verification errors (7%)

Ground navigation errors (6%)

ATC threats -> ATC threats -> Crew Errors -> Crew Errors -> UASUAS

2426 Flights

236 mismanaged ATC threats

280 errors

56 Automation errors

76 Communication errors

67 Aircraft handling errors

19 Cross-verification errors

16 Ground navigation errors

2UAS

58UAS

5UAS

9 UAS

21UAS

ATC threats ATC threats -> UAS-> UAS

2426 Flights

236 mismanaged ATC threats

280 errors

106 UAS

Lateral deviation 17%

Speed too high 14%

Vertical deviation 10%

Unstable Approach 10%

Continued Landing 9%

Incorrect Automation configuration 9%

Taxiway/runway incursion 6%

Speed too low 6%

Incorrect Aircraft configuration 5% Other 14%

Bottom line: 4% of flights had a

UAS arising from an ATC threat that was

mismanaged. ¾ occurred during

descent/approach/land

Archive Use #2: Industry Trends Archive Use #2: Industry Trends What does Noncompliance signify?What does Noncompliance signify?

Has to meet one of three conditions to be coded as Has to meet one of three conditions to be coded as intentional noncompliance in LOSA:intentional noncompliance in LOSA:

Committed multiple times during one phase of flight (e.g., Committed multiple times during one phase of flight (e.g., missing multiple altitude callouts during descent). (coded as missing multiple altitude callouts during descent). (coded as one intentional noncompliance error)one intentional noncompliance error)

Crew openly discusses they are intentionally committing an Crew openly discusses they are intentionally committing an action that is against published SOPaction that is against published SOP

Observer determines the crew is time-optimizing SOP when Observer determines the crew is time-optimizing SOP when time is otherwise available (i.e., performing a checklist from time is otherwise available (i.e., performing a checklist from memory)memory)

Most errors are coded as unintentional/ ‘honest’ mistakes. If Most errors are coded as unintentional/ ‘honest’ mistakes. If observer is in doubt, error is coded as unintentional.observer is in doubt, error is coded as unintentional.

Is Noncompliance a Problem?Is Noncompliance a Problem?

The average (across 30 LOSAs): 46% of flights The average (across 30 LOSAs): 46% of flights have one or more noncompliance errorshave one or more noncompliance errors

Range: 22% of flights at one airline to 90% at another Range: 22% of flights at one airline to 90% at another

Most common noncompliance errorsMost common noncompliance errors checklist performed from memory / nonstandard checklist checklist performed from memory / nonstandard checklist

useuse failure to cross-verify MCP/FCU altitude changes failure to cross-verify MCP/FCU altitude changes PF makes own MCP/FCU changes when hand-flyingPF makes own MCP/FCU changes when hand-flying

Conclusion 1: Noncompliance is just Conclusion 1: Noncompliance is just experienced pilots taking optimizing experienced pilots taking optimizing shortcuts… no big dealshortcuts… no big deal

Industry Trend: What does Industry Trend: What does Noncompliance signify?Noncompliance signify?

Across 30 LOSAs, airlines that have higher rates of Across 30 LOSAs, airlines that have higher rates of intentional noncompliance also have higher rates of:intentional noncompliance also have higher rates of:

Mismanaged threats (r = .60) Mismanaged threats (r = .60) Mismanaged errors - handling (r = .81)Mismanaged errors - handling (r = .81) Mismanaged errors - procedural (r = .80)Mismanaged errors - procedural (r = .80) Undesired aircraft states UAS (r = .81)Undesired aircraft states UAS (r = .81) Mismanaged UASs (r =.73)Mismanaged UASs (r =.73)

Conclusion 2: Increased noncompliance Conclusion 2: Increased noncompliance decreases safety margins… Noncompliance decreases safety margins… Noncompliance reflects the safety culture… reflects the safety culture…

Archive Use #3: Interface with Archive Use #3: Interface with other safety information other safety information

sourcessources Queries from / data-Queries from / data-

sharing with: sharing with: Airlines Airlines Incident Incident

Reporting Reporting systemssystems

IATA/ICAO (ITA)IATA/ICAO (ITA) Transport Safety Transport Safety

BoardsBoards Boeing Boeing ATC (NOSS ATC (NOSS

Archive?)Archive?)

And so - a NOSS Archive?And so - a NOSS Archive?

Have to be patient while the methodology Have to be patient while the methodology matures and the Archive grows, but once in matures and the Archive grows, but once in place:place:

You can benchmark within and across facilities to You can benchmark within and across facilities to determine strengths and vulnerabilitiesdetermine strengths and vulnerabilities

You can trend to determine industry best practices You can trend to determine industry best practices & problems & problems

A TEM-based NOSS Archive would allow ATC to A TEM-based NOSS Archive would allow ATC to “converse” freely with the LOSA Archive on “converse” freely with the LOSA Archive on matters of mutual interest to pilots and controllersmatters of mutual interest to pilots and controllers

The University of TexasThe University of TexasHuman Factors Research ProjectHuman Factors Research Project

www.psy.utexas.edu/HumanFactors