TopicsandTrendsinIncidentReports:UsingStructuralTopicModelingtoExploreAviationSafetyReportingSystemData

KennethD.KuhnRANDCorporation2017ATMR&DSeminar

Outline

• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling

• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion

AviationSafety,ATMR&D

• Aviationisremarkablysafeatthemoment,butimprovementmaybepossibleand....• NextGen,SESAR,andotherprojectsareleadingtomajorchangesinairtransportationoperations.• Forexample,theFAAisimplementingWakeRecategorization,“reducingseparationcriteriaformultiplerunwayoperations.”• RecentATMR&Dpapersincludemethodologiesforassessingsafetyimplications.• Fleming,Leveson,andPlacke,2013• FlemingandLeveson,2015

ThisStudy

• ThisstudydescribesanexploratoryanalysisofAviationSafetyReportingSystem(ASRS)data.

• Goal1:evaluatetheusefulnessofanovelmethodforidentifying,evaluatingaviationsafetyissues,trends,etc.

• Goal2(moreambitious):uncoverimportantandpreviouslyunreportedconnectionsorthemesinincidentreports.

AviationSafetyReportingSystem(ASRS)

• ASRSisbasedonconfidentialreportsofsafetyincidentssubmittedbypilots,airtrafficcontrollers,airlinedispatchers,andothers.• TheFAAandNASAdevelopedandmanagetheASRS,inpartto“providedataforplanningandimprovementstothefutureNationalAirspaceSystem.”• Similarsystemsexistelsewhere,includingCHIRPintheUnitedKingdomandREPCONinAustralia.• In2015,theASRSdatabaseincludedover1.3millionrecordsandwasaddingroughly7,500additionalreportseachmonth[NASA].

StatisticsonASRSIncidentReports

• Analystsanonymizesubmissionswhichbecomethe“narrativeportions”ofincidentreports.

• AnalystscodeinotherdataandaddresultstotheASRSdatabase.• flightmission(68%“Passenger,”14%personal,6%cargo,...)• reportingorganization(58%“AirCarrier,”16%government,)• locale(e.g.,“LGA.Airport,”“TUL.TRACON,”...)• month• …

• Researchquestion:How“representative”areASRSdata?

ExampleNarrativefromASRS

ResearchonASRSData

• BillingsandReynard(1984)combedthroughincidentreportsandsearchedforrelationships.• “Themostcommoncontrollererrorsinvolvefailuretocoordinatetrafficwithotherelementsoftheairtrafficcontrolsystem.”

• Bliss,Freeland,andMillard(1999)studiedtheroleofcockpitalarmsystemsinASRSincidentreports.

• Recentadvancesinthefieldsofmachinelearning,computationallinguistics,naturallanguageprocessing(NLP)allowforfasterandeasierexplorationofASRSdata.

FrequentlyObservedPhrasesinNarratives

PriorNLPStudiesusingASRSData

• ElGhaoui etal.(2013)describeasuiteofNLPmethodsandtestthemethodsonASRSdata.Themethodswereableto:• “revealcausalandcontributingfactorsinrunwayincursions”• “automaticallydiscoverfourmaintasksthatpilotsperformduringflight”(aviate,navigate,communicate,andmanagesystems)

• Tanguyetal.(2016)providesanoverviewofNLPapplicationstoaviationsafetydata.• “Itappearsthattopicmodellingisverysuitablefor[incidentreport]data”• topicmodelinguncovers“relevantaspectsof[these]documents,ascanbeseenthroughanexpert’sinterpretation”

TopicModeling

• Topicmodelingisamethodthatallowsanalyststoidentify“themainthemesthatpervadealargeandotherwiseunstructuredcollectionofdocuments”(Blei,2012).• ThemostcommonformoftopicmodelingislatentDirichletallocation(LDA).• Documentsandthewordswithinthemarederivedfroma“generativeprobabilisticmodel”(Blei etal.,2003).• ThenumberofwordsN inadocumentisarandomvariabledrawnfromaPoisson(ξ)distribution.

LatentDirichlet Allocation(LDA)Model

• Theparameterθ ofadocumentisrandomvariabledrawnfromaDirichlet(α)distribution.• Thetopicofeachwordzn isarandomvariabledrawnfromamultinomial(θ)distribution.• Eachwordwn isarandomvariablebasedonanotherdrawfromamultinomialdistributiondefiningp(wn|zn,β)terms.

[Blei etal.,2003]

StructuralTopicModeling(STM)

• Structuraltopicmodeling(STM)isanalternativetoLDAwhereparametersdescribingtopicproportions(θ terms)areassumedtobedrawnfromdistributionsthatarebasedoncovariatedata.• Topicsandwordsareassumedtobedrawnfromadistributionspecifictothedocumentbasedonthecovariatedata.• Applicationsinclude:• analysisofMOOCstudentcomments[Reichetal.,2015]• eventdetectionusingtwitterdata[Mishler etal.,2015]• evaluationoflinksbetweencorporatefundingandtheframingofscientificstudiesonclimatechange[Farrell,2016]

Outline

• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling

• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion

FirstSteps,ApplyingSTMtoASRSData

• Thisstudyfirstfocusedonallincidentsreportedfrom1/1/2010to12/31/2015.

• 17topicswereuncoveredwithinincidentreports.

• Togainsomeintuitionregardingtopicmeaning,lookatthewordslinkedtoeachtopic.• Prob - prob.ofwordoccurrenceconditionalontopic• Lift - prob.ofwordoccurrenceconditionalontopicdividedbyprob.ofwordoccurrenceacrosscorpus.• FREX - ratioofwordfrequencyconditionalontopictoword-topicexclusivity.

IdentifiedTopics

…

FurtherAnalysisUsingTopicData

• Wecanlookatcorrelationsamongthetopics.(Recallthatthereisatopicassignedforeachwordpositionineachdocument.)

• Wecanalsolookathowcovariatedataimpactstopicprobabilities.

• Covariatedataconsideredhere:• Phaseofflight• Flightmission• Month

CorrelationsAmongTopics

PhaseofFlightandEst.TopicProportion

●

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14

Estimated Marginal Effect

●When phase of flight = Landing, Topic label = ATC●Landing, Human Factors

●Surface, ATC●Surface, Human Factors

●Takeoff, ATC●Takeoff, Human Factors

●Other, ATC●Other, Human Factors

●Cruise, ATC●Cruise, Human Factors

FlightMissionandEst.TopicProportion

●

0.00 0.05 0.10

Estimated Marginal Effect

●When flight is a passenger flight, Topic label = smoke / fire●passenger, fuel pump / tank / landing gear

●other, smoke / fire●other, fuel pump / tank / landing gear

●cargo, smoke / fire●cargo, fuel pump / tank / landing gear

●private, smoke / fire●private, fuel pump / tank / landing gear

Est.TopicProportionOverTime

0.00

0.05

0.10

0.15

0.20

Time

Expe

cted

Top

ic P

ropo

rtion

ATCfuel pump, tank, landing gearweathermaintenance, fault

1/2011 1/2012 1/2013 1/2014 1/2015

Outline

• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling

• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion

PhrasesinASRSNarrativesfromSFO

TopicsinASRSNarrativesfromSFO

TopicCorrelationsatSFO

1

2

3

4

5

6

7

1 2 3 4 5 6 7Topic

Topic

−1.0

−0.5

0.0

0.5

1.0Correlation

FlightMission,Est.TopicProportionatSFO

●

−0.2 0.0 0.2 0.4 0.6 0.8

Estimated Marginal Effect

●passenger flight, topic label = taxi●passenger flight, topic label = approach

●other, taxi●other, approach

●private flight, taxi●private flight, approach

●cargo flight, taxi●cargo flight, approach

Outline

• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling

• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion

SpecificFindings

• Initialresultsdemonstrated(andquantified)theimportanceofhumanfactorsandairtrafficcontrol,withtheformerbeingmoreprominentontheairportsurface andthelattermoreprominentduringflight.• Thefrequencyoffuelpump,tank,andlandinggearissuesandthesparsityofsmokeandfireissuesforprivateaircraftwerealsorecorded.• AtSFO,methodshighlightedtheQuietBridgeVisualandTipToeVisualapproachpathsasparticularlyprominentinincidentreports.

MoreGeneralConclusions

• ASRSisusefulresourceforresearchers.• Naturallanguageprocessingtechniquesareusefulhere.• STMisabletoidentifyknownissuesandtouncoversomeissuesthathavenotbeenpreviouslyreported,butdoesnotnecessarilyproduceactionableinsights.• Resultscouldbeusedtosetprioritieswhenplanningfutureaviationsafetyresearch.• Subjectmatterexpertiseisneededtodevelopintuitivemeaningstotopics,tointerpretresults,andtoplanandperformfollow-onwork.

Thanks!

• ThisresearchwaspartiallyfundedbytheNASANextGen - ConceptsandTechnologyDevelopmentProjectunderprogramannouncementNNH14ZEA001N-CTD1.

• Thankyou!

• Questions?