View
8
Download
0
Category
Preview:
Citation preview
0
WelcometotheOPERA
1
AIDAin2019…achallenge
Nomoretrainingdata,onlyexamplesthatillustratetheevaluationIncreasinglydata-intensiveneurallearnersWhatdowedo???
2
Arangeofresponses…
• Justmakemachinelearningwork!
• Learning,augmentedwithexternaldata
• Half-half
• Include(some)learningbutonlyifit’seasy
• Forgetmachinelearning!3
5
4
3
2
1
Overview
1. Systemoverview2. TA1Englishentityandrelationprocessing3. TA1Rus/Ukrentityandeventprocessing4. TA1/2KBconstructionandvalidation5. TA3Hypotheses
4
SYSTEMOVERVIEWZaidSheikh,AnkitDangi,EduardHovy
5
OPERAarchitecture
TA1extractionengines
TA2corefengine
TA3hypothesisformation
Ontology CSR
PowerLoomdatabase
textspeechimagesvideo
6
OPERAframework
Coref
Mini-KBcreation/AIFvalidation
TA1Mini-KB
Input
Mini-KBcreation/AIFvalidation
Hypothesisformation
BeliefGraphconstruction
TA2Mini-KB
Queries
Englishentities
Mini-KBcreation/AIFvalidation
ImagesSpeech
Rus/Ukrentities
Text
Englishevents
Rus/Ukrevents
7
TA1frameworkInput
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
8
OPERATA2+TA3framework
Coref
Mini-KBcreation/AIFvalidation
TA1Mini-KB
Mini-KBcreation/AIFvalidation
Hypothesisformation
BeliefGraphconstruction
TA2Mini-KB
Queryinput
9
KBsandnotations
• AllresultswritteninOPERA-internalframenotation(json)andstoredinCSR(BlazeGraph)
• Input/outputconvertersfrom/toAIDAAIF
• TwoseparateKBcreationandvalidationprocedures,fortwoparallelKBs(givesinsurance,coverage,andbackup):– Chalupsky:usesPowerLoomandChameleonreasoner
– Chaudhary:usesspecializedrules10
5
4
Internaldryruns
• Internaldryrunmini-evalsusingthepracticeannotationsreleasedbyLDC
• Evaluatedresultsmanually
• Resultslookpromising,BUT…hardtocalculateP/R/F1forvariouspartsoftheTA1pipelinebecauseLDCdoesnotlabelallmentionsofevents,relationsandentities,justthe“salient”or“informative”ones(sowehavetojudgethemourselves…laboriousandnotguaranteed)
11
TA1TEXT:ENGLISHENTITIESANDRELATIONS
XiangKong,XianyangChen,EduardHovy
12
OPERATA1framework
Input
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
13
1.Entitydetection:Type-basedNERdata
• Multi-levellearning:– Trainseparatedetectorsfortype,subtype,andsubsubtype-leveltypeclassification
– Addressesdataimbalance– Mayintroducelayer-inconsistenttypes!
• Type-levelfromLDContology:– Trainingdata:KBPNERdataandasmallamountofself-annotateddata
• Sub(sub)type-level:– Trainingdata:YAGOknowledgebase(350k+entitytypes)obtainedfromHengJi—thanks!
14
2
3
2.Entitylinking
• Task:GivenNERoutputmentions,linkthemtothereferenceKB
• Challenges:Over-largeKB,noisyGeonames– PreprocessKB:Removeduplicatedandunimportantentries(i.e.,notlocatedinRussiaorUkraine,ornoWikipediapage)
• Approach,givenanentity:– UseLucenetofindallcandidatesinKB– Filterspuriousmatches– Buildconnectednessgraph,withPageRanklinkstrengthscores
– Prune(densify)graphtodisambiguateentity15
2
3.Entityrelationextraction
• Task:Extractentitypropertiesandeventparticipants• Four-stepapproach:
1. BERTwordembeddingsforfeatures2. Convolution:extractandmergealllocalfeaturesforasentence3. Piecewisemaxpooling:splitinputintothreesegments(byposition)
andreturnmaxvalueineachsegment,for2entities+1relation4. Softmaxclassifiertocomputeconfidenceofeachrelation
16
1
Englishentity/relationdiscussion
• Challengesandproblems– Subsubtypeissuperfine-grained;ourNERengineisstillnotrobustenough
– Wereturnbothtypeandsubsubtypelabels,butintheevalNISTwilljudgeonlyoneofthem
• Mostlylearned,butsomemanualassistance
17
321 2
TA1RUSSIANANDUKRAINIANMariiaRyskina,Yu-HsuanWang,AnatoleGershman
18
OPERATA1framework
Input
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
19
Goalsandchallenges
• Goal:ExtractentityandeventmentionsfromRussianandUkrainiantext,andbuildframes
• Challenges:– Lackofpretrainedoff-the-shelfextractors– Lackofannotateddatatotrainsystems– Highlyspecificontology
• Twopipelines:1. RusandUkrsourcetext2. MTintoEnglish
20
ExampleinputandoutputInput:Про-российскиесепаратистыатаковалиКраматорскийаэропорт.Translation:Pro-RussianseparatistsattackedKramatorskairport.Output:mn0:eventConflict.Attack, text:атаковали
Attacker:mn1,Target:mn3mn5:relationGeneralAffiliation.MemberOriginReligionEthnicity
Person:mn1,EntityOrFiller:mn2, text:Про-российскиесепаратистыmn6:relationPhysical.LocatedNear, text:Краматорскийаэропорт
EntityOrFiller:mn3,Place:mn4
mn1:entityORG, text:Про-российскиесепаратистыmn2:entityGPE.Country.Country, text:Про-российскиеmn3:entityFAC.Installation.Airport, text:Краматорскийаэропортmn4:entityGPE.UrbanArea.City, text:Краматорский
21
Approach1:ProcessinginRus/Ukr
UniversalDependencyParsing
ConceptualMentionExtraction(COMEX)
StanfordNLP UDPipe
Ontology Lexicon
• OurontologyisasupersetoftheNIST/LDContology• Lexiconsare(semi-)manuallycreatedfromthetrainingdata• Conceptualextractionusing(manual)rule-basedinference• Focusisonhighprecision
22
5
Parsing/tagging/chunkingpipeline
• Syntaxpipeline:– UDPipe1.2(Straka&Strakova2017)– Extractheadnounsanddependents– Notallentitiesandeventsneeded
• Eventframeconstruction:COMEX– OurontologyisasupersetoftheAIDAontology– Triggertermsmanuallymappedtoontology:
• Directmatching—manuallycuratedlistoftriggerwords• Englishtriggers—translationorWordNet/dictionarylookup
– Analysisguidedbyannotation:• LDCannotationsfromseedlingcorpus• Ownmanualannotationaswell
23
5
5
COMEXontology
*fighter-planeLDC_ent_146
*airplaneLDC_ent_142
*vehicleLDC_ent_140
*weaponLDC_ent_160
*physical-entity
*entity
*mil-vehicleLDC_ent_145
*MiG-29
• Multipleinheritance• Greatercoverage
24
5
COMEXlexicons
• Connectwordstoontologyconceptsviawordsenses• Providerulesforconnectingconceptsintoamentiongraph• Semanticrequirementsforslotfillersarespecifiedintheontology
W,атаковать,WS:attack-physical,WS:attack-verbalS,WS:attack-physical,*attack-physical,VERBA,WS:attack-physical, Attacker=Pull:active-subj;Pull:passive-subjA,WS:attack-physical, Target=Pull:active-dir-obj;Pull:passive-dir-objA,WS:attack-physical, Instr=Pull:active-subjA,WS:attack-physical, Place=Pull:obl-in#R,Pull:active-subj, nsubj, Trigger->Voice=ActR,Pull:passive-subj, obl, Trigger->Voice=Pass,Target->Case=Ins
Whilethelexiconscontainhundredsofwords,thenumberofrulesissmall25
Lexiconconstruction• InitialvocabularyandthecorrespondingconceptsfromtheavailableLDCannotations
• VocabularyenrichmentbyextractingallnamedandnominalentitiesfromtheseedlingcorpusfilesthatcontainatleastoneLDCannotation
• EventtriggerenrichmentusingWordNet• Cross-languagevocabularyenrichmentusingMTandalignment• Manualcurationoftheresultingvocabulary• Manualadditionofattributerules• Iterativeimprovementprocess:
1. Extractmentionsfromanewfile2. Scoreresults3. Addvocabulary,fixrulesanddocross-languagetransfer
26
5
SampleCOMEXperformance
English Russian UkrainianPrecision 0.91–1.0 0.93–1.0 1.0Recall 0.22–0.56 0.11–0.70 0.07–0.42F1 0.35–0.70 0.20–0.62 0.13–0.59Vocabulary 178 1483 1430*Rules 33 30 13
COMEXisthemost‘manual’ofOPERA’sTA1extractionmodules
27
(Thisworkcontinues;thenumberschangeeveryday)
Approach2:Rus/Ukr—>English• Pipeline:
– MTRus/Ukr–>EnglishusingMSAzure– RunOPERATA1extractors– AlignsourcetexttoextractedmentionsinEng
• Back-translatefromEng,includingXML-likeentity/eventtags
• Outputisgenerallygood(espwhennoXMLtags)
• Problemsinback-translation:– SometimesmessesuptheXMLtags– Mayswitcheventarguments– Maymessuppropernames(e.g.Slavyansk–>Slavska,Slavovsk,Slavic
– ThingsliketyposoruncommonwordsgettranslatedincorrectlyintoEng,butmaybeeasytofixinthesourceusingfuzzymatching 28
2
MT
Approachescomplementary
• Rus/Ukr:moreprecise– Lessnoise,betterentitytyping
• MT:moregeneral– Betteratnames,time/numbers,eventtyping
• Overlapsanddifferences:– Entityoverlap:84%ofRus/Ukr=44%ofMToutput– Eventoverlap:58%ofRus/Ukr=49%ofMToutput– Typeagreement:87%ofoverlap– Remainingmentions:65–70%correctoneachside– Differencesinspans,eventvs.entitychoices
29
Rus/Ukrentity/relationdiscussion
• Challengesandproblems– Slowmanualrulebuilding,limitedcoverage(buthighprecision)
– COMEX<—>AIDAontologyalignment– Noiseintranslation
• Mostlymanual
30
52 5 5
TA1/2KBCONSTRUCTIONANDVALIDATION
HansChalupsky
31
OPERATA1framework
Input
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
32
CSR:PowerLoom-basedCommonsemanticrepository
• ContainsallKEs– Containsdiscretetermpropositions,[structured]distributionalvectors/tensors,continuousembeddings
– Eachwithvectorofscores(e.g.,TA1extractionconfidence,sourcetrustworthiness,reasoningimplicationconfidence,cross-KEcompatibility,hypothesis-basedlikelihoods,etc.)
• RepresentedinPowerLoom(Chalupskyetal.2010)– Predicate-logic-basedrepresentationbasedonKIFthatisasupportedsyntaxofCommonLogic
– Dynamic,scalable,multi-contextualsystemtostore,manageandreasonwithinformation
– Blazegraphdatabasetechforscalabilityandintegration– Representhypothesesandprobabilitiesviareification
• IntheCSReverythingisahypothesis33
5
4
M9Approach:3-stepdecouplingforKBconstructionandvalidation
Multi-mediaAnnotations
NEREntityCorefEDLDBPediaEventsEventCorefRelationsAudioImage,Video…..
Extractors×N Annotation Ontology
Domain Ontology
AIDASeedling
34
KnowledgeAggregation
HypothesisIntegration,Evaluation,Inference
Domain Ontology Domain
Ontology Export
Ontologies
AIDAFull
…..
BackgroundKBBlazegraphTriplestore
Lotsoftypeheterogeneity
ReuseTACEVICdomainmodel
Exporttodifferenttargets
M18Approach:SingleaugmentedontologyforKBconstructionandvalidation
Multi-mediaAnnotations
NEREntityCorefEDLDBPediaEventsEventCorefRelationsAudioImage,Video…..
Extractors×N Domain Ontology
35
KnowledgeAggregation
HypothesisIntegration,Evaluation,Inference
AIDAFull
BackgroundKBBlazegraphTriplestore
Supportsometypeheterogeneity
Domainadditions,APItocode
Annotation Augmentations
Domain Augmentations
5
2
Incrementalcycleofhypothesisrepresentation,evaluation,refinement
Createhypothesis
representation
NERDoc.CorefEDLEventsEventCorefRelationsAudioImage,Video…..
Extractors×NDomain
Ontology
Select Import
Infer,EvalFix
Rules, Constraints
KB
• Cycle:– Usecorefsandotheridentitytoconnectannotations(mentionoverlap,
namelinks,EDL,within-doccoref,eventcoref)– Applyinferences,evaluateconstraints,detectconflicts,doattribution– Fixconflicts“ViktorYanukovych”?=“ViktorViktorovychYanukovych”
—no:irreflexive(parent)36
52
TA1/2KBintegrationchallenges• Challenges:Ontological
– Multipletypesystems:NERtypes,relationtypes,eventtypes,KBschemas,targetschemas…
– Missingtypes,conflictingtypesoncethingsarelinked– Types,eveniffine-grained,primarilyusefulasconstraints,notas
equalitysignal–“Humvee17generally-not-equal-toHumvee42”– Inferencerequirements:“Donechyna”and“Ukraine”arecompatible
locationsofaneventbutnotwithrespecttohaving“Donetsk”astheircapital
– Ontological“fluidity”—thingschangeuntillateinthegame
• Challenges:Datasparsityandnoise– Multi-lingualnamesandcross-lingualmatching– Language-specificnamingschemes(e.g.,patronyms)– Cross-lingualuseofcontextvectors– Nofine-graineddocument,textormediacontextallowedacross
documents– Linkingdecisionsaggregatesupportandontologicalconflictwhich
propagates 37
5
4
5
TA1scores
TA1Classqueries TA1Graphqueries
BestMAP
WorstMAP
TRECMAP
0.4843 0.4737 0.4773
0.4527 0.3697 0.4020
0.4379 0.2816 0.3278
0.4243 0.1470 0.1957
0.2290 0.0892 0.1244
Prec Recall F10.4715 0.2163 0.29660.4944 0.1328 0.20940.3605 0.0533 0.09290.0398 0.0312 0.03500.0138 0.0040 0.0062
Run:TA1a_OPERA_TA1a_aditi_V2
OPERA
TA3HYPOTHESISCONSTRUCTIONAditiChaudhary,AnatoleGershman,JaimeCarbonell
39
OPERATA2+TA3framework
Coref
Mini-KBcreation/AIFvalidation
TA1Mini-KB
Mini-KBcreation/AIFvalidation
Hypothesisconstruction
BeliefGraphconstruction
TA2Mini-KB
Queryinput
40
Candidatehypothesisgeneration
1. (HavecompletedBeliefGraphandbeliefscorepropagationthroughout)
2. RetrieveKEscorrespondingtoentrypoints
3. RetrieveeventsEandrelationsRthatmatchconstraints.If:– Zero-hop:theretrievedeventisonehypothesis– One-hop:obtainaneventforeveryrole.Prioritizeevents/relationswithmaximumoverlapwiththeroles—thismaygivemanypermutations
4. GeneratehypothesiscandidatesetH=h1,h2...hnfromtheretrievedEandR
5
Approach
Mention
matcher
BG
EvidenceKEs
Information Need
statement
EntrypointsarematchedtoEvidenceNodes
HypothesisGraph
candidates
HypothesisGraphselector
Role
inference
Augmented
HypothesisGraphs
Hypothesis
rankingand
filtering Hypotheses
42
Candidatehypothesisgeneration
Hypothesisranking
• GiveninformationneedIandcandidatesetH=h1,h2...hn,weneedtorankHbasedonrelevanceanddiversity
• Maximalmarginalrelevance:MMR=
– Sim(hi,I)=similarityscorebetweenhypothesishiandtheinformationneedI—givesrelevance
– Sim(hi,hj)=similarityscorebetweenhypotheseshiandhj—givesdiversity
λargmaxhi,E,HSim(hi,I)-(1-λ)argmaxhj,E,HSim(hi,hj)
Relevanceanddiversity
• Sim(hi,I)=Measuringrelevance… sumof:– PercentageofframescoveredinI– Percentageofeventssatisfyingtheeventframes– Percentageofrelationssatisfyingtherelationframes– Numberofrole-entityexactmatchconstraints
• Sim(hi,hj)=Measuringdiversity(=inversesimilarity)betweentwohypotheses…sumof:– Numberofoverlappingevents– Numberofoverlappingrelations– Numberofoverlappingentities– Numberofoverlappingargumentsfortheaskedframes
TA3M18evaluation
• Thiswasanextremelycomplextask• Wereceivedalotofnumbers• We’restillanalyzingthem
• Mostofthenumbersarenothelpfulforus• Manythingsconfuseus• Wewishformoredetailaboutcertainaspects
46
Task3a(usingown/anyTA2KB)
Hypossubmit-ted
Theoriesmatched
Correct-ness
Edgecohe-rence
KEcohe-rence
Relstrict
Rellenient Coverage
24 6 0.4393 0.4834 0.6655 0.2832 0.6554 0.0320
42 4 0.2607 0.2894 0.423 0.1343 0.4192 0.0127
7 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0035
2 1 0.4167 0.4167 1.0000 0 1.0000 0.0032
42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032
OPERA
Task3a(usingotherTA2teams’KBs)
Hypossubmit-ted
Theoriesmatched
Correct-ness
Edgecohe-rence
KEcohe-rence
Relstrict
Rellenient
Coverage
34 2 0.1042 0.2178 0.3711 0.1002 0.3512 0.0079
20 2 0.5107 0.6072 0.8145 0.3823 0.7973 0.0061
7 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0035
2 1 0.4167 0.4167 1.0000 0 1.0000 0.0032
42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032
Task3b(usingLDCKB)
Hypossubmit-ted
Theoriesmatched
Correct-ness
Edgecohe-rence
KEcohe-rence
Relstrict
Rellenient
Coverage
42 2 0.5961 0.6249 0.839 0.3829 0.839 0.0238
45 6 0.8065 0.8069 0.9589 0.6263 0.9589 0.0153
42 4 0.8266 0.8612 0.8979 0.8312 0.9034 0.0100
24 2 0.8207 0.8406 1.0000 0.5905 1.0000 0.0060
29 1 0.8925 0.9063 1.0000 0.7421 1.0000 0.0051
Hypothesisassessmentprocedure
• Assessmentprocedure:– Firstassesseachedgeascorrect/incorrect– Foronlycorrectones,matchagainstthegoldprevailingtheory
• Howassess/match?Decisions:– TypeandinformativementionofEdge– TypeandinformativementionofLeftside– TypeandinformativementionofRightside
50
Definedinontology.Smalldifferences.
Undefined.Manydifferencesofopinion
Confusion#1:Initialedgefiltering
• Differenceinassessedhypothesisscoresonsamegold-standardLDCKBinput:
• Whythediscrepancies?OurSIN-drivenhypothesiscreationwasdifferent.ButwhydoesLDC’sownPrecisionnotgetupto.80? 51
Edgescorrect EdgessubmittedOPERA 59.6% 2545BBN 80.6% 908GAIA 82.1% 571UTexas 82.6% 1079PNNL 89.3% 394LDConTA2 0.59Precision
Confusion#2:• WhydidGAIAdoalotbetteronGAIA’sownKBsthanonLDC’sKBs?
52
0.0320 _version2_QueryTypeBthroughE_GAIA_1.GAIA_2.GAIA_2_v2 GAIA2_v2runningonGAIAKBs
0.0248 _version4_QueryTypeBthroughE_GAIA_1.GAIA_2p.GAIA_2 GAIA2runningonGAIAKBs
0.0238 _version2_QueryTypeBthroughE_LDC_2.LDC_2.OPERA_TA3b_2 OPERArunningonLDCKBs
0.0153 _version4_QueryTypeBthroughE_LDC_2.LDC_2.BBN_TA3_v2a BBNrunningonLDCKBs
0.0127 _version2_QueryTypeBthroughE_OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2 OPERArunningonOPERAKBs
0.0100 _version3_QueryTypeBthroughE_LDC_2.LDC_2.UTexas_3 UTexasrunningonLDCKBs
0.0079 _version3_QueryTypeBthroughE_GAIA_1.GAIA_2.OPERA_TA3a_1 OPERArunningonGAIAKBs
0.0061 _version2_QueryTypeBthroughE_BBN_1.BBN_TA2_v2.GAIA_2 GAIArunningonBBNKBs
0.0060 _version2_QueryTypeBthroughE_LDC_2.LDC_2.GAIA_2 GAIArunningonLDCKBs
0.0051 _version3_QueryTypeBthroughE_LDC_2.LDC_2.PNNL_sheafbox_10 PNNLrunningonLDCKBs
coverage
Confusion#3:Informativementions
53
evt/rel:data:relation-instance-HYP-E102-3-r201907150216-23type:ldcOnt:Physical.LocatedNearhandle:womanofOdessaedgeprov:HC000Q7MI:(5700-0)-(5714-0)womanofOdessa-------------------------edge:ldcOnt:Physical.LocatedNear_EntityOrFillerarg:data:entity-instance-HYP-E102-3-r201907150216-0handle:thestrangledwomanofOdessa,whoforpro-Russianshas
becomeasymboloftheWestspartialityintheUkrainiancrisisassessed:CORRECT-------------------------edge:ldcOnt:Physical.LocatedNear_Placearg:data:entity-instance-HYP-E102-3-r201907150216-24handle:Odessaassessed:WRONG
Arg1:thewoman
54
edge:ldcOnt:Physical.LocatedNear_EntityOrFillerarg:data:entity-instance-HYP-E102-3-r201907150216-0handle:thestrangledwomanofOdessa,whoforpro-Russianshasbecomea
symboloftheWestspartialityintheUkrainiancrisisconf:1.000000assessed:CORRECTbestPT:E102Theory4argprov:HC000Q7MI:(5686-0)-(5804-0)thestrangledwomanofOdessa,whoforpro-Russianshasbecomeasymbol
oftheWestspartialityintheUkrainiancrisiscontext:xactlywhathappened.Thisscarcityofinformationexplainswhysomany
rumourshaveemergedaround>>thestrangledwomanofOdessa,whoforpro-RussianshasbecomeasymboloftheWestspartialityintheUkrainiancrisis<<.Thisphotowasoriginallypostedhere.
Arg2:Odessa
• Whyisitwrong?Sometheories:– OdessaisnotLocatedNearOdessa,itISOdessa– ThewomanwasborninOdessabutnowlivesinKiev– Thewomanwasonlyrumoredtohavebeenstrangled– …andmore… 55
edge:ldcOnt:Physical.LocatedNear_Placearg:data:entity-instance-HYP-E102-3-r201907150216-24handle:Odessaconf:1.000000assessed:WRONGargprov:HC000Q7MI:(5709-0)-(5714-0)Odessacontext:hisscarcityofinformationexplainswhysomanyrumourshaveemerged
aroundthestrangledwomanof>>Odessa<<,whoforpro-RussianshasbecomeasymboloftheWestspartialityintheUkrainiancrisis.Thisp
Challenges
• Hypothesesgraphsaretooextensive,aseventsareconnectedbyatleastonecommonargument—needtoaddrestrictions
• Backgroundknowledgesometimesrequiredtolinkentities(e.g.,SU25==militaryjet)—perhapspre-populateKBwithbackgroundknowledge?
56
FINALE
57
Finale
• OPERAisanend-to-endsystem– Successfulcombinationofmachinelearningandmanualcomponentsandapproaches
– Task3b(usingLDC’sKB)submissionhadthehighestcoverage
– Managedwithlimiteddataandchangingontology• Absorbed&processedGAIATA1&TA2outputs
– GottopTA2graphqueryF1(1a)scoreusingGAIAKBs• Currentfocus:
– (Ofcourse)improvementseverywhere– Newdomainandontology– Seriousintegrationofcomponent-levelscores 58
Somediscussionpoints
1. Multipleinheritanceintheontology
2. Mainroleofannotateddataisasexamples:continuousteam–LDCinteractiontogtsystemfeedback?
3. Itishardtoreconstructwhatexactlyassessorssaw.Ifthespecifictextualcontextthatanassessorlookedatforadecisionisrecorded,thenwecan– seethetexttheybasedtheirjudgmenton– maybealsogetsomefinerclassificationoftheerrortype
4. TA1bbias:Component-basedre-processingofsameresultswhengivennewhypotheses
59
60
60
543
2
1
Recommended