CLEF 2008 - Ǻrhus Robust – Word Sense Disambiguation exercise UBC: Eneko Agirre, Oier Lopez de Lacalle, Arantxa Otegi, German Rigau UVA & Irion: Piek Vossen.

  • Published on
    27-Dec-2015

  • View
    214

  • Download
    1

Transcript

<ul><li><p>Robust Word Sense Disambiguation exerciseUBC: Eneko Agirre, Oier Lopez de Lacalle, Arantxa Otegi, German Rigau UVA &amp; Irion: Piek VossenUH: Thomas Mandl</p></li><li><p>IntroductionRobust: emphasize difficult topics using non-linear combination of topic results (GMAP) This year also automatic word sense annotation:English documents and topics (English WordNet)Spanish topics (Spanish WordNet - closely linked to the English WordNet)Participants explore how the word senses (plus the semantic information in wordnets) can be used in IR and CLIRSee also QA-WSD exercise, which uses same set of documents</p></li><li><p>DocumentsNews collection: LA Times 94, Glasgow Herald 95Sense information added to all content wordsLemmaPart of speechWeight of each sense in WordNet 1.6XML with DTD provided Two leading WSD systems:National University of SingaporeUniversity of the Basque CountrySignificant effort (100Mword corpus)Special thanks to Hwee Tou Ng and colleagues from NUS and Oier Lopez de Lacalle from UBC</p></li><li><p>Documents: example XML</p></li><li><p>TopicsWe used existing CLEF topics in English and Spanish:2001; 41-90; LA 942002; 91-140; LA 942004; 201-250; GH 952003; 141-200; LA 94, GH 952005; 251-300; LA 94, GH 952006; 301-350; LA 94, GH 95First three as training (plus relevance judg.)Last three for testing</p></li><li><p>Topics: WSDEnglish topics were disambiguated by both NUS and UBC systemsSpanish topics: no large-scale WSD system available, so we used the first-sense heuristicWord sense codes are shared between Spanish and English wordnetsSense information added to all content wordsLemmaPart of speechWeight of each sense in WordNet 1.6XML with DTD provided </p></li><li><p>Topics: WSD example</p></li><li><p>EvaluationReused relevance assessments from previous yearsRelevance assessment for training topics were provided alongside the training topicsMAP and GMAPParticipants had to send at least one run which did not use WSD and one run which used WSD</p></li><li><p>Participation8 official participants, plus two late ones:Martnez et al. (Univ. of Jaen)Navarro et al. (Univ. of Alicante)45 monolingual runs18 bilingual runs</p></li><li><p>Monolingual resultsMAP: non-WSD best, 3 participants improve itGMAP: WSD best, 3 participants improve it</p></li><li><p>Monolingual: using WSDUNINE: synset indexes, combine with results from other indexesImprovement in GMAPUCM: query expansion using structured queries Improvement in MAP and GMAPIXA: expand to all synonyms of all senses in topics, best sense in documentsImprovement in MAPGENEVA: synset indexes, expanding to synonyms and hypernymsNo improvement, except for some topicsUFRGS: only use lemmas (plus multiwords)Improvement in MAP and GMAP</p></li><li><p>Monolingual: using WSDUNIBA: combine synset indexes (best sense)Improvements in MAPUniv. of Alicante: expand to all synonyms of best senseImprovement on train / decrease on testUniv. of Jaen: combine synset indexes (best sense)No improvement, except for some topics</p></li><li><p>Bilingual resultsMAP and GMAP: best results for non-WSDOnly IXA and UNIBA improve using WSD, but very low GMAP.</p></li><li><p>Bilingual: using WSDIXA: wordnets as the sole sources for translationImprovement in MAPUNIGE: translation of topic for baselineNo improvementUFRGS: association rules from parallel corpora, plus use of lemmas (no WSD)No improvementUNIBA: wordnets as the sole sources for translationImprovement in both MAP and GMAP</p></li><li><p>Conclusions and futureNovel dataset with WSD of documentsSuccessful participation8+2 participantsSome positive results with top scoring systemsRoom for improvement and for new techniquesAnalysisCorrelation with polysemy and difficult topics underwayManual analysis of topics which get improvement with WSDNew proposal for 2009</p></li><li><p>Robust Word Sense Disambiguation exerciseThank you!</p></li><li><p>Word senses can help CLIR</p><p>We will provide state-of-the-art WSD tagsFor the first time we offer sense-disambiguated collection All senses with confidence scores (error propag.)The participant can choose how to use it (e.g. nouns only)Also provide synonyms/translations for senses The disambiguated collection allows for:Expanding the collection to synonyms and broader termsTranslation to all languages that have a wordnetFocused expansion/translation of collectionHigher recall Sense-based blind relevance feedbackThere is more information in the documents</p></li><li><p>CLIRWSD exercise Add the WSD tagged collection/topics as an additional language in the ad-hoc taskSame topicsSame document collectionJust offer an additional resourceAn additional run:With and without WSDTasks:X2ENG and ENG2ENG (control)Extra resources needed:Relevance assessment of the additional runs</p></li><li><p>Usefulness of WSD on IR/CLIR disputed, but Real compared to artificial experimentsExpansion compared to just WSDWeighted list of senses compared to best senseControlling which word to disambiguate WSD technology has improved Coarser-grained senses (90% acc. on Semeval 2007)</p></li><li><p>QAWSD pilot exerciseAdd the WSD tagged collection/queries to the multilingual Q/A task Same topicsLA94 GH95 (Not wikipedia)In addition to the word senses we provide:Synonyms / translations for those sensesNeed to send one run to the multilingual Q/A task2 runs, with and without WSDTasks:X2ENG and ENG2ENG (for QAWSD participants only)Extra resources needed:Relevance assessment of the additional runs</p></li><li><p>QAWSD pilot exerciseDetails:Wikipedia wont be disambiguatedOnly a subset of the main QA will be comparableIn main QA, multiple answers are requiredIn addition, to normal evaluation, evaluate first reply not coming from wikipedia</p></li><li><p>WSD 4 AVEIn addition to the word senses provide:Synonyms / translations for those sensesNeed to send two runs (one more than other part.):With and without WSDTasks:X2ENG and ENG2ENG (control)Additional resources:Provide word sense tags to the snippets returned by QA results (automatic mapping to original doc. Collection)</p></li></ul>

Recommended

View more >