Overview of the INFILE track at CLEF 2009 multilingual INformation FILtering Evaluation

CEA LIST ELDA Univ. Lille 3 - Geriico 101/10/09

CLEF@

1

INFILE

Overview of the INFILE Overview of the INFILE track at CLEF 2009track at CLEF 2009

multilingual INformation FILtering Evaluationmultilingual INformation FILtering Evaluation

Romaric Besançon (1), Djamel Mostefa, Olivier Hamon, Khalid Choukri (2), Stéphane Chaudiron,Ismaïl Timimi (3)

(1) (2) (3)


CLEF@

2

INFILE

Presentation of the INFILE track

Information Filtering EvaluationFilter documents from a document stream according to

long-term information needs (user profiles)

Second edition of the INFILE track in CLEF1 participant in 2008use same data in 2009


CLEF@

3

INFILE

Presentation of the INFILE track

Mutlilingual

English, French, Arabic for both documents and topics

Two tasks

batch filteringthe whole corpus is given to the participants, which

must return a list of filtered documents for each topic

adaptive filteringdocuments are provided to the participants one at a

time through an interactive procedure, with possible automated feedback to adapt the filtering system

closer to real usage in a context of competitive intelligence


CLEF@

4

INFILE

Document Collection

Built from a corpus of news from the AFP (Agence France Presse)

almost 1.5 million news in French, English and Arabic

For the information filtering task:

100 000 documents to filter, in each language NewsML format

standard XML format for news (IPTC)


CLEF@

5

INFILE

Document example

document identifier

keywords

headline


CLEF@

6

INFILE

Document example

IPTC category

AFP category

content


CLEF@

7

INFILE

Topics

50 interest profiles

20 profiles in the domain of science and technology

developped by CI professionals from French institutes INIST, ARIS, Oto Research, Digiport

30 profiles of general interest Profiles developed in French/English Translated into Arabic


CLEF@

8

INFILE

Topics

Each profile contains 5 fields:

title: a few words description

description: a one-sentence description

narrative: a longer description of what is considered a relevant document

keywords: a set of key words, key phrases or named entities

sample: a sample of relevant document (one paragraph)

Participants may use any subset of the fields for their filtering


CLEF@

9

INFILE

Topic Example


CLEF@

10

INFILE

Some topic examples

101102107113115118119127129

Fight against doping in sportsport economyElectronic votingDigital DivideThe free museumsRising oil pricesthe subprimes crisisthe crisis in DarfurThe FARC rebelion

131132136137138140143144149

E-government stakesWireless network and healthAir pollution and air qualityFight against climate changeDrugs and biotechnologyFruits and vegetables intakes and cancer preventionAvian influenzaNanotechnologies and nanosciencesScientific research in Arctic

in general domain

in scientific information domain


CLEF@

11

INFILE

Constitution of the corpus

Same corpus as INFILE@CLEF 2008

With simulated feedback, we need the ground truth before the campaign

To build the corpus of documents to filter:find relevant documents for the profiles in the original

corpususe a pooling technique with results of IR tools

4 IR engines (Lucene, Indri, Zettair and CEA search engine), on several query fields combinations

iterative pooling using Mixture-of-Experts model


CLEF@

12

INFILE

Constitution of the corpus (2)

keep all documents assessed

documents returned by IR systems by judged not relevant form a set of difficult documents

choose random documents (noise)

collection

retrieved

assessed

relevant

test collection

random


CLEF@

13

INFILE

Corpus1

01

10

21

03

10

41

05

10

61

07

10

81

09

11

01

11

11

21

13

11

41

15

11

61

17

11

81

19

12

01

21

12

21

23

12

41

25

12

61

27

12

81

29

13

01

31

13

21

33

13

41

35

13

61

37

13

81

39

14

01

41

14

21

43

14

41

45

14

61

47

14

81

49

15

0

0

50

100

150

200engfreara

ara7312 7886 51241597 2421 1195

31,94 48,42 23,928,45 47,82 23,08

[0,107] [0,202] [0,101]

eng frenumber of documents assessednumber of relevant documentsavg number of relevant docs / topicstd deviation on number of relevant docs / topic[min,max] number of relevant docs / topics

Number of relevant documents for each topic, in each language


CLEF@

14

INFILE

Tasks

Batch filtering (02/04 - 30/05)documents and topics available to participantsreturn list of filtered documents per topic (unordered)

Adaptive filtering (03/06 - 10/07)topics available to participantsdocuments available one at a time (one pass test)

interactive protocol using a client-server architecture (webservice communication)

new document available only if previous one has been filtered

available simulated user feedbackfor adapatationlimited number of feedbacks (200)


CLEF@

15

INFILE

Evaluation metrics

Standard precision / recall / F-measure Utility (from TREC filtering tracks)

per profile and averaged on all profiles adaptivity: evolution curve (values computed each

10000 documents)

two experimental measuresoriginality

number of relevant documents a system uniquely retrieves

anticipationinverse rank of first relevant document detected


CLEF@

16

INFILE

INFILE Participants

9 registered 5 submitted runs

batch filtering

3 participants, 12 runs interactive filtering

2 participants, 3 runs27

countryIMAG Institut Informatique et Mathématiques Appliquées de Grenoble FranceSINAIUAIC

société CADEGE FranceUOWD

team name institute

University of Jaen SpainUniversitatea Alexandru Ioan Cuza of IASI Romania

HossurTechUniversity of Wollongong (Comp.Sci & Engineering) Dubai


CLEF@

17

INFILE

INFILE results

Repartition of runs by task and languages

arafre

eng

eng

ara

fre

batchadaptive


CLEF@

18

INFILE

INFILE results – monolingual batch filtering

F-scoreIMAG IMAG_1 1597 413 0,26 0,30 0,21 0,21UAIC 1597 1267 0,09 0,66 0,13 0,05UAIC 1597 1331 0,06 0,69 0,09 0,03UAIC 1597 1331 0,06 0,69 0,09 0,03UAIC 1597 1507 0,06 0,82 0,09 0,03IMAG IMAG_2 1597 109 0,13 0,09 0,07 0,16IMAG IMAG_3 1597 66 0,16 0,06 0,07 0,22SINAI 1597 940 0,02 0,50 0,04 0,00SINAI 1597 196 0,01 0,08 0,01 0,13

monolingual englishteam run num_rel num_rel_ret precision recall Utility

uaic_4uaic_1uaic_2uaic_3

topics_1googlenews_2


CLEF@

19

INFILE

INFILE results – crosslingual / adaptive filtering

team run num_rel num_rel_ret precision recall F-score UtilityUAIC uaic_4 2421 1120 0,09 0,44 0,12 0,05UAIC uaic_3 2421 1905 0,06 0,75 0,10 0,03UAIC uaic_2 2421 1614 0,06 0,67 0,09 0,02

team run num_rel num_rel_ret precision recall F-score UtilityHossurTech 4 2421 790 0,05 0,31 0,06 0,05

team run num_rel num_rel_ret precision recall F-score UtilityHossurTech 1 1597 819 0,10 0,45 0,10 0,07

crosslingual english / french

monolingual french

crosslingual french / english

57% best mono90% same team mono

crosslingual better than monolingual


CLEF@

20

INFILE

INFILE results – anticipation/originality

team run recall anticipation originality originality(best)IMAG IMAG_1 0,30 0,43 1 4UAIC uaic_4 0,66 0,73 4UAIC uaic_1 0,69 0,75 0UAIC uaic_2 0,69 0,75 0UAIC uaic_3 0,82 0,86 93 267IMAG IMAG_2 0,09 0,22 0IMAG IMAG_3 0,06 0,14 0SINAI topics_1 0,50 0,57 9 9SINAI googlenews_2 0,08 0,10 15UOWD base 0,01 0,05 0 0HossurTech hossurtech_1 0,45 0,59 18 20

team run recall anticipation originality originality(best)UAIC uaic_4 0,44 0,58 0UAIC uaic_3 0,75 0,83 82 1292UAIC uaic_2 0,67 0,76 0HossurTech hossurtech_4 0,31 0,53 177 177

english target language

french target language

strongly correlated with recall

too few pariticipants


CLEF@

21

INFILE

Approaches

Filteringadapted Information Retrieval tools (Lucene)SVM classifier with external ressources (GoogleNews)textual similarity measures with thresholds reasoning model (human plausible reasoning)

Adaptationadaptation of selection thresholdsuser feedback as parameter in reasoning model

Crosslingualbilingual dictionariesmachine translation


CLEF@

22

INFILE

Conclusion and after…

Increasing participation, reasonable result, but not enough…

Currently, no INFILE track planned for next year

interest in multilingual filtering ?2/3 runs on monolingual Englishnot enough participants for crosslingual to have

comparative results

no funding INFILE evaluation kit will be made available

corpus of documents / topics / relevance assessments tools for the interactive adaptive filtering proceduretools for the evaluationdistributed by ELDA

Documents

Overview of the INFILE track at CLEF 2009 multilingual INformation FILtering Evaluation