Cross-Language Evaluation Forum (CLEF)

IST-2000-31002Expected Kick-off Date: August 2001

Carol PetersIEI-CNR, Pisa, Italy

Carol Peters:Carol Peters:Carol Peters:

blabla

Carol Peters:

blabla

Concertation Event, Vienna, 21 June 2001

2

Cross-Language

Evaluation Forum

ObjectivesPromote research in cross-language system development for European languages by providing an appropriate infrastructure for:

system evaluation, testing and tuningcomparison and discussion of results between

R&D groups working on common problemsbuilding test-suites for cross-language system

developers

Concertation Event, Vienna, 21 June 2001

3

Evaluation for Cross-Language Systems

Why Evaluation is Important for CLIR

CLIR systems are still in experimental stage of development

Evaluation activities stimulate progress through objective assessment and also by comparison of systems and approaches

Concertation Event, Vienna, 21 June 2001

4

Evaluation for Cross-Language Systems

evaluation methodolgy reference multilingual document

collection statements of information needs (>

queries) in multiple languages objective assessment of results comparative analysis of results

Creating the infrastructure for an evaluation campaign

Concertation Event, Vienna, 21 June 2001

5

Cross-Language

Evaluation Forum

Background Jan. 2000 - CLEF launched as collaboration

between DELOS NoE and US National Institute for Standards and Technology (NIST) and the TREC Conferences Methodology for CLEF is an adaptation of TREC

evaluation methodology for multilingual context

CLEF 2000 and 2001 organised within DELOSFrom August 2001, CLEF becomes

independent

6

Concertation Event, Vienna, 21 June 2001

CLEF 2001Task Description

Four main evaluation tracks in CLEF 2001:multilingual information retrieval bilingual information retrievalmonolingual (non-English) information

retrievaldomain-specific IR

plusexperimental track for interactive C-L

systems

7

Concertation Event, Vienna, 21 June 2001

CLEF 2001Multilingual Data Collection

Multilingual comparable corpus of news agencies and newspaper documents for six languages (DE,EN,FR,IT,NL,SP). Over 1 million documents

Common set of 50 topics (from which queries are extracted) created in 9 European languages (DE,EN,FR,IT,NL,SP+FI,RU,SV) and 3 Asian languages (JP,TH,ZH)

8

Concertation Event, Vienna, 21 June 2001

Topics either DE,E,F,I or FI,NL,SP,SV

English German French Italian

Participant’s MLIR/CLIR Information Retrieval System

documents

CLEF 2001 Multilingual IR

One result list of DE, FE, F and I documents ranked in decreasing

order of estimated relevance

9

Concertation Event, Vienna, 21 June 2001

CLEF 2001Bilingual IR

Task: query language DE,FR,IT,FI,NL,SP,SV, RU,ZH,JP,TH - target document

collection is English

Goal: retrieve documents for target language, listing results in ranked list

Easier task for beginners !

10

Concertation Event, Vienna, 21 June 2001

CLEF 2001Monolingual IR

Task: querying document collections in FR|DE|IT|NL|SP

Goal: acquire better understanding of language dependent retrieval problemsdifferent languages present different

retrieval problems issues include word order, morphology,

diacritic characters, language variants

11

Concertation Event, Vienna, 21 June 2001

CLEF 2001Domain-Specific IR

Task: querying a structured database from a vertical domain (social sciences) in German

German/English/Russian thesaurus and English translations of document titles

Monolingual (DE) or cross-language (DE, EN, RU) task

12

Concertation Event, Vienna, 21 June 2001

CLEF 2001Participation

30 groups: 8 N.American; 18 European; 4 Rest of the World

Runs submitted for all tasks: Cross-Language = 20 groups

Multilingual = 8 groupsBilingual -> EN = 18 groupsBilingual -> NL = 3 groups

Monolingual = 20 groups Domain-specific = 1 group

A total of approx 200 runs were submitted

13

Concertation Event, Vienna, 21 June 2001

Approaches to CLIR

CLEF 2000CLEF 2000commercial MT systems (Systran, Lernout

and Hauspie Power Translator)bilingual dictionary look-upaligned parallel corpora (web-derived)similarity thesaurus (using comparable

corpora)

Different strategies experimented for query expansion and results merging

14

Concertation Event, Vienna, 21 June 2001

Evaluation - Summing up

system evaluation is not a competition to find the best

evaluation provides opportunity to test, tune, and compare approaches in order to improve system performance

an evaluation campaign creates a community interested in examining the same issues and comparing ideas and experiences

Concertation Event, Vienna, 21 June 2001

15

Cross-Language Evaluation Forum

Intentions for CLEF 2002/2003Intentions for CLEF 2002/2003study evaluation methodologies wrt user

needs addition of more languagesaddition of new tasks (eg interactive CLEF)C-L evaluation for other document types

(eg speech)produce CLIR system test-suites for the

R&D community

Concertation Event, Vienna, 21 June 2001

16

Cross-Language Evaluation Forum

For more information:

http://www.clef-campaign.orgor

carol@iei.pi.cnr.it