Welcome to CLEF 2008 Carol Peters ISTI-CNR Pisa, Italy.

  • Published on
    16-Jan-2016

  • View
    215

  • Download
    0

Embed Size (px)

Transcript

  • Welcome to CLEF 2008Carol PetersISTI-CNR Pisa, Italy

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF Objectives

    Stimulate the development of multilingual IR systems (for European languages !)

    To create a CLIR/MLIA community

    Construct publicly available test-suites Conducting annual evaluation campaigns

    Designing tracks/tasks to meet emerging needs and to stimulate research in theright directionObjective: truly multilingual/multimedia systems

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • Come to CLEF and see Europe!

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF2008 Coordination

    Athena Research Center, GreeceBusiness Information Systems, U. Applied Sciences Western Switzerland, Sierre, SwitzerlandCentre for Evaluation of Human Language & Multimodal Communication Technologies (CELCT), ItalyCentruum vor Wiskunde en Informatica, Amsterdam, NLComputer Science Department, U. Basque Country, SpainComputer Vision and Multimedia Lab, U. Geneva, CHData Base Research Group, U. Tehran, IranDept. of Computer Science, U. IndonesiaDept. of Computer Science & Medical Informatics, RWTH Aachen U., GermanyDept. of Computer Science and Information Systems, U. Limerick, Ireland Dept. of Medical Informatics and Clinical Epidemiology, Oregon Health and Science U., USADept. of Information Engineering, U. Padua, ItalyDept. of Information Science, U. Hildesheim, GermanyDept. of Information Studies, U. Sheffield, UKDept. Medical Informatics, U. Hospitals and University of Geneva, SwitzerlandEvaluations and Language Resources Distribution Agency, Paris, France

    German Research Centre Artificial Intelligence, DFKIGESIS-IZ Social Science Information Centre, GermanyInformation and Language Processing Systems, U. Amsterdam, The Netherlands Information Science, U. Groningen, The NetherlandsInstitute of Computer Aided Automation, Vienna University of Technology, AustriaLaboratoire d'Informatique pour la Mcanique et les Sciences de l'Ingnieur (LIMSI), Orsay, FranceU. Nacional de Educacin a Distancia, Madrid, SpainLinguateca, Sintef, Oslo, NorwayLinguistic Modelling Lab., Bulgarian Acad SciMicrosoft Research Asia NIST, USAResearch Computing Center of Moscow State U.Research Inst. Linguistics, Hungarian Acad. SciencesSchool of Computer Science and Mathematics, Victoria U., AustraliaSchool of Computing, DCU, IrelandTALP , U. Politcnica de Catalunya, Barcelona, SpainUC Data Archive and School of Information Management and Systems, UC Berkeley, USAU. "Alexandru Ioan Cuza", IASI, Romania

    CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, PisaThe following Institutions are contributing to the organisation of the different tracks of the CLEF 2008 campaign:

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEFSteering CommitteeMaristella Agosti, U.Padove, ItalyMartin Braschler, Zurich, Switzerland Amedeo Cappelli, ISTI-CNR & CELCT, Italy Hsin-Hsi Chen, National Taiwan U., Taipei, TaiwanKhalid Choukri, ELRA/ELDA, Paris, France Paul Clough, University of Sheffield, UK Thomas Deselaers, RWTH Aachen University, GermanyGiorgio Di Nunzio, U. Padova, ItalyDavid A. Evans, Clairvoyance Corporation, USA Nicola Ferro, U. Padova, ItalyChristian Fluhr, CEA-LIST, Fontenay-aux-Roses, France Norbert Fuhr, University of Duisburg, GermanyFrederic C. Gey, U.C. Berkeley, USA Julio Gonzalo, LSI-UNED, Madrid, Spain Donna Harman, NIST, USA Gareth Jones, Dublin City University, Ireland Franciska de Jong, University of Twente, Netherlands Noriko Kando, NII, Tokyo, Japan Jussi Karlgren, SICS, Sweden Michael Kluck, German Institute for International and Security Affairs, Berlin, Germany

    Natalia Loukachevitch, Moscow State University, Russia Bernardo Magnini, ITC-irst, Trento, Italy Thomas Mandl, U. Hildesheim, GermanyPaul McNamee, Johns Hopkins University, USA Henning Mller, University & University Hospitals of Geneva, SwitzerlandDouglas W. Oard, University of Maryland, USA Anselmo Peas, LSI-UNED, Madrid, SpainMaarten de Rijke, University of Amsterdam, Netherlands Diana Santos, Linguateca, Sintef, Oslo, Norway Jacques Savoy, University of Neuchatel, Switzerland Peter Schuble, Eurospider Information Technologies, Switzerland Richard Sutcliffe, University of Limerick, Ireland Max Stempfhuber, Informationszentrum Sozialwissenschaften Bonn, Germany Hans Uszkoreit, German Research Center for Artificial Intelligence (DFKI), Germany Felisa Verdejo, LSI-UNED, Madrid, SpainJos Luis Vicedo, University of Alicante, Spain Ellen Voorhees, NIST, USA Christa Womser-Hacker, University of Hildesheim, Germany

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF 2008: Track CoordinatorsAd Hoc: Abolfazl AleAhmad, Hadi Amiri, Eneko Agirre, Giorgio Di Nunzio, Nicola Ferro, Thomas Mandl, Nicolas Moreau, Vivien PetrasDomain-Specific: Vivien Petras, Stefan BaerischiCLEF: Paul Clough, Julio Gonzalo, Jussi Karlgren QA@CLEF: Danilo Giampiccolo, Anselmo Peas, Pamela Forner, Iaki Alegria, Corina Forscu, Nicolas Moreau, Petya Osenova, Prokopis Prokopidis, Paulo Rocha, Bogdan Sacaleanu, Richard Sutcliffe, Erik Tjong Kim Sang, Alvaro Rodrigo, Jodi Turmo, Pere Comas, Sophie Rosset, Lori Lamel, Djamel MostefaImageCLEF: Allan Hanbury, Paul Clough, Thomas Arni, Mark Sanderson, Henning Mller, Thomas Deselaers, Thomas Deserno, Michael Grubinger, Jayashree KalpathyCramer, and William Hersh Web-CLEF: Valentin Jijkoun and Maarten de Rijke GeoCLEF: Thomas Mandl, Fredric Gey, Giorgio Di Nunzio, Nicola Ferro, Ray Larson, Mark Sanderson, Diana Santos, Paula CarvalhoVideoCLEF: Martha Larson, Gareth JonesINFILE: Djamel MostefaDIRECT: Marco Duissan, Giorgio Di Nunzio, Nicola Ferro

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF 2008: Participating Groups

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF: Trend in Participation

    Europe = 69(51); N. America = 12(14); Asia = 15(14), S. America = 3(1), Africa = 1(0)

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008 Multilingual textual document retrieval (Ad Hoc) Mono- and cross-language information retrieval on structured scientific data (Domain-Specific) Interactive cross-language retrieval (iCLEF) Multiple language question answering (QA@CLEF) Cross-language retrieval in image collections (ImageCLEF) Multilingual retrieval of web documents (WebCLEF) Cross-language geographical information retrieval (GeoCLEF)CLEF 2008 TracksPilots: Cross-language Video Retrieval (VideoCLEF) Multilingual Information Filtering (INFILE)

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008No. of Participants per TrackAd Hoc: 26(22)Domain-Spec: 6(5)iCLEF: 6(na)QA@CLEF: 29(28)ImageCLEF: 42 (35)

    WebCLEF: 3(4)GeoCLEF: 11(13) plus VideoCLEF: 5INFILE: 1

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF 2000 2008Participation per Track

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF 2008:Test Collections2000News documents in 4 languagesGIRT German Social Science database2008CLEF multilingual comparable corpus of more than 3M news docs in 15 languages: BG,CZ,DE,EN,ES,EU,FI,FR,HU,IT,NL,RU,SV,PT and PersianThe European Library Data in DE, EN, FR (>3M docs)GIRT-4 social science database in EN and DE, Russian ISISS collection; Cambridge Sociological AbstractsOnline Flickr database IAPR TC-12 photo database (20,000 image, captions in EN, DE); ARRS Goldminer database (200,000 medical images)IRMA: 10,000 images for automatic medical image annotation INEX Wikipedia image collection (150,000 images)Dutch / English documentary TV videosAgence France Press (AFP) newswire in Arabic, French & English

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF 2008: HighlightsBig rise in participation 100 groups in 2008 (81 in 2007); workshop >150 Participants (115 in 2007)Expansion of test-suitesAd Hoc: new collections TEL & Persian new tasksDomain-specific holds its own!Enormous success of ImageCLEFConfirmation of interest in QA@CLEF, GeoCLEFiCLEF lots of interestWebCLEF & INFILE what happened???

    CLEF 2008 Proceedings Published

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF 2008 ProceedingsAdvances in Multingual and MultiModal Information Retrieval8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007 Revised Selected Papers Series: Lecture Notes in Computer Science , Vol.5152 Peters, C.; Jijkoun, V.; Mandl, Th.; Mller, H.; Oard, D.W.; Peas, A.; Petras, V.; Santos, D. (Eds.) 2008, XXI, 922 p. With online files/update., Softcover ISBN: 978-3-540-85759-4

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2000 2008 ResultsCreation of strong CLIR research community (increase in participation over years )Promotion of research in key areas (multilingual IR results merging cross language access in multimedia interactive query formulation and results presentation)Encouraged takeup of techniques/resources between research groupsStimulated synergy between researchers from different areas (IR, NLP, Image Processing, User Interfaces, )Literature: Working Notes, Proceedings and other publications report state of the art plus emerging trendsProduction of language resources test suitesCLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2007 Workshop, Budapest, Hungary 19-21 September 2007 Points for DiscussionWhat new tasks/evaluation methodologies are needed to address more advanced information requirements? How can we best reduce the gap between research and application communities?Who are the users?Does CLEF have a future?

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF & TrebleCLEFCLEF is an activity of the TrebleCLEF Coordination Action under the Seventh Framework Programme of the European Commission. TrebleCLEF organises a set of dissemination activities in the multilingual information access field. CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008Treble-CLEFThe CLEF research results have led to development of a new generation of multilingual retrieval system prototypes BUT lack of technology transferTreble-CLEF extends the CLEF activity by:continuing to promote MLIA R&D via evaluation campaigns;providing a consistent training activity: tutorials, workshops, summer school;producing best practice guidelines for system implementation;providing resources to encourage the multilingual system development.

    www.trebleclef.eu

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • ApproachEvaluationtest collections and laboratory evaluationuser evaluationlog analysis

    Best Practices & Guidelinessystem-oriented aspects of MLIA applicationscollaborative user studiesuser-oriented aspects of MLIA interfaces

    Dissemination and Trainingtutorialsworkshopssummer school

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008Treble-CLEF EventsWorkshop on Novel Methodologies for Evaluation in Information Retrieval, ECIR08, Glasgow, ScotlandBest Practices WorkshopsWorkshop on Best Practices for the Development of Multilingual Information Access Systems, Segovia, Spain, June 08Workshop on Best Practices for System Developers: Bringing Multilingual Information Access to Operational Systems, Winterthur, Switzerland, October 2008Workshop on Best Practices in Query Log Analysis, Spring 2009

    MLIA Technology Day Dissemination of results of Best Practices Workshops, Fall 2009

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008Workshop Memory StickWorkshop Programme List of ParticipantsBook of AbstractsCLEF 2008 QuestionnaireMap at Workshop VenueSocial Dinner - 17 September 2008TrebleCLEF Other

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008Treble-CLEF Summer SchoolFocus: How to build effective MLIA systems and How to evaluate themProgram will cover the following areas: Multilingual Text Processing (language specific tokenization, indexing, stemming);Cross-Language Information Retrieval (approaches and technologies used for CLIR); Multingual Information Retrieval and MultiModality (querying, retrieving & presenting results from a multingual/multimedia collectionSystem Architectures and Multilinguality (theory & practice) Resources for MLIA (information on language processing tools and linguistic resources);Best Practices in User-oriented MLIAEvaluation for Multilingual Systems and Components

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008CLEF2008 QuestionnaireAim of the Questionnaire is to collect information on the current needs of MLIA system developers in terms of applications, resources, evaluation activities

    Compile the questionnaire online atwww.trebleclef.eu/clef_2008_questionnaire.php

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

  • CLEF 2008Thank you for your attentionandENJOY THE Workshop !CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

    CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

Recommended

View more >