What’s happening in iCLEF?(the iCLEF Flickr Challenge)Julio Gonzalo (UNED),
Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED)
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
A piece of good news for CLEF!
People have started to hyperlink MT versions of web pages!
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
The original URL
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
The world is in danger
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
The world needs Multilingual Information Access research/solutions
…(that do care about users)
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Retrospective: goals for iCLEF 2006 Find realistic scenarios where multilingual search
arises naturally multilingual web communities growing around(mostly) non-textual items (ebay, Flickr)Explore Web 2.0 “new text” challenges: folksonomies,
blogs, wikis, etc. Reduce the cost of entry for new participants.
Provide a default multilingual search interface to Flickr. Improve the cost-effectiveness of the traditional
experiment design. Let participants work on their own experimental designs.
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Flickr
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Flickr advantages
Naturally multilingual Web 2.0 challenges (folksonomies) Replicability (unlike general web image
search facilities)
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2006
Topics Ad hoc: find as many
photographs of (different) european parliaments as possible.
Creative: find five illustrations for this article about saffron in Italy.
Visual: What is the name of the beach where this crab is lying on?
Methodology Participants must
propose their own evaluation measures and experiment design.
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Experiments
UNED focused on user’s behaviour How users deal with native/passive/unknown languages? Do they actually use CLIR facilities when available?
SICS focused on user’s perceptions Satisfaction (all tasks) Completeness (creative,ad-hoc) Quality (creative)
Sheffield also measured search effectiveness How many facets were retrieved (creative, ad-hoc) Was the image found? (visual)
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Example result: choice of target languages along time.
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Achievements
Found a natural CLIR task Flickr works for CLIR testing New perspectives for experiment design. Moved from bilingual to multilingual search
problems.
Stick to Flickr in 2007
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Why participation was low?
Kept on recruiting users and forcing them to search whatever we asked for. Experiment design still too costly. User populations still too small.
Cost of entry still too high Building a Flickr interface / adapting the provided interface
was not trivial. Experiment design too open for newcomers.
Schedule collapses with standard tracks. A problem for CLEF groups potentially interested.
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
How to improve?
Kept on recruiting users and forcing them to search whatever we asked for. Experiment design still too costly. User populations still too small.
Cost of entry still too high Building a Flickr interface / adapting the provided interface
was not trivial. Experiment design too open for newcomers.
Schedule collapses with standard tracks. A problem for CLEF groups potentially interested.
Move to Flickr/Web users Make search task a game
Provide experiment design Participants ≈ Searchers
Adjust iCLEF calendar
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007 (i) Target search task
海 (sea, wave)
Find this image Clear goal for the user Clearly defined measures of success (appropriate
for a “Hall of Fame” experience) Invokes different searching strategies
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007 (ii) Evaluation aspects User’s behaviour (Observational/Log Analysis)
Search strategies with native/passive/unknown languages User’s perceptions
Usefulness Difficulty Clarity, etc.
Search effectiveness Success rate and its correlation with user profiles, search
strategies, etc.
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007 (iii) CL search interfaceSingle web interface for all experiments.
Available for: Groups willing to recruit users and conduct their
own experiments. Any web user wishing to enter the iCLEF Hall of
Fame (by finding more images than anyone else!). Groups willing to experience CL searching from
the user’s perspective: who is the best CLEF team searching cross-language?
Full experiments
Log
an
aly
sis
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
iCLEF 2007/2008 schedule
End of October: Feedback from you! on the multilingual search interface experiment design evaluation measures
Mid-December: Guidelines + interface goes public Groups start their own experiments. People can start competing in the search challenge. Organization starts collecting logs from Flickr users
Mid-April: Flickr/Web user Logs released to interested groups Thematic session in Chorus Conference on Multimedia
Search and Access (hosted by Yahoo!). Paper submission: aligned with CLEF 2008
iCLEF 2007/8Lessons learntiCLEF 2006 Good news FlickrRetrospective
Start testing the interface now!
http://bender.lsi.uned.es:8080/iCLEF2007
Feedback to [email protected]