D AFFODIL Strategic Support Evaluated Claus-Peter Klas Norbert Fuhr Andre Schaefer University of Duisburg-Essen

DAFFODILStrategic Support Evaluated

Claus-Peter KlasNorbert Fuhr

Andre Schaefer

University of Duisburg-Essen

15.09.2004 University of Duisburg-Essen

Outline Introduction to DAFFODIL

Evaluation Procedure Hypotheses Main Results

Outlook


DAFFOFIL is Virtual Digital library with strategic

support given by high-level search functions

an agent-based system with currently over 35 different agents providing the low-level and high-level functions

connects ~15 data sources for searching and browsing from the computer science area



DAFFODIL: Concepts Strategic Support Scientific Workflow Personalization

Personal library Collaboration Awareness & Annotation

Proactivity


Strategic Support is provided by high-level functions

like metadata search

duplication elemination citation search author network based strategems journal and conference run

in a integrated way


Scientific workflow

Choose data source

structuredstoring

interpret information

create new knowledge

retrieve knowledge


Structured folders document metadata fulltext authors conferences journals websites queries

Collaboration through group folders

Personalization: Personal Library


Personalization: Awareness & Annotation

Long term queries: Query-Profiles: interval processing Authors: new publications Journals, conferences: new issues,

proceedings See whats new:

Message via Email, Fax, SMS, ...

Annotation for scientific discurs


Proactivity

Contextual monitoring of users& system

Direct action Spellchecking Authorname completion Related terms

Recommend next step based on current stage based on collaborative usage


Evaluation framework

Formal bases given by [Saracevic&Covi] Why to evaluate ? What to evaluate ? How to evaluate ?


DAFFODIL Evaluation Scheme Hypo I: More effective IR through strategic support Hypo II: More effective IR through (pro)-active

system Hypo III: More effective IR through personalisation

and collaboration Criteria

Effectiveness Efficiency

Measure Time Relevance User Satisfaction


Evaluation Phases 1st Phase

14 students in parallel sessions System efficiency & bug hunting

2nd Phase 8 researchers Focus on user: Efficiency and

Effectiveness 3th Phase (expert phase)

3 librarians, 3 professors Comments, critics, hints


Gathering Evaluation Data

Tasks on questionnaire 1st phase: protocolls Other phases: video capturing of

monitor and face store usage see reaction store comments

11h video data


Real world task classified 5 Tasks (4 presented) on 4 levels, classified by a

„information need typology matrix“ work task information need

no knowledge

full knowledge

one object

heterogenousset of objects

1L

2L 3L

4L


Result (1st Phase)

Phase 1: 14 students Error hunting, system efficiency Outcome:

speed up 5-7 times Find a lot of bugs

Comment: „Even so slow when searching, i want to use it, since it saves much time, handling the found knowledge“


Main Hypotheses more effective and efficient IR

through strategic support given with DAFFODIL compared to other (DL) information system

more effective, because user can cognitivly process and interpret information more easily

more efficient, because user needs to invest less time


Result (2nd phase) Task 1: Known item instantiation (L1) Find document by given title (precise)

„Search strategies in content-based image retrieval“

43

43

0

10

20

30

40

50

60

70

80

sec.

DAFFODIL WWW

Task 1Cognition time in sec.

Search time in sec.


Task 2: Known item instantiation (L2) Find document by given title (not precise)

„Methods of automated reasoning“

24

11743

0

20

40

60

80

100

120

sec.

DAFFODIL WWW DAFFODIL T1


Search time in sec.

24

117

0

20

40

60

80

100

120

sec.

DAFFODIL WWW


Search time in sec.


ResultTask 3: A real search (L3) Find articles by author X in the area

of digital libraries within the last 4 years.

Results All participants found with Daffodil at

least 12-19 relevant articles. With Google only two persons found

articles any articles, the other gave up. Actually, user said, it will be hard with

google prior to search.


Result Task 5: New area (L4) find literature about a specific subject:

interactive information retrieval;information searching; information seeking

not known by any participant prior to the evaluation

Explorative Search Scanning result lists; related terms feature

Results: Participants at least found a couple of entry

points, to further search Got very soon an idea, what they don‘t want.


Result (3th phase)Experts: Free Search Three librarians and three professors Free Search with own theme Results:

input for improving satisfied

Comments: librarian

„rich functionality“ „D. mediates the knowledge about the tactics to the

user“ professor

„D. seems in a state that it can be used“ „I will propose it to my assistents“


Summary of evaluation Evaluation with 28 participants

showed, that a user is more effective when searching for scientific literature with the strategic support, here given by the DAFFODIL system, but not always more efficient (task1).

But, as the tasks become more complex,the hypotheses hold.


Outlook Continue Evaluation

other systems Long-Term Evaluation

One task from start to end new paper for ECDL 2005

Participants are welcome

Try it outhttp://www.daffodil.de

Documents

D AFFODIL Strategic Support Evaluated Claus-Peter Klas Norbert Fuhr Andre Schaefer University of Duisburg-Essen