Georg Buscher , Andreas Dengel, Ludger van Elst German Research Center for AI (DFKI)

Georg Buscher

Georg Buscher, Andreas Dengel, Ludger van ElstGerman Research Center for AI (DFKI)

Knowledge Management Department

Kaiserslautern, Germany

SIGIR 08

Query Expansion UsingGaze-Based Feedback on the

Subdocument Level

Query Expansion Using Gaze-Based Feedback on the Subdocument Level, slide 2 Georg Buscher

1. Motivation

2. Reading detection and document annotation technique

3. Implicit feedback methods

4. Study design

5. Results

Outline

/


Outline

1. Motivation



4. Study design

5. Results

/


Background and Motivation

Relevance feedback à la Rocchio is well understood Feedback is mostly applied for entire documents Precision presumably gets better when acquiring feedback on the

subdocument level Drawbacks of such fine-grained feedback:

– Too much cognitive load for explicit feedback– Too little implicit feedback data through explicit interactions (e.g. highlighting)

document

Relevance feedbackon the document level

/

Relevance feedbackon the subdocument level

Use eye gaze as source for implicit feedback on the subdocument level


Outline

1. Motivation



4. Study design

5. Results


Eye Tracking

Unobtrusive Relatively precise

(accuracy: 1° of visual angle) Expensive

Mostly used as „passive“ tool for behavior analysis, e.g. visualized by heatmaps:

We use eye tracking for immediate implicit feedback taking into account temporal fixation patterns


Reading Detection

1. Starting point: Noisy gaze data from the eye tracker.

2. Fixation detection and saccade classification

3. Reading (red) and skimming (yellow) detection line by line

See G. Buscher, A. Dengel, L. van Elst: “Eye Movements as Implicit Relevance Feedback”, in CHI '08


Gaze-Based Document Meta Data

5. Store reading information as document annotations in a semantic Wiki

4. Line-matching by applying optical character recognition

See G. Buscher, A. Dengel, L. van Elst, F. Mittag: “Generating and Using Gaze-Based Document Annotations”, in CHI '08


Outline

1. Motivation



4. Study design

5. Results


Implicit Relevance Feedback for Query Expansion

Input: viewed documents having one specific task in mind Find terms that best describe the user‘s current interest. Use these terms for query expansion

task / information needcontext

terms describing theuser‘s current interest /

context


Three Implicit Feedback Methods to Evaluate

Input:viewed

documents

Gaze-Filter TF x IDF

Gaze-Length-Filter

Interest(t) x TF x IDFbased on length of coherently read text

based on read or skimmed passages


Gaze-Length-Filter

# long read or skimmed passages containing tInterest(t) =

# all read or skimmed passages containing t

Long passages are passages containing at least 230 characters (i.e. more than the following two lines).

The heuristic assumes that shorter text parts only rarely convey sophisticated concepts to the reader.

It further assumes that readers are generally not very interested in the contents of short read or skimmed text parts. Therefore all terms contained in short read or skimmed text parts get a lower interest value.



Input:viewed

documents


Gaze-Length-Filter

Reading Speed

ReadingScore(t) xTF x IDFbased on read vs. skimmed passages containing term t




Reading Speed

P are all read or skimmed passages containing term t.

The heuristic assumes that more thoroughly read text parts (and therefore their terms) are more likely to be of interest to the user than cursorily viewed parts.

1ReadingScore(t) =

|P |tΣ

p є Pt

r(p)

t



Input:viewed

documents

Baseline TF x IDF


Gaze-Length-Filter

Reading Speed

ReadingScore(t) xTF x IDFbased on read vs. skimmed passages containing term t

based on opened entire documents




Outline

1. Motivation



4. Study design

5. Results


Study Design

1. Informational task given• 2 different tasks• Task description in simulated email• Participants had to imagine being journalists

2. Read pre-selected documents• Email attachments• Document structure carefully chosen

3. Search for more information on Wikipedia• 3 different queries:

main topic, sub-topic, related topic

4. Give relevance feedback for the first20 result entries per query

Read about topic in email

Look through 4 emailattachments to get

started with the topic

Find more informationby querying search

engine

Give explicit relevancefeedback

3x

2x


Topic: perceptual organs of animals

Pre-selected documents: 4 Wikipedia articles about cats, sharks, dogs, bats

– The articles described all facets of the species.

– Each article contained several paragraphs dealing with perception-related issues.

3 different queries– Main topic query: more material about perception– Sub-topic query: more material about visual perception– Related-topic query: perceptual organs for the earth‘s magnetic

field

Task Example


Result List Generation

Create basic result list Create expanded

queries(+ top 50 terms)

Re-rank that list for every query expansion variant

Merge the re-ranked result lists in a balanced, ordered way

Present merged list to the participant

User query

Variation: Baseline

Variation: Gaze-Filter

Variation: Gaze-Length-Filter

Variation: Reading-Speed

Re-ranked list 1

Re-ranked list 2

Re-ranked list 3

Re-ranked list 4

Expanded query 1

Expanded query 2

Expanded query 3

Expanded query 4

Result list

Merged result list

Viewed documents

User


Outline

1. Motivation



4. Study design

5. Results


Overview

21 participants

60-80 minutes per participant

111 issued user queries

2220 explicit relevance ratings

Distribution of the relevance ratings


Precision and Discounted Cumulative Gain (DCG)


Mean Average Precision

Powerful improvement of all gaze-based variants over the baseline

Reading-Speed variant is less effective than GF and GLF

GLF might be a bit better than GF?

** : p < 0.01 * : p < 0.05 (*): p < 0.1 (two-tailed paired t-test)


Query Type Differentiation

Generally similar trend within each query type

MAP consistently decreases from main topic to sub topic to related topic queries

– Narrow information needs especially for related topic queries– Wikipedia did not contain too many relevant pages

MAP of the Baseline decreases much more (-0.25)compared to GF (-0.17), GLF (-0.18)

Asterisks mark significance of improvement overthe baseline

B: BaselineGF: Gaze-FilterGLF: Gaze-Length-F.RS: Reading-Speed


Pages about animal species

Inappropriate Context

The baseline method extracts terms that might be far away from the user‘s current topic of interest.

Expanding the query with these terms can lead in a wrong and for the user unpredictable direction.

The more distant the topic of the user’s next query is (i.e. related topic query), the more negative is the effect of unsuitable terms for expanding the query.

Animal perception

Parts of animal perception

(e.g. only visual and auditory perception)

Gaze-based methods

Animal species

Baseline method


Conclusion

Gaze data can effectively be analyzed and used as a source for implicit feedback

Reading behavior detection on its own provides useful information for query expansion and re-ranking

Precision can be improved just by adding those terms to a query that have been read before

Future Work More realistic web search scenarios (e.g. not only on Wikipedia) More sophisticated heuristics for interpreting gaze-based

feedback Gaze also for long-term implicit feedback (e.g. desktop search)


Interested?

Interested in implicit feedback for personalization?– E.g. scrolling behavior, click-through, mouse movements, eye

tracking, EEG, bio sensors, emotions, magic, …

Please let me know!– [email protected]– Workshop?

http://web.mac.com/trevorpenney/iWeb/Laboratory%20Site/Emotion_files/angry_man.png


Thank you for your attention!

Special thanks for the travel grant by- ACM SIGIR- Amit Singhal made in honor of Donald B. Crouch- Microsoft Research made in honor of Karen Sparck Jones

Documents

Georg Buscher , Andreas Dengel, Ludger van Elst German Research Center for AI (DFKI)