Upload
vanlien
View
215
Download
0
Embed Size (px)
Citation preview
Overview
♠ Memorability study♠ Recognition study♠ Assumptions♠ Implementation issues♠ Evaluation issues♠ Choose your own adventure
Memorability Study♠ Participants issued
self-selected query♠ After an hour, asked
to fill out a survey♠ 129 people
remembered something
Data Analysis
Probability of being remembered♥ Anything? # of words? # of fields?♥ Features
♣Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc.
♣Query features: query type, query length, # of search in session, elapsed time, etc.
♠ Remembered rank v. real rank♥ Map remembered rank to real rank
“Memorability”
00.10.20.30.40.50.60.70.8
1 2 3 4 5 6 7 8 9 10
Rank - R
P(R
emem
|R,C
)
Clicked - C Not clicked
Recognition Study
♠ Same set-up as Memorability Study♠ Follow-up survey: Results the same?
♥ Case 1: Old results♥ Case 2: New results♥ Case 3: Clicked to top♥ Case 4: Intelligent merging
♠ 92 people have completed both steps
16%
74%
65%
17%
Assumptions♠ Re-search v. search♠ Memorable v. relevant♠ Results change v. stay the same♠ Hide change v. show change♠ Forget v. remember as forgettable♠ Merge v. identify old or new
Why? How to test? What if I’m wrong?
Implementation Issues
♠ Page of cached result may disappear♠ Multiple result pages♠ Identifying repeat queries
♥ User identified♥ Search sessions are not repeat queries♥ Exact query may be forgotten
Evaluation Issues
♠ Various goals to test♥ Does a merged list look like the original?♥ Does merging make re-finding easier?♥ Is search improved overall?
♠ Lab study♥ How to set up re-finding task?♥ Timing differences significant enough?
♠ Longitudinal study – What to measure?♠ What are good baselines?
Choose Your Own Adventure♠ Re-search v. search♠ Memorable v. relevant♠ Results change v. stay the same♠ Hide change v. show change♠ Forget v. remember as forgettable♠ Merge v. identify old or new
♠ Implementation issues♠ Evaluation issues
Choose Your Own Adventure♠ Re-search v. search♠ Memorable v. relevant♠ Results change v. stay the same ♠ Hide change v. show change♠ Forget v. remember as forgettable♠ Merge v. identify old or new
♠ Implementation issues♠ Evaluation issues (Done)
Hide Change v. Show Change
♠ Why I think change should be hidden♥ Example: dynamic menus
♠ How to prove♥ New results better, called the same or worse♥ Baseline for testing – 2 lists, change explicit
♠ What if we should show change?♥ Memorability suggests changes to highlight♥ Other applications where want to hide change
(Done)
Memorable v. Relevant
♠ Why I think memorability is important♥ Relevance at a future date is what matters♥ Necessary to hide change
♠ How to prove♥ Baseline for lab study with target first
♠ What if relevance is what’s important?♥ Mapping between memorable and relevant♥ Useful related work on implicit feedback
(Done)
Re-search v. Search
♠ Why I think people repeat searches♥ Information seeking literature♥ Re-finding consistently reported as a problem
♠ How to prove♥ Study shows prefer to follow known paths♥ Search log analysis
♠ What if people just want to search?♥ Memorable results ranked first♥ Other domains where list consistency matters
(Done)
Merge v. Identify Old and New
♠ Why I think results should be merged♥ Information need not necessarily one or other♥ People don’t like to do extra work
♠ How to prove♥ Search log analysis♥ Look at what people do in longitudinal study♥ Lab study – timing becomes an issue
♠ What if people want to identify query type?♥ Other applications where merging is useful
(Done)
Results Change v. Stay the Same
♠ Why I think results change♥ How search engines work♥ Personalization and dynamic content
♠ How to prove♥ Track query results
♠ What if results don’t change?♥ Probably will in future applications♥ Existing applications where lists change
(Done)
Forget v. Remember as Forgettable
♠ Why I think people forget♥ Visual analogy
♠ How to prove♥ Lab study – Do people find new information?♥ Longitudinal study – Ever click on new result?
♠ What if remember as forgettable?♥ Build better model of memorability♥ Highlight important changes
(Done)
Implementation Issues
♠ Page of cached result may disappear♠ Multiple result pages♠ Identifying repeat queries
♥ User identified♥ Search sessions are not repeat queries♥ Exact query may be forgotten
(Done)
Evaluation Issues
♠ Various goals to test♥ Does a merged list look like the original?♥ Does merging make re-finding easier?♥ Is search improved overall?
♠ Lab study♥ How to set up re-finding task?♥ Timing differences significant enough?
♠ Longitudinal study – What to measure?♠ What are good baselines?
(Done)
Jaime [email protected]
Why Do People Orienteer?
♠ Easier than saying what you want♠ You know where you are♠ You know what you find
♠ The tools don’t work
Query Changes
♠ Most changes are simple♥ Capitalization♥ Phrasing♥ Word ordering♥ Word form♥ New queries shorter
♠ What about longer time horizons?♠ Recognition v. recall
Result List Changes
♠ Tracked 10 queries on Google for a year+♠ 1.18 of top 10 disappear each week♠ Rate of change likely to increase, e.g.:
♥ Raw personalization♥ Relevance feedback
♠ People forget their queries♥ 28% of queries forgotten within an hour