Modeling User Searching Behaviors and Search Assistance Usage via Transaction Logs: Everything You Know is Wrong

May 2012

Mary C. Schlembach, William H. Mischo, Joshua Bishoff, Elizabeth M. German

Grainger Engineering Library Information CenterUniversity of Illinois at Urbana-Champaign

Today’s Library Gateways• Academic libraries are moving from federated

broadcast search systems to webscale discovery systems (WDS) with centralized aggregated indexing

• Web 2.0 library systems give users single-entry search boxes with faceted limiting operations

• As we transition to WDS, understanding and supporting user search behaviors is important

What do we know about User Searching?• Web search engine studies show: 2.4 terms per

query, 72% of sessions are single queries• There is little use of advanced features• OPAC studies give contradictory information on

the types of searches that users prefer• It appears that the system default search is the

most heavily used search type, whether it is keyword, subject, or title

• Markey 2007

Our Study of Custom Transaction Log Data • We conducted an evidence-based study of user

behavior in the Easy Search University of Illinois Gateway

• We looked at total of 1.4 million searches and 1.5 million clickthroughs performed by users in Fall 2010 and Spring 2011 semesters

• These transaction log studies were supported by National Science Foundation and Institute of Museum and Library Studies

University of Illinois Library Gateway• Revised Gateway portal introduced in September

2007, powered by Easy Search federated search system:oRecommender and discovery systemo Employs Search Assistance mechanismsoHelps with search strategy modification and

navigationo Takes users into native interfaces at point of

completed searchoWrites out custom transaction logs

Easy Search Examples

Easy Search Examples

Transaction Log Format

Transaction Logs• 3.5 million user actions, including 1.7 million

searches in 2011 • Transaction logs are used in the development

of context-specific and adaptive search assistance for users

• Our transaction log analysis is complemented with user interviews and surveys

• Relational database of two tables with analysis using SQL queries

Search Assistance Features• Context-specific suggestions and

prompts for:– Spelling changes – Redo as author search– Direct links to frequent or special search terms

or Libguides– Links to matching e-journal title(s)– Links to citation linker (JAL) for full-text

retrieval– Limit results to exact phrase, title word(s), or

title phrase– Dark background searches displayed if needed

Additional Search Assistance Features• List of related terms• Performing additional phrase and title searches

(multiword search arguments – ISI study• Displays e-journal and database list results• Pass-through command language search• Ask-a-Librarian live link• Additional target searches are performed based

on user query (Open WorldCat, LC Classification)

• Links to e-book titles displayed on OPAC results line

Search Starting PointsSearch Starting Points:

Percentage of All Searches:

Easy Search tab (default) 51.6%

Departmental Library Site Search

14.3%

Journal Title Tab 12.8%

Books Tab 12.6%

Journal Article Tab 5.1%

Advanced Search Tab 3.41%

• 65.9% of searches are from the single entry search boxes on the default Easy Search tab or on a departmental library site.

Search Starting Points• 56.35% of the gateway use is from off-

campus • 39.26% of all searches are follow-on

searches performed from the pull-down search menu on the results page  

• A large percentage of searches – almost 66% -- are default keyword searches

• But there is also some willingness to go to facet-like tabs for subset searches

Known Item Searches

• The increase in specific journal article searches is partially due to users entering ‘cut and paste’ citations into Easy Search

  2007 Study 2011 Study

Total Number of Searches Examined

3,100 8,474

Overall % of Known-Item Searches 49.4% 51.2% of Searches54.8% of Sessions

Percent of the 49.4% Percent of the 51.2%

Author/Title Searches 7.4% 6.04%

Author Searches 28.9% 16.9%

Book/Monographic Title 40.5% 33.6%

Index/Abstract Title 6.8% 4.2%

Specific Journal Article Search 5.7% 26.9%

Specific Journal Title Search 11.8% 12.4%

Examples • Supporting Annotation as a Scholarly Tool—Experiences From the

Online Chopin Variorum Edition• The Sound of Silence in Online Feedback: Estimating Trading Risks

in the Presence of Reporting Bias• Gait recognition using multi-bipolarized contour vector• Matouschek, Kellis, Serrano, Fersht Nature• kinsella & phillips 2005• Poor Historiraphy: the Poorest in American Higher Education• Social capital, income inequality, and firearm violent crime• Miller, K. (2009) Organizational Communication: Approaches and

Processes. 5th edition• university, need for money• J vet emerg crit care

Searching Characteristics • 1.4 million searches in the two

semesters• 66% of searches are default keyword

searches and 51% are known item searches – this has to be addressed

• 637,375 search sessions for an average of 2.19 searches per session

• 48% of sessions contain more than one query

Terms Per Query

• 4.33 average words per query. Longest query was 276 words. Total of 4,077,152 words used in the 942,027 sample searches

Number of Words in User Search Query

Frequency of Occurrence

1 102,0282 212,0723 219,4664 127,8755 77,1266 47,3157 32,0388 24,464>8 words 99,643   Average 4.33 words per query

More Search CharacteristicsCharacteristics of User Searches

Percentage of all Searches

Boolean Operators (“and”, “or”, “not”) entered13.03%

AND operator searches (the vast majority of ANDs appear as conjunctions in known titles)

12.76%

AND operator in all caps 0.41%OR operator-- lower and upper case entered

0.22%

NOT operator-- lower and upper case 0.13%Commas entered 6.03%Parentheses entered 0.88%Quotes (single and double) entered 3.95%Prepositions entered 27.26%“+ ” sign entered 0.42%Truncation symbol(*) entered 0.12%

Search AssistanceSearch Assistance Suggestions made by

SystemPercentage of all search sessions

Reduce results to exact phrase/title words 36.43%

Matching Journal Title found 17.79%

Spelling change suggested 12.61%

Redo as Author search suggested 7.69%

Direct Link/frequent search result found 6.09%

Citation entered – go to linker module 4.11%

• In 38.6% of all searches and 54.8% of all search sessions, the system made an explicit search assistance suggestion to the user

Search Assistance

• Spell change suggestions, direct links to frequent/special searches, journal title match links, and article phrase and title links are heavily used

Search Assistance Suggestions Employed by Searchers

Percentage of times each SA type was used

when suggested

**Percentage of Multi-

Query sessions

where SA was used

Spell change suggestions clicked 29.04% 7.86%Direct link clicked 57.56% 6.58%Journal Title match click 21.41% 6.95%Limit to Phrase/Title or clicked on title results 28.38% 15.76%Author Redo offer clicked 7.95% 1.36%Ask-a-Librarian clicked 0.14% 0.29%Complete citation passed to linker module 2.4% 0.25%

Title Searching • Journal title matches were clicked on

21.4% of the time they were presented, or in 6.95% of all search sessions

• Additional 115,281 sessions where users chose the “Search Journal Titles” tab

• Added article and book title links were clicked on in 7.35% of all searches and 11.54% of all sessions

• Title match search and display is common

Title Searching Issues • Journal name, journal article, and book

title searches are frequently performed

• WDS problem: these title matches do not typically appear as facet items

• WDS problem: in default keyword search mode, retrieval of title matches is at the mercy of the relevancy ranking and exact matches may be buried within long lists

Title Searching Issues (2) • WDS problem: when specific item is

not indexed, WDS will still return long lists of result matches from keyword or full-text matches

• WDS problem: webscale systems do NOT offer contextual or adaptive search assistance prompts  

Search Assistance Usage • We found that users employed our

search assistance mechanisms in 32.45% of all search sessions

• In 57.98% of the search sessions involving more than one query, the user clicked on or utilized one of our offered search suggestions, reformulation options, or relevant resource recommendations 

Clickthroughs to TargetsSpecific Resource Total Clickthroughs

Ebsco Combined Database Search 346,027UI Voyager Catalog 184,015UI VuFind Catalog 147,445Scopus 117,041ISI Web of Knowledge 80,891InfoTrac 65,649Statewide VuFind Catalog 51,409Google Books 33,303Springer e-Books 29,243E-Resource A-Z List 28,918Compendex 22,650Background Target Searches 19,294Elsevier e-Books 17.062Wiley e-Books 16,655ABI Inform 14,967INSPEC 13,772Google Scholar 11,065Newspaper Source 10.158HathiTrust 8,320

Clickthroughs by Category

• Clickthroughs into e-book content targets were 7.87% of all target clicks in 11.36% of all sessions

User Clickthroughs by Category

Percentages

Article databases 47.65%

All books 33.01% (OPACs 26.01%, E-Books 7%)

Journal/Database titles 17.41%

Web search engines 0.72%

Newspapers and news sites 0.87%

Reference titles 0.35%

What Works Best? • Users vote with their mice:

– Spell check – YES– Direct Link – infrequent but high clickthrough– Added title and phrase results – YES– Journal title matches - YES– Related terms – NO– Pull-down 10 hits – NO– E-Books from OPAC custom search – YES– Ask a Librarian – Not often but important

What Have We Learned?• The search assistance (SA) features

transparently provide users with advanced search functionality

• SA suggestions are made in 38.6% of all searches and 54.8% of all search sessions

• SA is used in 32.45% of all sessions and 57.98% of sessions with more than one query

Future Considerations• Transaction log analysis

can inform design decisions, but ...

• How useful is a blended display approach in WDS?

• We are working with Ex Libris and other vendors on incorporating search assistance into WDS