Measuring Our Relevancy:Comparing Results in a Web-Scale Discovery Tool,

Google & Google Scholar

Elizabeth Namei, University of Southern California Libraries, namei@usc.edu

and

Christal Young, University of Southern California Libraries, youngc@usc.edu

#summonrelevancy ACRL 2015 Conference

unexpectedresults

“I could throw an author

and title in [Summon] but

I know that works very

erratically….I often get

inundated by irrelevant

things…”

“[Summon] just doesn’t

generally turn up reliable results

for me. I’ll search a very

obvious keyword or a very

specific keyword and it won’t

turn up the most relevant results

first even though I know the

highly relevant results are in

there...”

English professor

Senior music major

In search of evidence

methods

We pulled out a random

sample of 384 queries

There were 1.2 million searches entered in Summon in Fall 2014.

From the 384 queries we eliminated:

● 63 queries for known items that USC did not

own

● 22 queries that had unrecognizable or

foreign characters

● 21 queries that were for “non-scholarly”

formats

Our

final

sample

size =

278

defining relevance

1. Relevant: a match for the known item shows up 1st or 2nd in the list of results.

2. Partially Relevant: a match for the known item is found 3rd-10th in the list of results.

3. Not relevant: known item is not listed in the first 10 results or no results were returned.

results

76%

91%79%

24%

100,000

(69/278)

(183/278)

(9/278)

(11/278)

(6/278)

Partial Citation queries (with 2 metadata elements) that did not find relevant results in Summon

and then there were none agatha christie

palladio four books on architecture

showed up 74th

shows up 75th

Quotes were used in only 6% of the queries (16/278)

but……

100% of these queries returned relevant results

Formatted citations (with 3 or more metadata elements) that did not find relevant results in Summon

Block G Hartman AM Dresser CM Carroll

MD Gannon J Gardner L. A data-based

approach to diet questionnaire design and

testing. Am J Epidemiol 1986; 124:453-469.

does not show up in first 10

results (6 metadata elements)

no results returned

(9 metadata elements)

Brooks M. K. (2010). Hospice services: The

dilemmas of technology at the end of life. In

T. S. Kerson J. L. M. McCoyd & Associates

(Eds.) Social work in health settings: Practice

in context (3rd ed. pp. 235-246). New York

NY: Routledge.

if fewer metadata elements had been used...

Block A data-based approach to diet

questionnaire design and testing. shows up 1st

(2 metadata elements)

Shows up 1st

(5 metadata elements)

Brooks M. K. (2010). Hospice services: The

dilemmas of technology at the end of life. In

T. S. Kerson J. L. M. McCoyd & Associates

(Eds.) Social work in health settings: Practice

in context

where do we

go from here?

Continue teaching users to be more strategic

searchers● Explain the whys of strategic

searching and not just the hows

● Explain the organization of information systems and the benefits field searching (especially for one word/common word titles)

● Provide troubleshooting tips for how to deal with the inevitable failed searches

follow the user

Push for In-house solutions● Using APIs ● Developing programs to

“clean” problematic queries

Lean on vendors• to improve relevancy • to improve “Did you mean”

suggestions• To provide query

reformulation suggestions

Create Smarter Library Systems

personalization

questions?comments?

Image creditsSlide 1 image https://flic.kr/p/9pfXxqSlide 2 image: http://bit.ly/1NkcLNJSlide 4 image: http://bit.ly/1G0s5fYSlide 5 image: http://bit.ly/1DQRREdSlide 6 image: http://thevisualharvest.com/lib/img/elements/to-the-drawing-board-graphic.svgSlide 8 image: http://relevantinfo.org/wp-content/uploads/2013/09/good-o-meter.jpgSlide 9 images:

● Jeopardy: http://www.ibm.com/smarterplanet/us/en/ibmwatson/assets/img/tech/watson_on_jeopardy.jpg● Summon: http://www.almanhal.com/ResourceImages/Publisher/StartigicPublisherLogos/ProQuest.png● Google: http://bit.ly/1I0b0RZ● Google Scholar: http://www.redcad.org/members/hadjkacemm/images/GS2.png

Slide 12 image: http://wishflowers.tumblr.com/post/10244886760Slide 16 image: http://bit.ly/1EdFthISlide 21 image: http://www.healthcarereformmagazine.com/voluntary-benefits/where-do-we-go-from-here-voluntary/Slide 22 image: http://samanthashorey.blogspot.com/2011_02_01_archive.htmlSlide 23 image: http://blog.cityspoon.com/2012/02/08/gathering-followers/Slide 24: Image: http://www.csfieldguide.org.nz/ArtificialIntelligence.htmlSlide 25: image: http://bloomreach.com/snapSlide 26: image: http://www.gtcw.org.uk/gtcw/index.php?lang=cy