Discovery study detailed results 2014 december

Discovery or Displacement?A Large-Scale Longitudinal Study of the Effect of Discovery Systems on Online

Journal Usage

December 2014

Combined Presentation

Michael Levine-Clark, University of Denver

John McDonald, University of Southern California

Jason Price, SCELC Consortium

…our customers insist that usage of our content

decreased after implementation of discovery service “X”.

A publisher told us . . .

https://www.flickr.com/photos/kongping/7192138660/in/gallery-flickr-72157645846953449/

Librarians speculate . . .

…of course discovery vendors direct their users to their own aggregated content.

http://www.flickr.com/photos/

hypotekyfidler/15012731920

Goals of our research

• Determine whether discovery services impact usage

• Help librarians and publishers understand how their choices impact use

• Help librarians, publishers, and vendors improve the discovery experience for end users

Why Do We Use Discovery Services?

• Too many sources of information

– Specialized

– Confusing

– What’s the right specialized database for my subject?

• Why do you search in one place for an article, another place for a different article, another for a book?

Terminology

Discovery Tool = Discovery Layer = Discovery Service = Discovery System

Web-scale discovery services

• Single source for finding information– Books

– Articles

– Local content

• Metadata and/or full text

• Content is pre-indexed and/or pre-harvested

• Single fast search

ILS

HathiTrust

MLA Bibliograph

y

Institutional Repository

Publisher Metadata

Discovery Service

Does implementation of a discovery service impact usage of publisher-

hosted journal content?

What did we measure?

• Whether there is an effect

• NOT why that effect exists (that’s a future study!)

Caveat 1: Publisher-hosted journals are only part of the picture

eBooks, pBooks, newspaper articles, aggregatorjournal content, etc.publisher journal content

The six publishers in this study

Not to scale!

52.3

12.4

1

2.8

32.5

Open web search

Library referrals

Social media

Academic

N/A

Journals Traffic Sources (SAGE, Conrad ALPSP 2013)

An assumption

• At any given institution, given a relatively stable user base, the total search effort will remain roughly the same.

Discovery services

Will take up an increasing amount of a finite time for searching

Will draw users from other (more or less efficient) search tools

Will alter the overall productivity of searches (users will find more or less)

Will alter the overall efficiency of users (users will access more or less full-text)

Caveat 2: More usage may not be better!

• Decreased usage might be a sign of greater efficiency– Relevant articles found faster = fewer articles to examine

OR

– Fewer articles examined because other relevant content types found

Did Journal Usage Change? (and if so, to what extent?)

• 4 discovery services– 6 libraries in each group

• A control group– 9 libraries that did not implement a discovery

service in this time period

• 6 major journal publishers

– 9,206 Journals in the study

– 163,545 Observations (Library + Journal)

Participating libraries

• 157 asked for permission, 155 granted permission– 124 from the US, 33 from other English-speaking countries

• Has your library used a different discovery service in the past?– Only libraries answering “No” were selected

• Is your discovery service featured on your homepage?– All participants answered, “Yes, with a search box”

• To what extent did your library market the discovery service at its release?– 4 said “None”, one from each vendor

– 12 said “A limited extent”, 2 WCL, 2 EDS, 4 Summon, 4 Primo

– 8 said “A significant extent, 4 WCL, 2 EDS, 1 Summon, 1 Primo

Dataset• 33 Libraries

– 28 US, 2 CA, 1 each from UK, AUS, NZ

– WorldCat book holdings

> Average: 1,114,193 ; Range: ~300k to ~2.6mil

– 4 discovery groups, of 6 libraries each

– 1 control group, 9 libraries• Implementation dates (Discovery Libraries):

> 2010 (3), 2011 (19), 2012 (2)

• 6 Publishers

• 9,206 Journals

• 163,545 Usable Observations

Methodology

Compared COUNTER JR1 total full text article views for the

12 months before vs 12 months after implementation date

Jun

e 2

01

0St

art

Imp

lem

enta

tio

nM

ay 2

01

1

May

20

12

End

Year 1 Year 2

Included implementation month in Year 1 to ensure that

both periods included an entire academic year

Examine Data for Outliers

Analyzing Usage Change: % vs Total

Use 12 months before

Use 12 months

after% Change

TotalChange

Journal A 500 600 20% 100

Journal B 5 15 200% 10

Which is the better measure?

Is it the same for publisher- & journal-level data?

Observations by Publisher

Tota

l S

tudent

FT

E

Journals by Library & Service

EDS Primo Summon WorldCat Control

Average Journal Usage by Library

Testable Effects

• Discovery Service

– Implemented by multiple libraries

– Used to find content from all publishers

• Publisher

– Accessible in all discovery services

– Accessible across all libraries

• Library

– Uses content from multiple publishers

– Uses only one discovery service

Full Model

Including Discovery Service, Publisher, and Library

Including Discovery Service, Publisher, and Library

Nested ANOVA Model Results

How does usage change differ across discovery services?

A

BB

C

D

Letters indicate statistically significant differences (Tukey multiple comparisons, p <

.05)

How does usage change differ across publishers?

Publisher (sorted by Mean Change)

C

Letters indicate statistically significant differences (Tukey multiple comparisons, p <

.01)

D

BBB

A

How does usage change differ across publishers?

Does usage change vary across libraries?

Institution (sorted by Mean Change)

Usage Change Per Institution: All Journals

Control EDS Primo Summon WCL

Usage Change By Institution: Pub 1

6.32

Primo WCLSummonEDSControl




WCLEDS PrimoControl Summon


WCLEDSControl Primo Summon

Usage Change Per Institution: Pub 5


Usage Change by Institution: Pub 6

WCLSummonPrimoEDSControl

Publisher 1 by Discovery System






Conclusions

1. Each discovery system caused an increase in usageof publisher-hosted journal articles

– The size of the increase differed across discovery systems

2. Usage change differed among libraries using the same discovery system

3. Discovery systems had stronger effects on some publisher’s content than on others

4. [These results provide a snapshot of a very small portion of the content that discovery services are intended to expose, and should not be used to assess discovery service quality]

Summary of Results

• Discovery Service

– Every service increased usage compared to control

– Some services increased usage more than others

• Library

– The degree of usage change differed among libraries using the same discovery service

• Publisher

– Usage change differed across publishers:

• 1 of 6 publishers saw a significant decrease

• 2 of 6 publishers saw significant increases

• 3 of 6 publishers saw no detectable change

Conclusions to be avoided…

Our research does not indicate that:

a) one service increases usage more than another for every library or publisher

effects vary across libraries and publishers

b) one service is better than another

libraries or their users may benefit from increased usage of other content instead

higher usage may indicate lower efficiency

Next Steps

• Design & test for effects of:

– Aggregator full text availability

– Linking configuration options in discovery services

• Expand pool of libraries

• Explore the why?

• Other possibilities

– Journal Subject?

– Journal age (archive vs current)?

– eBook usage?

1. What is the best service: Summon, Primo, EDS or Worldcat?

• Our research can’t prove one is better than another, and usage is only one reason to install a discovery system

2. Why was there so much variation in the effect of implementation across publishers at my institution? Why did some increase and some decrease?

Potential Librarian Concerns (1)

3. What about configuration differences?• This undoubtedly contributes to some variance, so be

careful to review and maintain your configurations and preferences.

4. What about other resource types (ebooks, print books, etc)? • They may be affected, we haven’t studied these.

5. Was it missing metadata that caused the differences?• Maybe, we can’t know for sure, so the publishers and

vendors need to come up with best practices for metadata exchange to ensure everyone is on an equal playing field (encourage participation in NISO ODI)

Potential Librarian Concerns (2)

Potential Publisher Issues (1)

1. Are users being directed to Aggregated full text before publisher hosted full text?

• Maybe, but it might be the Library’s configuration that is to blame.

• This could be counterproductive for aggregators.

2. Does implementation of abstract-based discovery reduce the ranking of publisher content that is not indexed (or available full text) in a library’s aggregator databases?

• Maybe, but it its up to the Library to decide if that is a desired effect or not.

Potential Publisher Issues (2)

3. Why was there so much variation in the effect of implementation on change in usage of our content across institutions using the same discovery service?

• Because usage can affect the publisher bottom line, this is a key question. Some publishers are finding it of value to invest in research and engagement of customers at the extremities of change after implementation.

4. How can publishers know what is being done with the metadata they send?

• Discovery vendors need to be more proactive in proving the positive benefits of providing more robust metadata

1. How do we prove we’re content neutral?• Develop best practices that indicate to libraries their

configuration and linking choices.• Allow for independent studies• Make the case for the risks related to bias?

2. Our systems make other resource types (ebooks, print books, etc) more discoverable, does anyone value that? • More research needs to be done to assess

effectiveness for other content types

3. Does increased usage necessarily mean a discovery system is better? • Definitely not! Depends on what a library values.

Potential Discovery Vendor Concerns

What is the value of increased usage?

• For publishers and libraries:

– lower cost per use

– greater return on investment

– but there may be usage tradeoffs with other content types

• For users, usage decreases might be preferable

– Decreased usage might be a sign of greater efficiency

• Relevant articles found faster = fewer articles to examine

OR

• Fewer articles examined, but other (more?) relevant content types found

Long Term Goals

1. Determine whether discovery services impact usage of various content types

• Journal articles first, but more to come

2. Reveal key factors that explain why some of the differences exist

3. Help librarians and publishers understand how their choices impact use

4. Help librarians, publishers, and vendors improve the discovery experience for end users

Next Steps

• Design & test for effects of:

– Aggregator full text availability

– Journal age (archive vs current)

– Journal Subject

– Overall usage trends

– Configuration options in Discovery services

• Expand pool of libraries

• Explore WHY

Questions?

Michael Levine-Clark

[email protected]

John McDonald

[email protected]

Jason Price

[email protected]

mailto:[email protected]



Education

Discovery study detailed results 2014 december