SUSHI and COUNTER

SUSHI and COUNTER

Understanding the Data Around Us: Gathering and Analyzing Usage Data

NISO Usage Data Forum Dallas, TX

November 1, 2007

Oliver PeschEBSCO Information Services

[email protected]

Overview

• Background

• Update on COUNTER

• SUSHI

• Looking ahead

Overview

• Background


• SUSHI

• Looking ahead

Usage Statistics

• Usage data importance grows with e-collections- Collection management- Budget management

Usage Statistics


• Credibility and consistency…- Different vendors using different terminology- Inconsistencies in processing lead to over counting- Formatting differences make comparison challenging

Usage Statistics


• Credibility and consistency…- Different vendors using different terminology- Inconsistencies in processing lead to over counting- Formatting differences make comparison challenging

…COUNTER…

Usage Statistics


• Credibility and consistency…- Different vendors using different terminology- Inconsistencies in processing lead to over counting- Formatting difference comparison challenging

• Consolidation and meaningful reporting…- Many vendors and reports to process- Collection-level views needed

…COUNTER…

Usage Statistics



• Consolidation and meaningful reporting- Many vendors and reports to process- Collection-level views needed

…COUNTER…

…Usage Consolidation tools (ERM)…

Usage Statistics




• Retrieving and processing…- Obtaining reports is time consuming- Formatting and other adjustments still needed

…COUNTER…


Usage Statistics




• Retrieving and processing…- Obtaining reports is time consuming- Formatting and other adjustments still needed

…COUNTER…


…SUSHI…

Overview

• Background


• SUSHI

• Looking ahead

Update on COUNTER

Peter Shepherd

Director

COUNTER

October 2007

Background

• Understanding usage

- Different approaches

- Role of usage statistics

• Usage statistics

- Should enlighten rather than obscure

- Should be practical

- Should be reliable

- Are only part of the story

- Should be used in context

• COUNTER

- Achievements

- Current status

- Future challenges

So how are we getting there?

• ICOLC Guidelines for statistical measurement of usage of web-based information resources

• National Commission on Libraries and Information Science (NCLIS) Electronic access and use-related measures

• NISO – Z39.7 (Library Statistics)

• ISO – 2789 (library statistics) and 11563 (library performance measures)

• MESUR – investigate metrics derived from the network-based usage of scholarly information

• COUNTER (Counting Online Usage of NeTworked Electronic Resources)

Why COUNTER?

• Goal: credible, compatible, consistent publisher/vendor-generated statistics for the global information community

• Libraries and consortia need online usage statistics- To assess the value of different online products/services

- To support collection development

- To plan infrastructure

• Publishers need online usage statistics- To experiment with new pricing models

- To assess the relative importance of the different channels by which information reaches the market

- To provide editorial support

- To plan infrastructure

COUNTER Codes of Practice

• Definitions of terms used

• Specifications for Usage Reports

- What they should include

- What they should look like

- How and when they should be delivered

• Data processing guidelines

• Auditing

• Compliance

COUNTER: current Codes of Practice

1) Journals and databases- Release 1 Code of Practice launched January 2003

- Release 2 published April 2005 replacing Release 1 in January 2006

- Now a widely adopted standard by publishers and librarians

- 70 vendors now compliant

- 10,000+ journals now covered

- Librarians use it in collection development decisions

- Publishers use it in marketing to prove ‘value’

COUNTER: current Codes of Practice

2) Books and reference works- Release 1 Code of Practice launched March 2006

- 8 vendors now compliant

- Relevant usage metrics less clear than for journals

- Different issues than for journals

• Direct comparisons between books less relevant

• Understanding how different categories of book are used is more relevant

Journal and Database Code of Practice

Usage Reports• Journal Report 1

- Full text article requests by month and journal

• Journal Report 2- Turnaways by month and journal

• Database Report 1- Total searches and sessions by month and database

• Database Report 2- Turnaways by month and database

• Database Report 3- Searches and sessions by month and service

Code of Practice for books

• Book Report 1- Number of successful requests by month and title

• Book Report 2- Number of successful section requests by month and title

• Book Report 3- Turnaways by month and title

• Book Report 4- Turnaways by month and service

• Book Report 5- Total searches and sessions by month and title

• Book Report 6- Total searches and sessions by month and service

Journal Report 1Full text article requests by journal

Html and PDF totals reported separately

COUNTER Audit

• Independent audit required within 18 months of compliance, and annually thereafter

• Audit is online, using scripts provided in the Code of Practice

• Auditor can be:- Any Chartered Accountant

- Another COUNTER-approved auditor

• ABCE is the first COUNTER-approved auditor- Industry-owned

- Not-for-profit

- Independent and impartial

- Part of ABC (Audit Bureau of Circulations)

- Providing website traffic audits for over 150 companies and certifying over 1400 domains

- Have successfully completed test audits on COUNTER usage reports

ABCE Audit fees

FEES CHARGEABLE Standard FeeCOUNTER

Members

Year 1 Year 2 Year 1 Year 2

[A] Registration Fee £1,556 £612

[B] Annual Subscription £852 £852 £621 £621

Journal Report 1 £1,480 £1,480 £1,480 £1,480

Journal Report 2 £740 £740 £740 £740

Database Report 1 £1,110 £1,110 £1,110 £1,110

Database Report 2 £740 £740 £740 £740

Database Report 3 £370 £370 £370 £370

Total cost of Journal Report 1 audit £3,888 £2,332 £2,713 £2,101

COUNTER: deriving metrics from Journal Report 1

• Local metrics

- For libraries and library consortia

- At journal, collection and publisher level

- To compare the cost-effectiveness of journal subscriptions

- To assess the value of Big Deals

• Global metrics

- For authors, funding agencies, libraries and publishers

- At journal, collection and publisher level

- To compare quality and value

COUNTER: ‘local’ metrics

• JISC (UK Joint Information Systems Committee)

- Funded by UK higher education funding councils

- Supports higher education in the use of information and communications technologies (ICT)

• Access to information and communication resources

• Advice on creation and preservation of digital archives

• Implications of using ICT

• Network services and support

• Research to develop innovative solutions

• National overview of online journal usage

- Develop a reliable, widely applicable methodology

- Use COUNTER Journal Report 1 ‘article full-text requests’

Local metrics: an example

• COUNTER data was analysed in relation to:

- usage range

- Price band

- Subject category

• Metrics derived from this analysis

- Trend in number of full-text article downloads

- Full text article requests per title

- Full text article requests per publisher package

- Full text article requests per FTE user

- Most requested titles

- Usage of subscribed vs.. unsubscribed titles

- Cost per full-text article downloads

- Cost per FTE user

• Summary report available at:

www.ebase.uce.ac.uk/projects/NESLi2.htm

Local metrics: an example

• Growth in full-text article downloads

- Publisher A: 12%- 208%

- Publisher B: 12%- 59%

- Publisher C: 23%- 154%

- Publisher D: 22%- 81%

• Cost per full-text article download

- Publisher A: £0.97- £5.26

- Publisher B: £0.70 - £2.91

- Publisher C: £0.80 - £3.29

- Publisher D: £0.45 - £2.26

COUNTER: ‘global’ metrics

• Impact Factor

- Well-established, easily understood and accepted

- Endorsed by funding agencies and researchers

- Does not cover all fields of scholarship

- Reflects value of journals to researchers

- Over-emphasis on IF distorts the behaviour of authors

- Over-used, mis-used and over-interpreted

• Usage Factor

- Usage-based alternative perspective

- Would cover all online journals

- Would reflect value of journals to all categories of user

- Would be easy to understand

COUNTER: Membership

• Member Categories and Annual Fees (2008)

- Publishers/intermediaries: $825

- Library Consortia: $545

- Libraries: $412

- Industry organization: $412

- Library affiliate: $165 (non-voting member)

• Benefits of full membership

- Owner of COUNTER with voting rights at annual general meeting, etc.

- Regular bulletins on progress

- Opportunity to receive advice on implementation

- Vendors: no compliance fee; reduced price audit fees

http://www.projectcounter.orghttp://www.projectcounter.org

Apply for COUNTER membership

Apply for COUNTER membership

Overview

• Background


• SUSHI

• Looking ahead

SUSHI: Objectives

• Solve the problem of harvesting and managing usage data from a growing number of providers.

• Promote consistency in usage formatting (XML)

• Automate the process

SUSHI: What is it?

• An XML Message

• Methods to transfer the message between two systems

• Implemented as a Web service

• Using SOAP (Simple Object Access Protocol)

SUSHI : The Exchange

Report Request

<Requester>

<Customer Reference>

<Report Definition>

Report Response<Requester>

<Customer Reference>

<Report Definition>

<Report as payload>

SUSHI: Architecture

• The next series of slides graphically show a SUSHI transaction

- Library system requests a usage report

- SUSHI client makes the request

- SUSHI server processes request

- SUSHI server prepares COUNTER report

- SUSHI server “packages” and returns response

- SUSHI client processes COUNTER report

Content ProviderLibrary

Internet

The Library and Content Provider’s systems are both connected to the internet.

The Library and Content Provider’s systems are both connected to the internet.


SUSHIClient

Internet

The SUSHI client is software that runs on the library’s server, usually associated with an ERM system.

The SUSHI client is software that runs on the library’s server, usually associated with an ERM system.

ERM


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

The SUSHI server is software that runs on the Content Provider’s server, and has access to the usage data.

The SUSHI server is software that runs on the Content Provider’s server, and has access to the usage data.


SUSHIServer

(web service)

UsageData

SUSHIClient

Request

Internet

ERM

When the ERM system wants a COUNTER report, it sends a request to the SUSHI client, which prepares the request.

When the ERM system wants a COUNTER report, it sends a request to the SUSHI client, which prepares the request.

?


SUSHIServer

(web service)

UsageData

SUSHIClient

Request

Internet

ERM

The SUSHI request is sent to the Content Provider. The request specifies the report and the library the report is for.

The SUSHI request is sent to the Content Provider. The request specifies the report and the library the report is for.

?Request


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

The SUSHI server reads the request then processes the usage data.

The SUSHI server reads the request then processes the usage data.

?Request


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

The SUSHI server creates the requested COUNTER report in XML format.

The SUSHI server creates the requested COUNTER report in XML format.

?

COUNTER


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

A response message is prepared according to the SUSHI XML schema.

A response message is prepared according to the SUSHI XML schema.

?

COUNTER

Response


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

The COUNTER report (XML) is added to the Response as its payload. The response is sent to the client.


?Response

COUNTER


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM



?Response

COUNTER


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

The SUSHI client processes the response and extracts the COUNTER report.

The SUSHI client processes the response and extracts the COUNTER report.

?Response

COUNTER


SUSHIServer

(web service)

UsageData

SUSHIClient

Internet

ERM

The extracted COUNTER report is passed to the ERM system for further processing.

The extracted COUNTER report is passed to the ERM system for further processing.

COUNTER

SUSHI: from concept to standard in record time!

• The timeline in brief…

- Nov. 2004 - Meetings between Cornell & Innovative

- July 2005 – Cross-Industry Committee forms:Libraries; ILS vendors; Content providers

- Fall 2005 – Technical discussions

- Winter 2006 – Live harvests

- Spring 2006 – NISO involvement

- September 2006 – Draft standard for trial use

- May 2007 – Successful trial period ends

- Sept 2007 – Z39.93 passed NISO ballot

- October 2007 – Z39.93 certified by ANSI

NISO SUSHI Working Group

• Adam Chandler (co-chair), Cornell

• Oliver Pesch (co-chair), EBSCO Information Services

• Patricia Brennan, Thomson Scientific

• Ted Fons, Innovative Interfaces, Inc.

• Bill Hoffman, Swets Information Services

• Tim Jewell, University of Washington

• Ted Koppel, Ex Libris

SUSHI: Contributing Partners

Founding Members:

• EBSCO

• Ex Libris

• Innovative Interfaces, Inc.

• Swets Information Services

• Thomson Scientific

Newer members:

• Endeavor Information Systems

• Florida Center for Library Automation

• College Center for Library Automation (CCLA) from the State of Florida Community Colleges

• Otto Harrassowitz

• OCLC

• Project Euclid

• Serials Solutions

• SirsiDynix

Overview

• Background


• SUSHI

• Looking ahead: COUNTER

COUNTER: Current issues

• Interface effects on usage statistics- E.g. downloading HTML and PDF of the same article in one

session- COUNTER has tested data filter solutions, but what does the

duplicate downloading signify?

• Reporting separately purchasable digital archive usage- Currently all usage for a journal is usually reported together- Separately purchasable archives mean we need separate

reports for archival content, or a year of publication breakdown of usage

• Usage in Institutional Repositories- Growth in Institutional Repository (IR) content- Need for credible IR usage statistics- IR usage statistics already being collected, but no standards

• Improving consortial usage reports- Current usage reports inadequate- New reports in XML format

COUNTER: Future challenges

• Improving/extending the Codes of Practice

- Reliability ( audit, federated searches, prefetching)

- Usability (number of compliant vendors, XML format, additional usage reports)

- Additional data (year of publication, article level reports)

- Categories of content (Institutional Repository content)

• Deriving metrics from the Codes of Practice

- Journals (cost per use, Usage Factor)

- Databases?

- Books?

COUNTER: Next steps

• Release 3 of Code of Practice for Journals/Databases

- Features: prioritisation on basis of demand and practicality

- Process: consultation via focus groups,etc; publication of draft CoP

- Timetable: focus groups November/December 2007; draft R3 in early 2008; final R3 in mid-2008; implementation of R3 in 2009

• Release 2 of Code of Practice for Books

- Review R1 in practice

• Other categories of content ( eg Institutional Repositories)

• Metrics derived from the COUNTER usage statistics

- Cost per use

- Usage Factor

Overview

• Background


• SUSHI

• Looking ahead: SUSHI

SUSHI: Looking ahead

• Growing adoption…

• COUNTER Release 3

- Consortia reports

- New COUNTER Schema

- SUSHI compliance

• Extendible design

- Other reports

- Other “payloads”

Overview

• Background


• SUSHI

• Looking ahead: Usage Factor

Usage Factor: UKSG Project

• Assess the feasibility of developing and implementing journal Usage Factors- Level of support from author, librarian and publisher

communities

- Data from which UF would be derived

• COUNTER Journal Report 1?

• Article numbers

• Process for consolidation, calculation and reporting of UFs

- Factors in the calculation

• Level of reporting

• Total usage

• Articles

• Report submitted in May 2007

Global metrics: Usage Factor

Usage Factor =

Total usage over period ‘x’ of articles published during period ‘y’

Total articles published during period ‘y’

UKSG Project: feedback

• Are the COUNTER usage statistics sufficiently robust?

• Frustration at lack of comparable, quantitative data on journals

• Should items covered be restricted to articles?

• Many journals still have significant usage in print

• Diversity of views on the factors in the calculation

- Specified usage period

- Specified publication period

• Usage data is more susceptible to manipulation

• Will the journal be a meaningful concept in the future?

• Two measures with different limitations are better than one, and UF will be derived from a set of credible, understandable data

• Usage data will be used as a measure of value, whether publishers like it or not

Usage Factor: Recommendations

• That the UF concept be developed further, with a view to testing it as a practical, implementable measure of journal quality, value and status:-

- Test each of the individual elements in the UF equation using real publisher usage data

- Compare UF journal rankings with IF journal rankings

- Refine and investigate further the different workflow/organizational scenarios for the definition, calculation and dissemination of UFs

Usage Factor: Recommendations

• If satisfactory results are obtained from the above investigations and tests, it is likely that the system would have to be tested for one or two years to check the large scale validity of the outcomes before a comprehensive list of journal UFs is published.

Usage Factor: Next steps

• On May 18, 2007 the UKSG Committee accepted these recommendations.

• It asked the Project Working Group to draw up a set of practical proposals for putting the recommendations into practice.

• Working group efforts underway

• Perform pilot using actual usage data

• Determine:

- If a “Usage Factor” reasonable and practical

- What “issues” need to be addressed, within COUNTER? Elsewhere?

• NISO Website:http://www.niso.org/committees/SUSHI/SUSHI_comm.html

• COUNTER Website:http://www.projectcounter.org

• UKSG Usage Factorhttp://www.uksg.org/usagefactors

• SUSHI Schemas:http://www.niso.org/schemas/sushi/index.html

• SUSHI FAQ:http://docs.google.com/View.aspx?docid=d2dhjwd_63tkkwf

- What is the relationship of the COUNTER payload schema to the SUSHI schema? - What variable information has to be supplied in a SUSHI (client) request? - What variable information has to be supplied in a SUSHI (server) response? - What COUNTER reports can be delivered with SUSHI? - Does SUSHI support older Releases of the COUNTER reports?

Resources

Thank you!

Oliver Pesch

[email protected]

Documents

SUSHI and COUNTER