Assessing the Bang for the Buck: Tools for Measuring & Evaluating Electronic Resources, Part II

Denise M. DavisConsultant, EMIS Training Project

August 19, 2004Carnegie Center, Phoenix, AZ

This training is made possible by a grant from IMLS and the Arizona State Library, Archives and Public Records

Purpose

This site provides assistance and continuing education for those seeking to use e-metrics in their library. E-metrics are measures of the library's electronic resources and services.

What are E-metrics?

E-metrics are measures of the library's electronic resources and services, e.g., number of Virtual Reference Transactions. E-metrics can focus on any aspect of the library's electronic resources or services or can be used with other measures .

E-metrics offer one approach among others to evaluate various aspects of networked information services. See the Selected Networked Information Service Evaluation Approaches Chart [HTML file EvalApproachChart.htm] for brief overviews of other evaluation approaches.

Better understand the measurement process

Follow the measurement lifecycle, begin with:

E-Metric SelectionE-Metric Data Collection E-Metric Data Analysis E-Metric Presentation & Use

Knowing purpose and audience for the measure before adopting

Choose an Existing E-metric Rather than Re-invent the Wheel

Fitting measure choice into an existing evaluation plan

Avoiding data collection barriers that reduce utility

Ensuring data analysis is possible

Preparing for effective presentation and use of results

Does the cost of data collection, analysis & use exceed the benefit ?

Seeking the perfect measure

Getting What Data are Needed

Network use data collecting are impacted by:

IT governance / management and system configurations

Comparability from service providers, to content provider (vendor), to Integrated Library System (ILS) platform and design

Data collection methods and dissemination formats and methods

Traditional Reporting Structures Need Review

Technology staff may be centralized with local or county government, school districts, etc. This may make data reporting a value added service for your library.

Technology staff may be on contract – the local ISP. Be sure you request monthly usage logs as part of your contract with the ISP.

Collaboration between library staff in technical services and public services is critical. In some cases, those “labels” no longer exist, so be sure the right people are involved in developing data collection procedures.

Achieving Results

What is the question you want to answer?

What data are needed to answer the question?

Is the question a benchmark, performance measure, or an outcome? Depending on the question, data may be analyzed differently.

There are no mistakes, only opportunities to understand how, why and how frequently library services and collections are used.

Modules

EMIS Site Introduction How to navigate the EMIS web site and to

make use of the features available on the web site efficiently.

Introduction to E-Metrics

E-metrics Introduction

E-metric Selection Data Collection Data Analysis Data Presentation & Use

E-Metrics Summary

Information Request Virtual Reference Transactions (7.3.1-- NISO) How to account and get credit for the library's virtual

reference services including: e-mail correspondence, library website reference forms, "chat," video-conferencing, audio-looping, co-browsing, etc.

Electronic Collection Units / Records Examined (7.8.1.1-- NISO) Searches / Menu selections (Queries) (7.8.1.2--

NISO) Sessions(7.8.1.3-- NISO)

Services Number of Public Access Workstations(7.7.1-- NISO)

How to collect and use data on the number library workstations available to provide the public access to the Internet.

Public Access Workstations Users (7.7.2- NISO) How to collect and use data on the number of library users of

the library's workstations that provide the public access to the Internet.

User Orientation and TrainingUser Training (7.9.5-- NISO) How to define, collect and analyze core data

of information technology user training.

Operating Expenditures

Electronic Access Expenditures (6.2.8-- NISO)

How to design methods for collecting, analyzing electronic access expenditures within and among other relevant library expenditures, and using these data to market library services.

Electronic Material Expenditures (6.2.9-- NISO)

How to collect and analyze electronic materials expenditures in a context of other library expenditures, issues to consider, methods of reporting these data, and models of data analysis.

Library Visits (7.8.1.5 – NISO)

Increasingly, "visits" to the library occur from outside the library itself. Library users come to library websites, access databases, and do use a number of library services/resources from their homes, offices, or other locations. This module describes how to count virtual visitors, and present them as part of an overall library visitor count.

E-metric Selection

ObjectivesThis module acquaints you with e-metric selection issues including:

Determining purpose and audience for the measure before adopting; Choosing an existing e-metric rather than re-inventing the wheel; Fitting measure choice into an existing evaluation plan; Avoiding data collection barriers that reduce utility; Ensuring data analysis is possible; Preparing for effective presentation and use of results; Does the cost of data collection, analysis & use exceed the benefit; and Seeking the perfect measure and the need to reduce error.

Data Collection

ObjectivesThis module acquaints you with e-metric data collection issues

including: Appointing a Data Coordinator; Addressing the basics of data collection; Using new data collection techniques; Addressing scheduling issues; Obtaining data from the library's external partners; Preparing data collectors; Addressing staff training needs; and Allowing additional time to roll out data collection for a new e-

metric.

Data Analysis

Objectives This module acquaints you with e-metric data analysis issues including: Determining your audience(s) and purpose(s); Determining when is an estimate good enough; Describing the basic data analysis strategies; Describe the strategies whereby measures be added together; Describe the strategies whereby measures be subtracted; Describe the strategies whereby measures be divided; and Describe the strategies whereby measures be multiplied.

See the Annual Report of Library Electronic Resources and Services (MS Word)for further ideas.

Data Presentation & Use

Objectives

This module acquaints you with e-metric data presentation and use including:

Determining your audience(s) and purpose(s); Determining whether the audience like the data; Anticipating the need to explain the results; Determining the model for presenting the data effectively.

See the Annual Report of Library Electronic Resources and Services (MS Word) for further ideas.

Virtual Reference Transactions (7.3.1– NISO)

Objective At the end of this module, you should be able to: Define the Virtual Reference Transactions e-metric; Prepare your library to collect this data; Execute the procedures for data collection; Use the results of this e-metric for decision making;

and Explain some of the issues involved in collecting and

reporting this e-metric.

Electronic Collection Units / Records Examined (7.8.1.1-- NISO)

Content in the electronic collection that is delivered to a user. The sub-categories that follow provide for a detailed breakdown by type of content delivered (full-content unit, or descriptive record) and system delivering the content (Library Collection, Commercial Service or OPAC).

7.8.1.1.1 Library Collection Full-Content Units Examined Number of full-content units from library electronic collection examined, downloaded, or otherwise supplied to a user. Exclude OPAC or commercial services (i.e., online databases).

7.8.1.1.2 Commercial Services Full-Content Units Examined Number of subscription service full-content units examined, downloaded, or otherwise supplied to user, to the extent that these are recordable and controlled by the server rather than the browser. (ICOLC Guidelines, December 2001). Note 1: Journal articles - by journal title with ISSN and title listed. Note 2: Ebooks ? by book title with ISBN and title listed. Note 3: Reference materials ? by content unit appropriate to the resource (e.g., dictionary definitions, encyclopedia articles, biographies, etc.). Note 4: Non-textual resources ? by file type as appropriate to resources (e.g., image, audio, video, etc.). (ICOLC Guidelines, December 2001)

7.8.1.1.3 Library Collection Descriptive Records ExaminedNumber of descriptive records concerning the library?s electronic collection delivered to a user. Determined by the record type appropriate to the resource, e.g., abstract, archive, and index. Exclude OPAC or commercial services (i.e., online databases).

7.8.1.1.4 Commercial Services Descriptive Records ExaminedNumber of descriptive records concerning the library?s commercial services delivered to a user. Determined by the record type appropriate to the resource, e.g., abstract, archive, index. Exclude OPAC or library collection descriptive records (i.e., library website services and collections).

7.8.1.1.5 OPAC Descriptive Records ExaminedNumber of descriptive records from the library?s online catalog delivered to a user. Exclude commercial services or library collection descriptive records (i.e., library website services and collections).

Searches / Menu selections (Queries)

(7.8.1.2-- NISO)

7.8.1.2 Searches/Menu Selections (Queries)A search is defined as intending to represent a unique intellectual inquiry whether conducted through a search form submitted to the server or through the use of menu selections (e.g. browsing a list of subjects.)

7.8.1.2.1 Library Collection Searches (Queries)A search is defined as intending to represent a unique intellectual inquiry. Typically a search is recorded every time a search request is submitted to the server. Limited to library electronic collection services. Include menu selection searches. Exclude spider/crawler searches. Excludes OPAC searches. (ISO 2789, Annex A)

7.8.1.2.2 Commercial Services Searches (Queries)A search is defined as intending to represent a unique intellectual inquiry. Typically a search is recorded every time a search request is submitted to the server. Limited to commercial services. Mistyped search strings do not represent unique intellectual inquiries. Include menu selection searches. Exclude spider/crawler searches. (ISO 2789, Annex A)

7.8.1.2.3 OPAC Searches (Queries)A search is defined as intending to represent a unique intellectual inquiry. Typically a search is recorded every time a search request is submitted to the server. Limited to the library online catalog service.Include menu selection searches. Exclude spider/crawler searches. (ISO 2789, Annex A)

Sessions (7.8.1.3-- NISO)

7.8.1.3 SessionsA session is defined as a successful request of an online service or library?s online catalog. It is one cycle of user activities that typically starts when a user connects to the service or database and ends by terminating activity that is either explicit (by leaving the service through exit or log-out) or implicit (timeout due to user inactivity).

7.8.1.3.1 OPAC SessionsA session is defined as a successful request of the library?s online catalog. It is one cycle of user activities that typically starts when a user connects to the OPAC and ends by terminating activity in the OPAC that is either explicit (by leaving the database through log-out or exit) or implicit (timeout due to user inactivity). In some cases, e.g. OPAC use inside the library, several users one after the other might make use of the same workstation, and sessions could not be separated. In most systems, a session is cut off after a specified time of non-use, thus avoiding part of the problem. The average timeout setting would be 30 minutes. If another timeout period is used this should be reported. Browser or proxy cashing will be likely to reduce the number of requests registered in logfiles. (ISO 2789, Annex A; modified to exclude commercial services).

7.8.1.3.2 Commercial Services SessionsA session is defined as a successful request of a commercial service (e.g., online database). It is one cycle of user activities that typically starts when a user connects to a database and ends by terminating activity in the database that is either explicit (by leaving the database through log-out or exit) or implicit (timeout due to user inactivity). (ISO 2789, Annex A, modified to exclude OPAC sessions). Note 1: For multiple databases compiling several individual databases further information should be provided as to the separate databases hosted. Note 2: In some cases, e.g. database use inside the library, several users one after the other might make use of the same workstation, and sessions could not be separated. In most systems, a session is cut off after a specified time of non-use, thus avoiding part of the problem. The average timeout setting would be 30 minutes. If another timeout period is used this should be reported. Browser or proxy cashing will be likely to reduce the number of requests registered in logfiles.

Use - Databases

Digital Documents eBooks Commercial Services Descriptive Records Examined Commercial Services Full-content Units Examined Commercial Services Searches (Queries) Commercial Services Sessions Databases Items Requested in Electronic Databases Library Collection Searches (Queries) OPAC Descriptive Record Examined Rejected Sessions Searches / Menu selections (Queries) Sessions Number of Sessions Turnaways Units / Records Examined Users of Electronic Resources per Typical Week

Services

Number of Public Access Workstations(7.7.1-- NISO)

How to collect and use data on the number library

workstations available to provide the public access to the Internet.

Public Access Workstations Users (7.7.2- NISO) How to collect and use data on the number of

library users of the library's workstations that provide the public access to the Internet.

User Orientation and Training

User Training (7.9.5-- NISO) How to define, collect and analyze core data

of information technology user training.