Mining Virtual Reference Data for an Iterative Assessment Cycle

Amanda Clay PowersVirtual Reference Project Manager

Ag & Life Sciences Librarian / Assistant ProfessorMississippi State University Libraries

http://amandaclaypowers.com@amandaclay

17th Annual Reference Research ForumJune 26, 2011

Mining Virtual Reference Data for

an Iterative Assessment Cycle

Managed from within Reference

Requires Systems/Reference Work

Ongoing transcript review in place

Chat at the MSU Libraries

@amandaclay

ACRL Value of Academic Libraries: A Comprehensive Research Review and Report (2010)

SACS Accreditation ReviewMSU Libraries

Measurement & Evaluation Committee formed

Assessment on the Mind

@amandaclay

@amandaclay

March 2010 – Web site redesign begins

April 2010 – Transcript analysis proposed

May 2010 – Analysis delivered

August 2010 –Web site and Discovery launched

Chronology

@amandaclay

@amandaclay

@amandaclay

@amandaclay

with Web Services Manager Clay Hill and Digital Projects Web Services Specialist Julie Shedd

Write-up of April 2010 data analysis and results published in the Journal of Web Librarianship

Includes Wordle analysis of 1800+ email and chat questions

“The Role of Virtual

Reference in Library Web

Site Design: A Qualitative Source for

Usage Data”

@amandaclay

Faculty Chat Wordle:

Entry Questions and Departments

@amandaclay

Do “Topic Searches,” “Finding Known Article Searches,” and “Online Catalog Help” searches decrease in frequency?

Will reluctant librarians, burned by federated searching, adopt Discovery?

Lingering Questions

@amandaclay

“Integrating the sampling of virtual reference data into an iterative assessment cycle….can allow for ongoing measures to be taken of the library’s effectiveness.” (Powers et al, 111)

The paper recommends twice yearly assessments using the methodology.

Taking Your Own

Medicine

@amandaclay

Based on Flynn, Gilchrist and Olson. 2004. “Using the Assessment Cycle as a Tool for Collaboration.” Resource Sharing & Information Networks 17(1):187-203.

Cycle of Assessment

Goal:To evaluate the

Web Site redesign and

Discovery Implementation

@amandaclay

Using a modified Grounded Theory (GT) model

Evaluate chat transcripts from April 2010

Follow up with an evaluation of April 2011 using the same model

Methodology

IRB approvalExcel Spreadsheet—

identifier, entry question, “Questions,” “Answers”

Review transcripts and create short descriptions of transactions

Refer to the data to find patterns

Use patterns to create codes

Code data to quantify

Breaking It Down

@amandaclay

Lower number of chat transactions in April 2011

For coding, types of questions asked = questions

For coding, tools used in answering questions = answers

Constructs

@amandaclay

@amandaclay

Number of Chats in April (2008-2011)

-25% ?

@amandaclay

Photo credit: Ryan Hoke, MSU Meteorology Student and Storm Chaserhttp://ryanhoke.com/

Identifying Patterns

in Question Types

Question Category Pattern Identified / Code Created

articles on a topic Topic Search

placing holds in the OC; using Discovery to find books in the library

Finding citation

three articles on a topic

topic search (all formats) looking for state guidelines on cheesemaking

trouble with using Scifinder, problems with the link resolver

Finding citations (two)

books on animal testing

Literary Criticism search, database searching questions, looking for general research strategies on a paper topic,

TOPIC SEARCH

@amandaclay

38%

28%

44%

@amandaclay

Question Analysis Results

Decrease in “Topic Search” questions

Slight decrease in “Finding Known Articles”

Increase in questions related to the Online Catalog/Holds

@amandaclay

But What About the

Chat Librarians?

Tools used in chat are the “answers” counted as defined:

Databases from Libraries’ Database Portal list

Service Referrals to Specialists (internal or external)

Other Resources not above include ILL, librarian answered (RR), guides, print sources, etc.

@amandaclay

April 2010 April 2011

Databases Used to Answer Chat Questions

@amandaclay

@amandaclay

Discussion

Overall chat transactions decrease by 25%

Topic Search questions decrease

Discovery supplants use of Academic Search Premier for chat librarians

Online catalog/holds questions and references increase

@amandaclay

Discovery is not taking the place of the Online Catalog despite being included in Discovery search

In February 2011, 17 libraries were added for a total of 8 systems with 42 libraries in our Online Catalog

What’s Up with the

Online Catalog?

@amandaclay

Difficult to compare 120 questions and 90 questions

Assessment needs change over time—past data analysis may need to be revisited

Limit yourself to what you really need to know

Challenges

@amandaclay

Looking Forward: Modifications for

Future Assessment

@amandaclay

Need more data points to confirm Topic Search and possible Finding Known Articles reduction

Discovery has been adopted by chat librarians in lieu of Academic Search Premier

Something is going on with the online catalog worth investigating

Conclusions

Questions?