View
553
Download
0
Embed Size (px)
DESCRIPTION
My presentation at the 17th Annual Reference Research Forum at #ala11.
Citation preview
Amanda Clay PowersVirtual Reference Project Manager
Ag & Life Sciences Librarian / Assistant ProfessorMississippi State University Libraries
http://amandaclaypowers.com@amandaclay
17th Annual Reference Research ForumJune 26, 2011
Mining Virtual Reference Data for
an Iterative Assessment Cycle
Managed from within Reference
Requires Systems/Reference Work
Ongoing transcript review in place
Chat at the MSU Libraries
@amandaclay
ACRL Value of Academic Libraries: A Comprehensive Research Review and Report (2010)
SACS Accreditation ReviewMSU Libraries
Measurement & Evaluation Committee formed
Assessment on the Mind
@amandaclay
@amandaclay
March 2010 – Web site redesign begins
April 2010 – Transcript analysis proposed
May 2010 – Analysis delivered
August 2010 –Web site and Discovery launched
Chronology
@amandaclay
@amandaclay
@amandaclay
@amandaclay
with Web Services Manager Clay Hill and Digital Projects Web Services Specialist Julie Shedd
Write-up of April 2010 data analysis and results published in the Journal of Web Librarianship
Includes Wordle analysis of 1800+ email and chat questions
“The Role of Virtual
Reference in Library Web
Site Design: A Qualitative Source for
Usage Data”
@amandaclay
Faculty Chat Wordle:
Entry Questions and Departments
@amandaclay
Do “Topic Searches,” “Finding Known Article Searches,” and “Online Catalog Help” searches decrease in frequency?
Will reluctant librarians, burned by federated searching, adopt Discovery?
Lingering Questions
@amandaclay
“Integrating the sampling of virtual reference data into an iterative assessment cycle….can allow for ongoing measures to be taken of the library’s effectiveness.” (Powers et al, 111)
The paper recommends twice yearly assessments using the methodology.
Taking Your Own
Medicine
@amandaclay
Based on Flynn, Gilchrist and Olson. 2004. “Using the Assessment Cycle as a Tool for Collaboration.” Resource Sharing & Information Networks 17(1):187-203.
Cycle of Assessment
Goal:To evaluate the
Web Site redesign and
Discovery Implementation
@amandaclay
Using a modified Grounded Theory (GT) model
Evaluate chat transcripts from April 2010
Follow up with an evaluation of April 2011 using the same model
Methodology
IRB approvalExcel Spreadsheet—
identifier, entry question, “Questions,” “Answers”
Review transcripts and create short descriptions of transactions
Refer to the data to find patterns
Use patterns to create codes
Code data to quantify
Breaking It Down
@amandaclay
Lower number of chat transactions in April 2011
For coding, types of questions asked = questions
For coding, tools used in answering questions = answers
Constructs
@amandaclay
@amandaclay
Number of Chats in April (2008-2011)
-25% ?
@amandaclay
Photo credit: Ryan Hoke, MSU Meteorology Student and Storm Chaserhttp://ryanhoke.com/
Identifying Patterns
in Question Types
Question Category Pattern Identified / Code Created
articles on a topic Topic Search
placing holds in the OC; using Discovery to find books in the library
Finding citation
three articles on a topic
topic search (all formats) looking for state guidelines on cheesemaking
trouble with using Scifinder, problems with the link resolver
Finding citations (two)
books on animal testing
Literary Criticism search, database searching questions, looking for general research strategies on a paper topic,
TOPIC SEARCH
@amandaclay
38%
28%
44%
@amandaclay
Question Analysis Results
Decrease in “Topic Search” questions
Slight decrease in “Finding Known Articles”
Increase in questions related to the Online Catalog/Holds
@amandaclay
But What About the
Chat Librarians?
Tools used in chat are the “answers” counted as defined:
Databases from Libraries’ Database Portal list
Service Referrals to Specialists (internal or external)
Other Resources not above include ILL, librarian answered (RR), guides, print sources, etc.
@amandaclay
April 2010 April 2011
Databases Used to Answer Chat Questions
@amandaclay
@amandaclay
Discussion
Overall chat transactions decrease by 25%
Topic Search questions decrease
Discovery supplants use of Academic Search Premier for chat librarians
Online catalog/holds questions and references increase
@amandaclay
Discovery is not taking the place of the Online Catalog despite being included in Discovery search
In February 2011, 17 libraries were added for a total of 8 systems with 42 libraries in our Online Catalog
What’s Up with the
Online Catalog?
@amandaclay
Difficult to compare 120 questions and 90 questions
Assessment needs change over time—past data analysis may need to be revisited
Limit yourself to what you really need to know
Challenges
@amandaclay
Looking Forward: Modifications for
Future Assessment
@amandaclay
Need more data points to confirm Topic Search and possible Finding Known Articles reduction
Discovery has been adopted by chat librarians in lieu of Academic Search Premier
Something is going on with the online catalog worth investigating
Conclusions
Questions?