Scholars Portal :An Evaluation Overview
Mary Cavanagh, doctoral candidateFaculty of Information Studies
FIS 1325S
Adapted with permission of A. Darnell, OCUL Scholars Portal
Three sets of data
Usage data available through ScholarsPortal (SP) reports
Ontario Council of University Libraries(OCUL) Librarian & User Survey
Assocation of Research Libraries (ARL)– MINES data
Article Downloads
july 2002 - june 2005
toronto 3,567,865 32.69%
waterloo 1,053,530 9.65%
guelph 846,129 7.75%
ottawa 776,910 7.12%
western 712,242 6.53%
york 646,119 5.92%
trent 493,234 4.52%
brock 462,398 4.24%
carleton 440,817 4.04%
mcmaster 437,115 4.01%
queens 411,584 3.77%
windsor 352,208 3.23%
ryerson 323,293 2.96%
laurier 169,920 1.56%
laurentian 122,639 1.12%
lakehead 62,093 0.57%
nipissing 18,882 0.17%
uoit 15,633 0.14%
total 10,912,611 100.00%
Top Ten Downloads
Downloads Title(Publisher)
87877 Lancet, The(Elsevier Science)
85393 Social Science and Medicine(Elsevier Science)
81374 Phytochemistry(Elsevier Science)
79992
Journal of Personality and Social Psychology(American Psychological
Association)
56511 Animal Behaviour(Elsevier Science)
50223 Personality and Individual Differences(Elsevier Science)
49551 Sex Roles(Kluwer Academic/Plenum Publishers)
43775 Developmental Psychology(American Psychological Association)
36867
International Journal of Eating Disorders(Wiley Subscription
Services, Inc., A Wiley Company)
36699 Child Abuse and Neglect(Elsevier Science)
Un-met Demand
RefWorks Accounts
Citations Captured
RefWorks Users
Reason for Use
User Satisfaction
Scholars Portal Search Working GroupReport of Findings
November 2005Sandra Langlands, TorontoZita Murphy, RyersonPatti Ryan, YorkHarriet Rykse, WesternMartha Whitehead (chair), Queen’sDeborah Wills, Wilfrid LaurierPeter Zimmerman, Windsor
Working Group Mandate
Solicit feedback from librarians andusers on Illumina functionality
Conduct research on the “single search”approach to searching multipledatabases
Make recommendations to OCUL’sScholarly Information Resources TaskGroup regarding Illumina promotionand instruction
User Study Objectives
Learn about students’ and researchers’satisfaction with aggregated search results
Draw conclusions regarding proposed accessroutes in Scholars Portal Search
Methodology
Usability study principle - 5 usersAt our 7 institutions - range of sizes and
programs23 users in total - 10 undergrad, 1 staff
(novice), 5 graduate students, 7 facultyRange of disciplines
Methodology
Users were asked to comment on threedifferent sets of search results inScholars Portal Search, from:– A search of all databases– A search of one of subject area groupings
of databases, chosen from the drop-downmenu
– A search of a single database
Observations
Which results did users prefer?Inconsistent, but half of the faculty and
graduate students preferred the resultsfrom the search of all databases
Subject Area groupings are problematicSingle database search results preferred
by some novice users
Observations
Relevance, reaction to large result setsPoorer than algorithms that use full-text;
date sort more useful than relevance sortFew users looked at number of hits right
away, and high not always a deterrentJudged results by first few citations and
rarely went past first screenFew noticed descriptors
Observations
Interface and search functionalitySome observations similar to OCUL
staff:– Shouldn’t have to select databases– Quick search assumed ‘and’, not phrase– Tabs weren’t noticed– A “did you mean” search suggestion
would be good
Anywhere search most common
What Are Other StudiesSaying?
Faculty and grad students often findsignificant, previously unknown citations
Don’t make users choose or read beforesearching
Groupings eliminate “those long listsreally make me feel stupid”
But subject profile choices are difficult,and many choose ‘multidisciplinary’
SELECTED RESOURCES ONUSER TESTING
Augustine, S. and Greene, C. (2002). Discovering how students search a library web site: a usability casestudy. College & Research Libraries 63(4): 354-3. - Good example of a usability test.
Crowley, G.H. et al (2002). User perceptions of the library's web pages: a focus group study at TexasA&M University. Journal of Academic Librarianship 28(4): 205-210. - Describes the focus groupprocess.
Dickstein, R. and Mills, V. (2002). Usability testing at the University of Arizona Library: how to let theusers in on the design. Information Technology and Libraries 19(3): 144-51. Step-by-step descriptionof user-centred design.
Fichter, D. (2004). Heuristic and cognitive walk-through evaluations. Online 28(3): 53-56. Comparesand contrasts the two methodologies.
Head, A.J. (1999). Web redemption and the promise of usability Online 23(6): 21-32. clear, succinctdiscussion of different types of tests.
McGillis, L. and Toms, E.G. (2001). Usability of the academic library web site implications for design.College & Research Libraries 62(4): 355-368. Includes useful description of methodology.
McMullen, S. (2001). Usability testing in a library web site redesign project. Reference Services Review29(1): 7-22. Includes sample questions.
Mach, T. et al (2004). Designing for experts: how scholars approach an academic library web site.Information Technology and Libraries 23(1): 16-22. Describes how usability sessions were conducte
Spool, J.M. et al (2004). Designing for the Scent of Information. UIE Report. **Highly recommended forwebsite design
Usability Research Lab, NCSU Libraries User Interface Engineeringhttp://www.lib.ncsu.edu/dli/usability/library_usability.htmlhttp://www.uie.comUsability Resources, compiled by Darlene Fichter useit.com (Jakob Nielsen)
http://library.usask.ca/~fichter/usability/http://www.useit.com/Prepared by Deborah Wills, Wilfrid Laurier University Library, December 2004
Demonstrating Value and Creating Value:Evidence-Based Library Management
Martha KyrillidouDirector of the ARL Statistics and Measurement Program
Association of Research Libraries
Brinley FranklinVice Provost for University Libraries
University of Connecticut
Scholars Portal Forum, Ontario Council of University LibrariesFebruary 1 2006
Toronto
www.arl.org/stats/
Bangor University considers removing librarians posted byBlake on Thursday January 27, @07:30AM -753 hits
Ms Information writes "News from the University of Wales Bangorin the UK. senior management no longer feel that subject librarians /academic liaison librarians are needed in the modern academic library.They have made restructuring proposals which include removing allbar one of the subject librarians and a tier of the library management,including the Head of Bibliographic Services. The universitymanagement thinks that technology has 'deskilled' literaturesearching. As far as I know, this proposal is unprecedented in theUnited Kingdom.In essence, there will remain 4 professional librarians serving a'research-led' university of 8,000 plus FTEs and with 8 library sites.These will be the university librarian, cataloguing librarian,acquisitions librarian and Law librarian.Has anything like this happened anywhere that you know of? If so,what have been the effects?
Libraries Remain aLibraries Remain aCredible Resource in 21Credible Resource in 21stst
CenturyCentury
98% agree with statement98% agree with statement, , ““My My …… library librarycontains information from credible andcontains information from credible andknown sources.known sources.””
Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.
Changing BehaviorsChanging Behaviors
Recent Survey:Recent Survey:Only Only 15.7% agreed with the statement15.7% agreed with the statement ““TheTheInternet has not changed the way I use theInternet has not changed the way I use thelibrary.library.””
Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.
Total Circulation
400,000
450,000
500,000
550,000
600,000
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
Total Circulation
Note. M. Kyrillidou and M. Young. (2003).ARL Statistics 2002-03. Washington, D.C.: ARL, p.8.
Reference TransactionsReference Transactions
90,000
100,000
110,000
120,000
130,000
140,000
150,000
160,000
170,000
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
Note. M. Kyrillidou and M. Young. (2003).ARL Statistics 2002-03. Washington, D.C.: ARL, p.8.
ARL Overall
MINES for LibrariesTM
Measuring the Impact of Networked Electronic InformationServices
MINES is a transaction-based research methodologyconsisting of a web-based survey form and a randommoments sampling plan.
MINES typically measures who is using electronic resources,where users are located at the time of use, and their purposeof use.
MINES was adopted by the Association of ResearchLibraries (ARL) as part of the “New Measures” toolkit inMay, 2003.
MINES is different from other electronic resource usagemeasures that quantify total usage (e.g., Project COUNTER,E-Metrics) or measure how well a library makes electronicresources accessible (LibQual+TM).
• How extensively do sponsored researchers useOCUL’s Scholars Portal? How much usage is for non-funded research,instruction/education, student research papers, and course work?
• Are researchers more likely to use the Scholars Portal from inside oroutside the library? What about other classifications of users?
• Are there differences in Scholars Portal based on the user’s location(e.g., in the library; on-campus, but not in the library; or off-campus)?
• Could MINES, combined with usage counts, provide an infrastructure tomake Scholars Portal usage studies routine, robust, and easily integratedinto OCUL’s administrative decision-making process for assessingnetworked electronic resources?
Questions Addressed ByMINES for Libraries™ for the
OCUL Scholars Portal
OCUL/MINES MethodologicalConsiderations
The sampling plan was determined at the outset. Surveys were
conducted once a month for two hours a month between May, 2004
and April, 2005
The selection of the monthly survey periods were weighted based on
usage counts by time of day and were chosen randomly.
Participation was mandatory, negating on-respondent bias, was based
on actual use in real-time, and was brief (to minimize user
inconvenience).
OCUL designed the local questions, mounted the survey, collected
data and sent it to ARL for tabulation in aggregate and by individual
institution.
If more than one search was conducted by a user, the survey form was
auto-populated with initial responses as the default.
OCUL/MINES MethodologicalConsiderations (continued)
Each participating library explained the survey and its confidentialityprovisions to their local constituency.
Research ethics officers and/or Ethics Review Boards, where necessary,reviewed and approved the survey instrument and methodology.
OCUL determined that individual institutions and their institution-specific data collected during the survey periods would not bedisclosed. Individual data was anonymous.
The mandatory nature of the survey required discussion on somecampuses and caused one OCUL member library to withdraw from thestudy.
Two institutions pre-tested the survey in January, 2004. Data collectionprogramming and configurations/links had to be revised in Februaryand March, 2004.
After completing the survey, users were connected to their desiredScholars Portal networked electronic resource.
OCUL Scholars Portal UsageAffiliation
Reason for Use
21.6%Other4388Other
4.6%Course Reading925Course Reading
3.1%Recommended Librarian620Recommended Librarian
30.0%Reference/Citation6090Reference/Citation
12.0%Recommended Colleague2436Recommended Colleague
50.4%Important Journal10219Important Journal
PercentReason for Use (n=20293)FrequencyReason for Use (n=20293)
How extensively do sponsored researchersuse OCUL’s Scholars Portal? How muchusage is for non-funded research,instruction/education, student researchpapers, and course work?
MINES for Libraries™shows that theScholars Portal resources are heavily usedby faculty and students in all OCUL. Themajority of the use is from the sciencesand the medical field and particularly inthose fields the majority of the use is forsponsored research purposes.
Are researchers more likely to use theScholars Portal from inside or outside thelibrary? What about other classificationsof users?
Most faculty, graduate professionals andundergraduates uses of the ScholarsPortal are from outside the librarybuilding. Undergraduates though do showmany uses of the Scholars Portal fromwithin the library as they are probablybecoming more exposed to these resourcesby having more physical contact with thelibrary.
Are there differences in Scholars Portalbased on the user’s location (e.g., in thelibrary; on-campus, but not in the library;or off-campus)?
Most of the faculty and graduateprofessionals use Scholars Portal eitherfrom on-campus locations outside thelibrary or from other off-campus locations.Most of the uses from these locationsoutside the library are for sponsoredresearch purposes.
Could MINES, combined with usagecounts, provide an infrastructure tomake Scholars Portal usage studiesroutine, robust, and easily integratedinto OCUL’s administrative decision-making process for assessingnetworked electronic resources?
Yes
Issues with web surveys:brief bibliography
Cook, Colleen; Heath, Fred; and Russell L. Thompson. 2000 (December). “A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys.” Educational andPsychological Measurement 60(6): 821-836.
Couper, Mick P.; Traugott, Michael W.; and Lamias, Mark J. 2001. "Web Survey Design andAdministration," Public Opinion Quarterly, 65 (2): 230-253.
Covey, Denise Troll. . 2002. Usage and Usability Assessment: Library Practices and Concerns.CLIR Publication 105. Washington DC: Council on Library and Information Resources.
– http://www.clir.org/pubs/reports/pub105/contents.html Dillman, D.A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method. 2nd
Ed. New York: John Wiley & Sons. Gunn, Holly. 2002. “Web-based Surveys: Changing the Survey Process.” FirstMonday
7(12).– http://firstmonday.org/issues/issue7_12/gunn/index.html
LIBQUAL+ ™ Spring 2004 Survey. 2004. Cook, Colleen, and others.– http://www.libqual.org/documents/admin/ARL_Notebook2004.pdf
Schonlau, Matthias; Fricker Jr., Ronald D.; and Elliott, Marc N. 2002. Conducting ResearchSurveys via E-Mail and the Web. Santa Monica, CA: RAND.
Tenopir, Carol, with the assistance of Brenda Hitchcock and Ashley Pillow. 2003 (August).Use and Users of Electronic Library Resources: An Overview and Analysis of Recent ResearchStudies. Washington DC: Council on Library and Information Resources.
– http://www.clir.org/pubs/reports/pub120/contents.htmls Thomas, Susan J. 2004. Using Web and Paper Questionnaires for Data-Based Decision Making:
From Design to Interpretation of the Results. Thousand Oaks, Corwin Press. Thompson, Bruce.; Cook, Colleen.; Thompson, Russell L. 2002. Reliability and Structure of
LibQUAL+™ scores: Measuring Perceived Library Service Quality. portal: Libraries and theAcademy.3-12.