Upload
roy-warner
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Discovery Tool Evaluation: A Three-Pronged Approach
9th Annual AMICAL Meeting and ConferenceSharjah, UAE
Background
• AUS implemented Encore Discovery in 2009• Other products like Serials Solutions Summon and
EBSCO’s EDS are actively marketing in the Middle East• Decision in 2010 to evaluate all products• Having Encore installed gave us opportunity to evaluate
products against the same databases, and think about how an ILS-integrated discovery solution compares to a less integrated solution
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Methodology
• Setting up trials of Summon and EDS• Scheduling vendor presentations of Summon and EDS• Testing all 3 products for ability to produce useful,
relevant and accessible results• Reviewing the features of each of the products• Performing usability testing with all products• Assessing the costs of implementing each product• Assessing the impact of implementing each product on
other library systems and operations
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Usability Testing
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Informal Usability Testing
• Think Aloud Protocol
• 5 participants per product
• Participants only told they would be testing a new discovery tool to find library resources
• Participants used their own research topic to complete realistic tasks
• Observers did not interact with participant except to give tasks and encourage thinking aloud
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Testing Environment
• 1 test administrator • 1 observer• Laptop with built in microphone for recording voice• Camtasia Studio with Powerpoint plug-in to capture
screen and voice• Browser of choice• Other software, websites as requested
– Google and Microsoft Word for spelling
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Tasks
Based on common assignments requirements seen at the Research Help Desk, participants were asked to find:
• an article• an article from a peer reviewed/scholarly/academic
journal• a newspaper article• a reference book• a general collection book• a book published in 2008 or 2009
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Observations
• Limit / facet confusion– “Format” or “Content Type” vs “Location” – Books, e-books, e-journal
• E-journal format was used to find articles
• Limits reset for each new search
• Click through to article– Expect to get article, but given a WebBridge or 360 Link page
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
System Usability Scale
• Participants filled out System Usability Scale questionnaire for perceived ease-of-use score
• 10 item Likert scale questionnaire used to determine user’s perception of the system for– Effectiveness– Efficiency– Satisfaction
• Resulting scores range from 0 to 100– Average SUS score across 500+ studies is 68
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
System Usability ScoresSystem Usability Scale Encore EDS Summon
1. I think that I would like to use this system frequently 4 3.2 4.4
2. I found the system unnecessarily complex 2.4 3 2.4
3. I thought the system was easy to use 4 3.4 4.2
4. I think that I would need the support of a technical person to be able to use this system
2.4 2.2 1.8
5. I found the various functions in this system were well integrated 4.4 3.4 3.8
6. I thought there was too much inconsistency in this system 2 2.6 1.4
7. I would imagine that most people would learn to use this system very quickly
4 3.2 4
8. I found the system very cumbersome to use 2.2 2.4 1.8
9. I felt very confident using the system 4 3.6 4
10. I needed to learn a lot of things before I could get going with this system
2.2 3.2 2
Average System Usability Score 73.5 58 77.5
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Overall Usability
• Summon– Pop-up abstract and number of results for limit in brackets
were very popular
• Encore – Worked very well for books– Article search interface (Synergy) not well understood or used
• EDS – Limits not intuitively named, extra “Update” click to refine– Can set mutually exclusive limits giving 0 results
• Preference to use web for finding newspaper articles
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Relevancy and Functionality
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Relevancy Test
• Testers - AUS public service librarians• Searches - Actual searches done this year in Encore• Scoring - Scale 1 - 5 + comments• Scorecards - One scorecard for each search in each
discovery tool. Extensive comments• Discovery tools tested
– Encore – Summon– EDS
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Actual SearchesSelected from Google Analytics
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Encore URLs
• /iii/encore/search/C|Scarbon+fiber+reinforced+polymer|Orightresult|U1?lang=eng&suite=pearl
• /iii/encore/search/C|Sglobalization|Orightresult|U1?lang=eng&suite=pearl
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Searches
• arranged+marriage• carbon+fiber+reinforced+polymer• Electronics+All-In-One+Desk+Reference+For+Dummies• electronics+for+dummies• globalization • jean+luc+godard• sex+education• uae+federal+laws• walt+disney• "Artificial+intelligence"+And+"Manufacturing"
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Types of Results
• Top 5 results– Number of:
• Books • Articles• Other (e-books, e-journals, DVDs, e-resources,
print journals, videos, etc)
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Types of ResultsBooks Articles Other
EDS 1 2 3 0EDS 2 0 0 5 reportsEDS 3 1 2 2 e-booksEDS 4 1 4 0EDS 5 0 5 0
Summon 1 0 5 0Summon 2 0 3 2 newspapersSummon 3 3 2 0Summon 4 1 3 1 e-journalSummon 5 0 5 0
Encore 1 3 2 0Encore 2 3 2 0Encore 3 1 2 2 e-booksEncore 4 2 2 1 e-journalEncore 5 3 2 0
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Relevancy Assessment
Overall reaction to top 5 results:
5 Excellent (just what I was looking for)4 Very good (on target and useful)3 Fair (acceptable)2 Poor (not useful)1 Bad (unrelated to topic)
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Test Searches Scorecard
Discovery Tool
Test Search
Top 5 results
Number of:
Books
Articles
Other (e-books, e-journals, DVDs, resources, print journals, videos, etc)
Relevancy assessment – overall reaction to top 5 results:
5 Excellent (just what I was looking for) 4 Very good (on target and useful) x 3 Fair (acceptable) 2 Poor (not useful) 1 Bad (unrelated to topic)
Comments:
Books were excellent, articles were poor.
Encore Synergy
3
2
arranged marriage
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
ResultsTester 1 Tester 2 Tester 3 Tester 4 Tester 5 AVG Total Score
Summon 1 1 1 4 1 1 1.6 8Summon 2 3 3 3 4 3 3.2 16Summon 3 5 5 5 5 4 4.8 24summon 4 4 5 5 5 3 4.4 22Summon 5 4 5 4 5 3 4.2 21SUMMON TOTAL 17 19 21 20 14 18.2 91
EDS 1 4 4 4 4 3 3.8 19EDS 2 2 3 2 3 2 2.4 12EDS 3 4 4 4 4 4 4 20EDS 4 4 4 3 3 3 3.4 17EDS 5 4 4 4 4 3 3.8 19EDS TOTAL 18 19 17 18 15 17.4 87
Synergy 1 2 3 1 3 3 2.4 12Synergy 2 3 3.5 2 3 5 3.3 16.5Synergy 3 2 3 3 5 2 3 15Synergy 4 2 3 3 2 3 2.6 13Synergy 5 4 4 2 4 4 3.6 18SYNERGY TOTAL 13 16.5 11 17 17 14.9 74.5
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Functionality
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Functionality Conclusions
Basically, all 3 systems offer similar user functionality – sorting, limiting, emailing and “did you mean” are available. There is no superior feature that would make one system preferable to another, just as there is no missing feature that would eliminate a system.
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Impact Assessment
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Pieces & Configurations
A-Z List Link Resolver Discovery Federated Search
1 Existing 360 Core WebBridge Encore Synergy Research Pro
2 Summon 360 Core 360 Link Summon na
3 Summon 360 Core WebBridge Summon na
4 EDS EH A-Z LinkSource EDS
5 EDS 360 Core WebBridge EDS
New
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Functional Equalization
• EDS Federated Search• EDS & Summon – Millennium Integration Pieces• III Mobile Interface (free if Airpac)• III Encore Harvesting• III Query API – in development
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Costs
• Sunk Costs to write off– Encore Server– WebBridge License
• Ongoing Costs– Quotes for each component
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Work Needed for any Change• Implementation
– Profiling of resources and catalog records– Interface customization where possible– Changes to website– Changes to electronic resources– Staff training– Modify instruction materials and curriculum
• Ongoing Work– Bibliographic Data Loads– Changes to Holdings Loads for EDS
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Potential Lost Functionality
• Tag Clouds & Authorities for related searching• Availability Facet• WebBridge/Pathfinder for Collection Maps• User tags• Relevancy modification• Millennium integration
– Automatic update/syncing– Hold placement– Collection map links– My Lists– Saving RP Groups in My Millennium
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Potential Added Functionality• Administrative interface for searching & user interface• Mobile Interface bundled in• Repository Integration bundled in• API’s bundled in• 1-Click in 360 Link
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Final Scoring
Factor Summon EDS Encore Weight Summon EDS Encore
Relevance 3 2 1 0.3 0.9 0.6 0.3
Usability 3 1 2 0.3 0.9 0.3 0.6
Financial Impact 3 1 2 0.2 0.6 0.2 0.4Implementation Work Impact 1 2 3 0.1 0.1 0.2 0.3
Ongoing Work Impact 2 1 3 0.1 0.2 0.1 0.3
12 7 11 2.7 1.4 1.9
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
Thank you
Thomas Hodge [email protected] Botter [email protected] Ann Nash [email protected]